Put simply: It's one of the cornerstones of the “internal market” (the current EU official term for what has also been called the “unique” or “single” market). The theory is that freedom of movement for goods, capital, services and persons ensure the most efficient allocation of resources and convergence between EU economies. Many people seem to imagine it's a detail, something that could easily be extirpated from the rest of EU rules or something other countries forced the UK to accept for unfathomable reasons. It's not, it's an integral part of a coherent package and something the UK was originally very keen on.
Similarly, you can't pick and choose between apparently silly regulations like those on the contents of pillows and whatnot, everybody has to abide by the same rules to allow products to be sold across the EU without having to deal with a separate standard on safety, quality, etc. for each country. That's the reason why Norway (which is in the EEA) and Switzerland (which rejected even that) still have to implement most of these rules anyway (with one big exception: agriculture). There is no way around it without seriously endangering the whole construction (which might nonetheless happen, I am not trying to predict where the negotiations will lead).
The long-term objective is that EU economies should reach a level of integration such that for each economic decision (hiring an employee, choosing an employer or supplier, buying a product, etc.), you could transparently make a transaction with a business or person elsewhere in the EU as if it was a business or person in your own country, without worrying about extra paperwork, delays, etc.
That's the theoretical and economical reason why freedom of movement for workers was an integral part of the project from the start (back in the days of the European Coal and Steel Community and then in the 1959 Treaty of Rome), long before the creation of the EU and the mostly failed attempts at extending its remit to things like defense or border security or the catastrophic experiment that is the euro.
Therefore, the original argument for all this is primarily about economics, as you suspected. Historically, the freedom of movement thus only covered workers. The Maastricht treaty added some limited rights for economically non-active people (e.g. retirees, under the condition that they have sufficient income) and case law extended them (e.g. based on the notion that living with your family should be a basic right) but freedom of movement for workers is still the most thorough (and freedom of movement is only about EU citizens; the rules about refugees and the border-free Schengen area fall under an entirely separate set of regulations).
The short-term political considerations are slightly different but also go in the same direction. For if Britain is able to extract significant concessions and gets to choose which rules it will continue to implement, why deny that to other countries?
Freedom of movement for persons is a particular focal point in the UK debate at the moment (but also implicitly free movement of goods, for that's what all the seemingly absurd rules about pillows, vacuum cleaners, bananas, cucumbers and the rest are about) but elsewhere it will be something else (including the freedom of movement for services – cf. the clichéed “Polish plumber” or the complaints about lorry drivers from Central and Eastern Europe – or capital – cf. discussion about tax deals with large companies in Luxembourg or Ireland).
The risk is that we would soon end up with a fragmented market where each country adds friction and bureaucratic requirements in different areas based on its current and changing priorities, essentially rolling back the EU single market. The EU would then revert to the kind of free(ish) trade that currently prevails between countries elsewhere in the world.
That would not necessarily be a catastrophe but would clearly mark the end of the single market and the EU as we know it so it's easy to see why countries, parties and politicians who are currently part of it and find it beneficial would not want to go down that road. The single market and freedom of movement (for persons and for other things) are very different in that respect from things like the euro or the Schengen area, which the UK could easily “opt-out” from in the past. Those are merely side-projects, not really tied in a very deep way to the existence of the EU itself.
Incidentally, it's interesting to note that the UK has always been a strong supporter of the single market idea against French push for centralised policy making à la Common Agricultural Policy. The European Union the British voters just rejected therefore owes much more to their country's than to the French or German vision of what it should be. Freedom of movement for persons is a case in point: The UK was very much in favour of enlargement and almost the only country not to put any restrictions on freedom of movement for citizens of the new member states during the 2004-2011 transitory period (although this was at least in part because it did not expect that many people to actually show up).