The voting threshold necessary to prompt the exit process was never actually decided at all as such prior to the referendum, but assumptions were made by politicians and journalists about what would be politically acceptable to an emergent notion of popular sovereignty. The law establishing the referendum did not specify any resulting action or level of vote necessary for action, and nor did any secondary regulations, nor the legislation for referendums in general.
The nearest Parliament seems to have got to thinking about the matter was in rejecting the SNP's amendment that would require a supermajority or 'quad lock', whereby a Scottish vote to leave (which in the event did not happen) was needed for action by the UK. The debate as a whole seems to indicate that little thought was given to the basis for a referendum within an unwritten constitution. This lack of clarity contributed after the event to more than four million people signing a petition for a second referendum requiring a clearer majority, and alternative suggestions by Geoffrey Robertson QC that MPs should overturn the perceived result: "Democracy has never meant the tyranny of the simple majority".
(However, there has also been some dispute about whether MPs technically even need to vote on exit - possibly against their own opinion - before the executive triggers the irreversible Article 50, which itself refers to the UK's 'own constitutional requirements', unclear as they are. It has been suggested that there are moral reasons for MPs to reflect the simple majority outcome, but Dr Yossi Nehushtan says it is "morally-politically inconceivable to treat a 52% majority decision in a referendum as authoritative with relation to constitutional principles".)
So where does the legitimacy of a 50% threshold come from? The Conservative manifesto 2015 says
We will negotiate new rules with the EU [specifically about benefit entitlement]... We will
then put these changes to the British people in a straight in-out
referendum on our membership of the European Union by the end of 2017.
'Straight in-out referendum' implies a referendum with two choices; this was designed to appeal to voters who liked the sound of 'in' as well as 'out'. It doesn't say anything about threshold, but it could be argued that it gives equivalence to each outcome, rather than recognising one of them as a more significant constitutional change. (Only Conservative MPs could be held to the wording in their manifesto.) A month before the referendum, Prime Minister David Cameron said "Obviously a referendum is based on a simple majority", but as the existence of the question implies this is far from obvious. It does seem that most MPs had made the same assumption by this point, and the question is why. Previous referendums, such as the Scotland devolution referendum 1979 had a threshold of 40% of eligible voters, which it failed to meet despite meeting a simple majority.
One likely influence was the most recent UK-wide referendum, on Alternative Vote. Although clearly a fundamental constitutional change, the Liberal Democrat party negotiators wanted a definite prospect of electoral reform and to be seen as equal partners in it. The coalition agreement therefore stated:
Both parties will whip their Parliamentary Parties in both Houses to
support a simple majority referendum on the Alternative Vote (emphasis added)
This subsequently entered law as s8 of the ensuing Act which unlike the EU referendum did clearly specify how the counted votes were to be interpreted, and may have set a precedent in many MPs' minds. Also note that the Conservative party tends to oppose any constitutional change, and have a fixed idea of decision-making procedure and little time to spend on STV or modern electoral systems. Therefore when the Prime Minister of the time said 'obviously' it was not merely rhetoric, but reveals a mindset formed in an environment not familiar or patient with constitutional reasoning and most concerned with precedents within his terms of office.
There is also the question of whether leaving the EU is a constitutional matter (where a civil society organisation would typically require a two-third majority). I've heard it claimed (by a Leave supporter) that Remain supporters deny that the EU membership affects national sovereignty and therefore it is not a constitutional matter. On the face of it, the way EU Regulations enter force (as part of an evolving trade treaty) without the intervention of the UK parliament, obtaining some legitimacy from election of MEPs, is a part of the current constitution. A professor of law at QMUL has no doubt the decision to leave is a constitutional issue, and mentions the possibility of a second referendum on the terms of an exit, although it's not clear what would happen if the terms are rejected and Article 50 has been triggered.
So we're left with a potential conflict between Parliamentary sovereignty on the one hand and on the other a 51.9% majority of voters on a higher turnout than the last general election. To some minds, avoiding that conflict to maintain the legitimacy of the political system is a more important moral imperative than anything else, which would have been a good argument for a sufficient threshold to ensure Parliament was aligned with the 'result'. Nat le Roux has written one of the best summaries of the ramifications of 'paradoxes of legitimacy' caused by advisory referendums. The AV referendum was binding, not advisory, but had a 50% threshold. The EU referendum was advisory, not binding with an undefined threshold, assumed to be 50%, which is 'potentially destabilising'. The idea of the 'will of the people' is used by Conservative Leave MPs to strengthen their position: they may believe Leave stabilises the Conservative party itself, which after all was one purpose of the referendum.
tl;dr: cock-up.
