There's a bit more background to this. Some of it is easily verifiable from other sources, but some is the view of Ivan Rogers who was the
British Permanent Representative to the EU (2013-2017) and was previously Adviser
for Europe and Global Issues to Cameron himself (2012-2013). Briefly:
So basically both internal and external factors caused Cameron to harden his stance on the EU. Externally, he seems to have intended to use the referendum threat as bargaining chip with the EU, as he apparently informed them sometime before the speech that he still intended to pursue EU reforms and a full in-out referendum (the public details on these pre-speech talks with EU about the referendum are scant though). Internally, it's pretty clear that he thought to shore up his Eurosceptic credentials after the October defeat.
Cameron himself promised, when Leader of the Opposition, that there would be such a referendum [on the Lisbon Treaty], if the Treaty had not already been ratified by all Member States before he took office. By the time he took office, it HAD been universally ratified, including here.
Plenty in his own party ignored the phrase “if it had not already been ratified” and wanted him to reopen the whole issue post ratification. He did not. Could not. But what he did do was to legislate domestically in the European Union Act of 2011 to put an effective lock on any further UK commitment to any further steps towards political integration.
He clearly hoped, via that Act, to calm the party’s post Lisbon Treaty frenzy, and thereby to persuade it to “stop banging on about Europe,” as he put it in first first Party Conference speech as Conservative leader.
His own view [at the time], often expressed to his close team, was that very few of the great domestic issues with which he wanted the Coalition government to grapple were much impacted by the EU, and that the public rarely viewed the EU question as amongst its central preoccupations. [...]
Yet that stance was, within 18 months of Cameron’s taking office, effectively in tatters. [The reason was] In a phrase: the eurozone crisis. [...]
in Westminster, the passage of legislation specifically and solely designed to cauterise eurozone wounds and address eurozone needs, inflamed Conservative opinion. The mood amongst many of the backbenchers was that the eurozone was the proverbial burning building with no exits, might be nearing collapse, needed our urgent assistance in preventing collapse, and needed Treaty changes in its own interests. So, no better time to name our price for Treaty changes of our own. Many were gunning for an in-out referendum right away. They were not talking much about free movement, borders or the European Court of Justice then. It was about divorce from the burning building.
[...] Cameron made clear that he was now no longer prepared to allow the 26 (as it then was) again to amend the Treaties to enable the changes now required by the eurozone members, unless, simultaneously, amendments were agreed to the Treaties which gave the UK satisfaction on the key anxieties he and George Osborne had about the relationship between the then 17 eurozone members and the then 10 non-members.
But the Cameron attempt to lever open the negotiation got nowhere. Berlin and Paris essentially went round him. A couple of days before the December European Council, Angela Merkel and Nicolas Sarkozy, at a European People’s Party Heads meeting at Marseilles decided to circumvent a potential British blockade of the amendment of the Treaties by doing the deal at 17 via a partly intergovernmental Treaty process, which would not require UK ratification. The resulting Fiscal Compact is now known as the Fiscal Stability Treaty. [...]
To others, notably Berlin and Paris, the UK’s proposed Protocol looked like the UK opportunistically exploiting an existential crisis of the eurozone merely to further long standing British interests. The resentment was deep, and the mood sour.
To Cameron, though, the eurozone members’ manoeuvre was the eurozone privileging its own urgent institutional reforms over his.
[...]
I do not know precisely when David Cameron decided when he would offer the public an in or out referendum. We were discussing it with him well into the summer of 2012, around G8, G20 and EU Banking Union summits, but it was gestating before that.
He personally deeply disliked the Lisbon Treaty settlement, and its institutional implications. [...] His first attempt to address those issues head on [during the Eurozone crisis] had not cracked it.
And he now knew that he would never get agreement to a de facto British veto on [EU institutional] developments, by returning issues from qualified majority to unanimity voting. He also knew that others had found a way to deliver the requisite short medium term changes to the eurozone without opening up the Treaties for what he thought the UK needed.
[...] For all that this held little interest for the public and the vast bulk of the media, in Cameron’s view this had become the central issue of our membership which later led him to the proposal, made in his Bloomberg speech a year later in January 2013, to attempt to renegotiate UK membership terms, to be followed by an in-out referendum. My contention here is that his thinking about the key permanent changes he needed, was set far earlier than other commentators have suggested, and was determined above all by his experiences in December 2011 [the rejection of his demands during the Eurozone crisis].
Cameron suffered a stinging defeat on the floor of the House in late October 2012, when his proposed stance for the impending EU Budget negotiations—which was to advocate a real terms freeze, far below the proposal on the EU leaders’ table—was rejected by a combination of the Labour opposition led by Ed Miliband, and 53 Tory rebels, marshalled by, amongst others, players who defected to Ukip within 18 months.
[...] By the time the Bloomberg speech was finally delivered in January 2013, the main contents had been well trailed with other EU leaders, and they knew that the key commitment to an in-out referendum was the central component. They also knew that Cameron would seek what he termed a “new settlement” before putting the issue to the people.
[...] Others calculated, in classic European fashion, that no leader would do what Cameron duly did: go down in flames in a [referenudum] vote rather than acquiesce and find a price. His calculation, on the contrary was that to do anything else would send precisely the wrong signal for the renegotiation ahead.
