No.
There may be some new issues to contend with, but there are already issues to contend with, and the CTA has survived these. It should be possible for it to continue to survive after the UK leaves the EU.
The common travel area mainly concerns citizens of the UK and Ireland. There is a separate British/Irish visa scheme (BIVS) that allows visas issued by one to authorize travel to the other, but you still have to enter through the country issuing the visa, and not all British and Irish visas are issued under the BIVS. For example, it began only in 2014, applicable only to nationals of India and China.
So having different visa policies in different parts of the CTA will be nothing new: they already have different visa policies. For example, citizens of Bolivia require a visa in the UK, but not in Ireland, while citizens of Namibia require a visa in Ireland, but not in the UK.
It can therefore be inferred that a difference in visa policy for EU/EEA/Swiss nationals will not by itself spell the end of the CTA.
If this happened and the British government gave the Irish migrants the right to travel freely, this would conflict with the EU as Ireland is part of the EU.
If Britain leaves the EU, it will not be obligated to treat all EU citizens identically because of their EU citizenship. Britain already gives Irish citizens the benefit of a more relaxed deportation procedure compared to other EU/EEA/Swiss citizens. From the Wikipedia article on the CTA:
In February 2007 the British government announced that a specially lenient procedure would apply to the deportation of Irish citizens compared to the procedure for other European Economic Area nationals. As a result, Irish nationals are not routinely considered for deportation from the UK when they are released from prison.
In addition, there are examples of other countries such as Canada and the United States giving to citizens of different EU countries different access to visa-free travel regimes. The EU continually discusses retaliation for doing this, in the form of requiring citizens of those countries to get visas to enter the Schengen area, but they have not yet done so. See, for example, http://www.euintheus.org/what-we-do/policy-areas/freedom-security-and-justice/visa-waiver/.
It is possible that they would impose such retaliation on a post-EU United Kingdom, but the UK might accept that and continue to give preferential treatment to Irish citizens. (Far more likely: the UK keeps visa-free travel for some EU countries such as France and Germany, while requiring visas for citizens of other EU countries such as Poland and Bulgaria.)
Links:
