The concept of Brexit is worrying for a number of reasons:
Impact on European Union
Legislative
EU becomes hard to justify if you set the precedent of a country leaving altogether
Territorial integrity of current members becomes hard to justify if you allow for an EU system with such a broad degree of acceptance (see Catalonia, Cyprus)
Economic
- The cornerstone achievement of the EU is the single market, incomplete as it may well be. Reversing a country out of this de-facto free trade agreement results in a loss of consumer and producer surplus for trade with the UK
Ideological
- European project is born out of the ashes of World War II, reversing the course was never the intent of the founding fathers
Impact on the UK
Most of the above, just from a UK perspective
Loss of investment confidence in a country that removes itself from the single market (City of London is the most worried - bankers don't need access to an 80m people market they need access to 500m people)
The argument made from the Unionist side tends to lean towards stressing the negative downside of the UK leaving. There are many good upsides of the UK being part of the project that have yet to be stressed and enumerated to make a more rounded campaign.
I personally don't think that the EU-crat upside is a widely shared belief. Yes, technically speaking you may remove a barrier to further European integration if you get rid of its most reluctant member. But imagine Lincoln had made the same argument on Southern secession instead of pushing the cultural norm of Unionism further, albeit unfortunately through violence. I see the UK debate separate from deeper Euro zone integration talks and even the concept of uniting the EZ into a sovereign state is removed from membership of individual states in the supranational organization of the EU.
The bottom line is that the EU is unwieldy, inefficient and probably in dire need of reform (in my personal opinion, needs federalization to work).
However, today it stands as layers and layers of treaties setting forth of some of the broadest, constructive and integrative international cooperation the European continent has ever seen. Removing oneself from this progress is to slide into the comforting temptations of nationalist exceptionalism that has no place in a community of nations that twice brought the world to war and has since forged the largest economy in the world through meaningful nation-building.