Actually, Swiss banks are far from unaccountable nowadays, smart people started moving their money to places like Singapore 10 years ago.
Since the mid-1990s, the mood has changed and some provinces (canton/Kanton) elected public prosecutors like Bernard Bertossa much more willing to satisfy requests coming from abroad. Switzerland's export industry (manufacture and services) is also substantial and tightly integrated with its neighbours so it had to yield to external pressure – especially through the OECD – in a way that Liechtenstein or Caribbean states haven't.
One key factor in all this is that tax evasion is a two-way street: Switzerland is profiting by channeling wealth that only exists thanks to its larger neighbours' economic activity through its banks but the sportsmen, managers, heirs and business owners who put their money in Switzerland also profit from the arrangement and they are far from powerless in their country of origin.
And there are many states which are in a kind of “grey area”, which do not have a reputation as full-blown tax and bank haven but do have a few exotic exemptions. Think about Belgium and personal taxes for people who made their fortune abroad or the Netherlands and its corporate tax exemptions (the so-called Dutch sandwich). They stand to gain if their neighbour would close one loophole but they don't want to give up their own special niche. That makes finding an agreement much more difficult and impairs any concerted effort at putting pressure on countries like Switzerland or fighting anything but the most egregious abuse.
So whenever there is an effort to push back against tax havens, there are always very long discussion about who and what to target. Case in point, the OECD created a list of uncooperative territories back in 2000. Some members wanted to include Switzerland in that list but the OECD always stopped short of that.
Incidentally, the EU and some member states like Germany or France have also long been complaining about the lack of cooperation but it has remained relatively powerless in this area, not only because Switzerland is not a member but also because Luxembourg is and was playing a lot of the same games, while having the ability to block or slow down EU action in the area. This is another reason why the OECD has taken the lead.