According to mutually assured destruction, you lose your insurance that other countries won't nuke you
A commonly cited reason is the concept of mutually assured destruction (MAD), which is similar to the prisoner's dilemma. From Wikipedia:
The MAD doctrine assumes that each side has enough nuclear weaponry to destroy the other side and that either side, if attacked for any reason by the other, would retaliate without fail with equal or greater force. The expected result is an immediate, irreversible escalation of hostilities resulting in both combatants' mutual, total, and assured destruction. [...]
The doctrine further assumes that neither side will dare to launch a first strike because the other side would launch on warning (also called fail-deadly) or with surviving forces (a second strike), resulting in unacceptable losses for both parties. The pay-off of the MAD doctrine was and still is expected to be a tense but stable global peace.
In other words, if one country with nuclear weapons launched them at another country with nuclear weapons, the other would retaliate and the end result is that both countries would be destroyed. Since having your country be destroyed is not in your interests, that means that nuclear weapons don't get fired.
So now let's say that you are a country who is not on good terms with another country. If you remove your nuclear weapons, then other countries can fire nuclear weapons at you without fear of their country being destroyed. Thus a country following MAD may decide that removing their nuclear weapons is too big of a risk if other countries are allowed to keep theirs.
The converse is that if another country removes their nuclear weapons, but you keep yours, then you have an advantage over them because you can threaten to nuke them without fear of your own country being destroyed, so those who have nuclear weapons tend to want to keep them.
So why don't all countries just agree together to not use nuclear weapons?
That's exactly the point of the Treaty on the Non-Proliferation of Nuclear Weapons (NPT), in which countries that had nuclear weapons before 1967 (referred to as nuclear-weapon states) pledge not to expand their use, and countries that did not have nuclear weapons before that time pledge not to create them.
As of this writing, the NPT has been signed by 191 nation states. There are only five UN-recognized countries that have not signed it, and they include the only four countries that have nuclear weapons, but are not nuclear-weapon states recognized by this treaty. All of these countries declined to sign the NPT, except for North Korea which signed it, but withdrew in 2003.
The book Avoiding the Tipping Point notes that while NPT hasn't removed all the nuclear weapons of the world, it has drastically limited their spread. At the time of the NPT's creation in 1968, there were estimates that within twenty years, 25-30 countries would have nuclear weapons. Today, only the four countries beyond the initial nuclear-weapon states have nuclear weapons.
A stronger version of this treaty, the Treaty on the Prohibition of Nuclear Weapons (a.k.a. Nuclear Weapon Ban Treaty), does provide a legally binding commitment against the stockpiling or use of nuclear weapons. Countries with nuclear weapons that choose to sign it are given a timetable to completely disarm their nuclear weapons. It was passed on 7 July 2017 and will be open for signatures on 20 September 2017. It will enter legal force 90 days after the 50th country signs it.
However, all current nuclear-weapon states and most NATO members (along with US allies Japan and Australia) have indicated that they do not plan to sign the Nuclear Weapon Ban Treaty, with many stating it is because they believe that having nuclear weapons has maintained peace in the region for over fifty years due to MAD.