The other answers have explained that the problem with conventional nuclear weapons is the step up of "going nuclear" being disproportionate. However, nuclear weapons need not be high powered.
In the case of Japan, low and high power nuclear weapons alone would probably not be a sufficient independent defence, because the seas are a significant resource, notably for food, along with a small land area not able to sustain tit-for-tat attacks that are described below. Theoretically if your country was large enough, then low level nuclear tactics could provide a deterrent. The demonstration of the ability, and willingness, to perform low-level nuclear attacks on enemy countries could be sufficient deterrent to prevent attack.
At their lowest power, nuclear weapons can target individuals/families: for example a Russian dissident was poisoned with nuclear materials smuggled into London.
These could be delivered by agents, and/or by proxies, such as domestic terrorists.
Larger levels of nuclear contaminants can be used to poison food/water sources, fuel distribution networks, places of cultural/religious/ecological significance, or urban areas. Dirty bombs with varying levels of radioactive contaminants could delivered by agents or by rockets.
EDIT:
As I have already mentioned, and commentators didn't understand, these tactics would not be effective for a country with a small land area because of the inability to sustain tit-for-tat exchanges which I foresaw might result when the origin of attacks was discovered, another thing the commentators misunderstood. Indeed the origin of these measures used for defence would be immediately obvious since they would be deployed as retaliation for a previous attack.
There has been no military retaliation for the Russian poisonings on British soil. That is because Britain realizes it would come off worse in tit-for-tat exchanges with Russia.
Considering defence only, it is possible to conceive of a large country with sufficient resources being able to conduct its defence with low and high powered nuclear weapons. The threat of low level nuclear contaminants landing on enemy soil would be sufficient deterrent to attackers. In the light of one comment I have removed the part of my answer about fomenting dissent because, as one commenter quite rightly noted, the key concept of defence is having graduated deterrent response to attack.