What is the decisive point for classifying a certain speech as unacceptable?
Even in democratic countries there are many types of speech which are illegal, not only hate speech. From Wikipedia:
Freedom of speech and expression, therefore, may not be recognized as being absolute, and common limitations or boundaries to freedom of speech relate to libel, slander, obscenity, pornography, sedition, incitement, fighting words, classified information, copyright violation, trade secrets, food labeling, non-disclosure agreements, the right to privacy, dignity, the right to be forgotten, public security, and perjury. Justifications for such include the harm principle, proposed by John Stuart Mill in On Liberty, which suggests that: "the only purpose for which power can be rightfully exercised over any member of a civilized community, against his will, is to prevent harm to others."
Of course different countries apply all or some of these exceptions to different degrees. As mentioned above, the general principle which guides these exceptions is the harm principle: if a speech is likely to cause harm to somebody, then it might not be covered by freedom of speech. The interpretations about what kind of harm and which level of harm is the exact limit are up to the lawmakers and courts of law of a country.