The problem with supporting violence against an ideological group, lies in the nature of society and the unspoken rules that are required for it to function.
At the most basic level, for a society that is larger than the number of people that one can verify as trustworthy (e.g. everything larger than a familiy structure), it is required that people show a basic amount of trust towards strangers.
More specifically, for easy and quick interaction with "strangers", society as a whole needs to have the understanding, that they are safe from arbitrary violence (put simpler: people must believe that they won't get killed by a random stranger when walking down the street).
( more on that topic: http://hbanaszak.mjr.uw.edu.pl/TempTxt/Newton_2001_TrustSocialCapitalCivilSocietyAndDemocracy.pdf )
In order to facilitate this, most states create a monopoly on violence.
At least for now, a society completely without violence doesn't exist. States offer the next best thing. They strongly regulate who can administer violence when and where (accordingly: the police, everywhere, when it serves to prevent a crime).
Which brings us back to your question:
If violence against a ideological group is allowed, that introduces a large amount of variance (Who is considered part of the group? Will other groups be targeted as well? Who will carry out the violence?), which hurts the trust in the state/societies ability to provide safety.
Instead of trusting in a single well defined ruleset and the organization that carries it out, people would have to trust that in the viewpoint of every individual they have contact with, they aren't egligible for punishment.
Slight hyperbole: People would stop interacting with each other out of fear. Society colapses. The End.