If an embassy decides to be troublesome with the host country, causing social issues, blaming people (such as celebrities), looking to bring people to court, and eventually try to start a war, what can be done about it?
I understand this question may be a bit vague - I don't have a concrete example, but I'm looking to learn "how far" can an embassy go, just because (as far as I know) embassies are considered foreign territory, and police can't enter embassies without permission, etc...
For the most part, embassies can only be established in a country with that country's permission (it gets more complicated if a country agrees to host an international organization). The host country can kick out any foreign diplomats at any time for any reason and does not have to explain its decision. It can also break off diplomatic relations entirely at any time for any reason, and again does not have to explain its decision. This normally will increase tensions, not decrease them.
Once this pattern of behaviour is clear, the Ambassador will be called for an interview with the Foreign Minister of the host state and told in no uncertain terms to stop this nonsense.
If they refuse to do so, diplomats who there is evidence against will be declared Persona Non Grata. The host country can do this at any time, for any reason they please, and it removes the diplomat's immunity to trial and punishment[1]. Diplomats who have been PNG'ed normally leave the country immediately, and are allowed to do so.
If expelling a few diplomats does not put a stop to the problem, then the host country will terminate diplomatic relations. At this point, "diplomats" from the guest country are given a chance to leave. If they do not, they are subject to arrest, trial and punishment like any other foreign criminal. The host country's police forces are likely to treat them as a priority, and if they try violence, to bring in the army to apply superior force.
An example of gross misconduct by diplomats can be found at Murder of Yvonne Fletcher. In that case, after a short siege, diplomatic relations were broken without any PNG'ing.
[1] Diplomats are not immune to being arrested. If they're engaged in a crime outside diplomatic premises they can be arrested like anyone else. However, once things have calmed down and they assert their diplomatic immunity, they cannot be held by the police, nor tried or punished. They are released into the custody of their ambassador, if they aren't immediately PNG'ed.
In addition to the answers by John and cpast, for practical purposes a lot might depend on:
One facet of what I have in mind here is diplomats engaging with civil society in their host country. In the more repressive states, that is certainly seen as "subversive" or "causing issues." Yet when the host countriy has publicly subscribed to the Helsinki Accords, including section VII, or something similar, they must be circumspect in how they complain.
A more sinister scenario is if a foreign government e.g. wants to engineer a coup, and the host government does not find it politic to throw their embassy out. Think back to Iran in the 1950s.
Somewhere in between is the case of diplomats being undiplomatic. A recent example is the current Ukrainian ambassador to Germany, Andriy Melnyk. He is getting rotated to another position, but the final straw might well have been the fight he picked with Poland. His conduct was somewhat effective as long as both the German and European public considered the German government too hesitant -- but he became increasingly unable to work work effectively with the host government, except through TV talk shows.
