Your question is really rather a complicated one, to which there isn't going to be any kind of simple answer. However let's try to adderess some of the issues as best as possible.
The first thing to note is that 'broken window theory' isn't mainly about policing. It's at least as much about fixing the broken windows if and when they are broken as it is about prosecuting window breakers. When 'broken window theory' is put into practice it is often as much about removing graffiti and cleaning up public spaces as it is about about the police.
You aren't very clear about your definition of 'social policing', but you might also like to include police activities that are not technically about law enforcement - school visits by police officers, and other activities aimed at establishing good relations between police and the public. Most police forces carry out those activities to some extent.
You don't give a clear definition of "offenses against social order" and how they differ from "criminality". Democracies that want to prevent such offenses generally do it by encoding such offenses into legislation. So if society wants to prevent people from littering, they pass a law preventing littering (making it 'criminal') rather than just prosecute litterers without a law.
Having said that, most jurisdictions have statutes that prohibit unacceptable social behaviour, often in ill-defined terms. Such laws are often called things like 'public nuisance'. Rarely is there a strict definition of what constitutes 'public nuisance', and the behaviours that can be prosecuted under it can vary with time, as what society accepts can change. 'Obscenity' is another catch-all whose limits can change as the general public view of what is acceptable changes.
In both of the above cases prosecution of offenders is done in exactly the same way as for any other offence, and through the same organizations.
You might also consider the example of the Antisocial Behaviour Order (ASBO) in the UK. This is a law that explicitly targets this kind of behaviour, but is codefied in the law (technically civil law rather than criminal in this case).
tl;dr
- 'Broken windows theory' is about much more than policing.
- Most jurisdictions can and do prosecute many kinds of socially unacceptable behviour under the criminal code
- Social unacceptability is frequently written into the criminal code
- Democracies don't usually have specific law enforcement for 'social' offenses. The job is done by regular police.
