That's simply not how international law works. Even when a country's regular military (not a private military contractor) openly defies strong international norms, prosecution is not a likely outcome. Countries are not simply “held liable” in that way.
The UN is really best understood as a forum and a facilitator, with key technical functions (through the various agencies in the UN system). It doesn't “prosecute” countries or initiate much. At most, some UN bodies might issue reports or condemn some actions but that's about it.
There are, to be sure, some international courts but their role is different:
The International Court of Justice (ICJ) helps resolve disputes between countries but can only really be involved with the parties' consent (i.e. if two countries are seeking a third party to help with the issue, not when one country is seeking to indict another).
The International Criminal Court (ICC) can prosecute individuals (not states) and is also hugely dependent on countries for its investigative powers. Importantly, the US famously declined to participate in this system (with some nuances between administrations on the level of cooperation with the court).
The UN also set up various types of ad hoc courts (the International Criminal Tribunal for the former Yugoslavia, the International Criminal Tribunal for Rwanda, the Special Tribunal for Lebanon, the Extraordinary Chambers in the Courts of Cambodia, the Special Court for Sierra Leone…) which have a limited mandate negotiated beforehand. All these were formally created with the consent of the countries involved (sometimes under a bit of international pressure) as a way to find some more-or-less neutral institution that could defuse a sensitive situation, not as a tool to hold a non-cooperative country responsible for something.
Instead, what could be expected in a scenario like the one described in the question is that one or more countries would resort to the usual tools of international diplomacy from formal complaints to breaking diplomatic relations, imposing sanctions (with or without UN backing) all the way to military retaliation. There is nothing stopping a country from calling out another one for letting a rebel group or mercenaries operate freely on its territory. But UN member states (rather than the UN on its own) would have the major role in all this.