What prevents countries from going this path?
Other countries. Namely, their military and economic influence. If other countries will fail to prevent this, no one else will (literally).
Leaving the UN isn't really an important part here:
Both of these countries are permanent members of the UN Security Council.
Resolving conflicts through a military force doesn't really disqualify the country from the membership in the UN. Your hypothetical country (let's call it Hyponia) doesn't need to leave the UN to pull this through. In fact, it would be unwise for it to leave the negotiating table during the whole process.
Here's your scenario:
... the newly formed nation rejects any legal continuity with its predecessor. ...
... a new country ... resolve[s] the long-standing dispute ... by force, which is condemned by everyone.
Then a new country could be formed, that announces its continuity with the original state and denies the wrong-doings of the second one, but fully enjoys the new de facto resolved dispute state.
I can see two possible outcomes:
No one buys this whole "regime-change/new country" thing. Neighboring countries get the support of the UN Security Council and they spend next year or so, carpet-bombing Hyponia (ISIS scenario), destroying it completely.
No one buys this whole "regime-change/new country" thing. Neighboring countries fail to get the support of the UN because Hyponia has a powerful enough military/allies to stand their ground (Kosovo scenario) and/or the whole region is a big mess no one really cares about (Somaliland scenario). No one intervenes, Hyponia enjoys status-quo with some lazily-applied economic sanctions.
As you can see, leaving the UN doesn't make Hyponia immune to foreign influence.
Either way, if Hyponia was already the member of UN it doesn't make much sense for it to leave the organization. Actually, it isn't even necessary to rename itself or go through regime change. It's much simpler for Hyponia to unilaterally break away from all of the peace treaties and declare a war with its neighbors. All without leaving the UN.