Basically, modern border disputes are resolved by making a treaty and hoping for the best. As can be seen in the current Israel-Palestine conflict, modern border disputes are often hard to resolve.
Historically, border disputes would often be solved by wars or coercion. Looking at Wikipedia's list of historical territory disputes, you can see that many territorial disputes involve war or bargaining, and looking at the (very incomplete)* list of border conflicts, you can see that historical border conflicts were larger and deadlier.
However, in the age of the United Nations and nuclear weapons, large wars over (relatively) small pieces of land, as well as large wars in general, are unlikely to happen. This is because Chapter VII of the UN's charter allows it to call upon its members(basically, the entire world) to use any means, including armed conflict if necessary, to ensure global peace in response to acts of aggression. Because of the this, large scale wars are mostly(with some exceptions) not a viable way to end a border conflict anymore.
There are certainly modern examples of modern border conflicts being resolved peacefully, both by the UN and independently by the relevant countries. In 2010 Burkina Faso and Niger submitted their border dispute to the UN's International Court of Justice, which resulted in a peaceful exchange of disputed land after the two countries agreed with the ruling. The 1998 Good Friday Agreement settled the status of Northern Ireland without the UN. However, these agreements don't always go well. In 1997 Russia and Ukraine signed the optimistically named Russia-Ukraine Friendship Treaty, which recognized the inviolability of existing Ukrainian borders. In 2014, Russia violated those borders.
In summary, because large scale war is mostly not viable because of UN agreements and nuclear weapons, both sides of a border dispute must submit the dispute to the UN and agree on the UN's rulings, or just create their own agreement. However, such an agreement relies on both sides following the terms of the agreement, and not just ignoring the agreement at a later date. There is no real way to force a country to follow an agreement short of economic coercion(sanctions, blockades) or war.
*Unrelated to the question, but when Wikipedia says 'you can help [this list] by expanding it', do they want you to find more examples or create more examples...