I think one of the best explanations for this comes in the preambulatory clauses of the Vienna Convention on Diplomatic Relations which codified diplomatic immunity, and which has been ratified by 192 states. It reads:
The States Parties to the present Convention,
Recalling that peoples
of all nations from ancient times have recognized the status of
diplomatic agents,
Having in mind the purposes and principles of the
Charter of the United Nations concerning the sovereign equality of
States, the maintenance of international peace and security, and the
promotion of friendly relations among nations,
Believing that an
international convention on diplomatic intercourse, privileges and
immunities would contribute to the development of friendly relations
among nations, irrespective of their differing constitutional and
social systems,
Realizing that the purpose of such privileges and
immunities is not to benefit individuals but to ensure the efficient
performance of the functions of diplomatic missions as representing
States,
Affirming that the rules of customary international law should
continue to govern questions not expressly regulated by the provisions
of the present Convention,
Have agreed as follows: ...
Diplomatic immunity is deemed necessary for the continuation of international diplomacy because it ensures a country's diplomats' safety from persecution by the authorities. You, do, however, raise a good point that this can often be taken advantage of - some examples that spring to mind include the estimated £116m of congestion charges owed to Transport for London by diplomats, and the death of Harry Dunn in a vehicle collision in 2019.
However, a blanket ban on the prosecution of diplomats precludes a situation where the diplomat's country might call into question the validity of charges against a diplomat, and also prevents countries manufacturing charges in the first place.
In addition, although the Vienna Convention ensures that diplomats are in some ways above the law, it doesn't make them completely immune. Immunity can be waived by the sending country (Article 32), and the host country can always expel diplomats by declaring them persona non grata (Article 9). In addition, there are exceptions to the immunity, for example, diplomats can be prosecuted with relation to any professional or commercial activity they have conducted outside of their official functions.
In general, though, diplomatic immunity is respected even in the most egregious cases of clear wrongdoing, as to do otherwise would be to invite other countries to retaliate with relation to one's own diplomats. It would also call into question the independence of the judiciary, potentially leading to an even worse breakdown in law and order than that caused by a few diplomats enjoying immunity from prosecution.