Before Serbia, NATO bombed the Serbs in Bosnia. And for that they even have an explainer, written in 1997.
The political basis for the Alliance's role in the former Yugoslavia was established at the North Atlantic Council meeting in Ministerial session in Oslo, in June 1992. At that time NATO Foreign Ministers announced their readiness to support, on a case by case basis, in accordance with their own procedures, peacekeeping activities under the responsibility of the Conference on Security and Cooperation in Europe (CSCE) - subsequently renamed the Organisation for Security and Cooperation in Europe (OSCE). This included making available Alliance resources and expertise for peacekeeping operations. [...]
In June 1993, NATO Foreign Ministers decided to offer protective air power for the United Nations Protection Force (UNPROFOR) in the performance of its overall mandate. In July, NATO aircraft began flying training missions for providing such close air support (CAS). On 10 and 11 April 1994, following a request from the UN Force Command, NATO aircraft, for the first time, provided close air support to protect UN personnel in Gorazde, a UN-designated Safe Area in Bosnia-Herzegovina.
So (according to themselves, at least) NATO acted to defend a UN mission, even though the UN is not part of NATO. You can call this mission creep if you like, but that's how they justified it.
Of course, most of those UN-flagged troops were from NATO countries anyway, but technically not in NATO countries' territory.
And Wikipedia also covers that:
The operation included the first combat engagement in NATO's history, a 28 February 1994 air battle over Banja Luka, and in April 1994, NATO aircraft first bombed ground targets in an operation near Goražde.
As for Serbia, there were no UN troops there to protect at the start. I think NATO simply extended R2P ("responsibly to protect") to the Kosovars. It's harder to find a political statement similar to the above, simply because they have a lot of materials on the NATO site about the Kosovo campaign, but most of it is operational information.
But perhaps there's this history overview of relevance
It was agreed, in addition, that the Organisation for Security and Cooperation in Europe (OSCE) would establish a Kosovo Verification Mission (KVM) to observe compliance on the ground and that NATO would establish an aerial surveillance mission. The establishment of the two missions was endorsed by UN Security Council Resolution 1203. Several non-NATO nations that participate in Partnership for Peace (PfP) agreed to contribute to the surveillance mission organised by NATO.
In support of the OSCE, the Alliance established a special military task force to assist with the emergency evacuation of members of the KVM, if renewed conflict should put them at risk. This task force was deployed in the former Yugoslav Republic of Macedonia (1) under the overall direction of NATO's Supreme Allied Commander Europe.
So (assuming that is correct) they did manage to convince Russia & China at UNSC that a NATO "aerial surveillance mission" is okay. Of course, Russia and China would not agree to strikes.
And then
On 20 March, the OSCE Kosovo Verification Mission was withdrawn from the region, having faced obstruction from Serbian forces to the extent that they could no longer continue to fulfil their task. US Ambassador Holbrooke then flew to Belgrade, in a final attempt to persuade President Milosevic to stop attacks on the Kosovar Albanians or face imminent NATO air strikes.
As ex-post-facto justification, after being bombed, Serbia agreed to admit peacekeepers, under a UNSC mandate (even chapter VII):
On 10 June the UN Security Council passed a resolution (UNSCR 1244) welcoming the acceptance by the Federal Republic of Yugoslavia of the principles on a political solution to the Kosovo crisis, including an immediate end to violence and a rapid withdrawal of its military, police and paramilitary forces. The Resolution, adopted by a vote of 14 in favour and none against, with one abstention (China), announced the Security Council's decision to deploy international civil and security presences in Kosovo, under United Nations auspices.
Acting under Chapter VII of the UN Charter, the Security Council also decided that the political solution to the crisis would be based on the general principles adopted on 6 May by the Foreign Ministers of the Group of Seven industrialised countries and the Russian Federation - the Group of 8 - and the principles contained in the paper presented in Belgrade by the President of Finland and the Special Representative of the Russian Federation which was accepted by the Government of the Federal Republic on 3 June. Both documents were included as annexes to the Resolution.
Following the adoption of UNSCR 1244, General Jackson, acting on the instructions of the North Atlantic Council, made immediate preparations for the rapid deployment of the security force (Operation Joint Guardian), mandated by the United Nations Security Council.
And from what I recall, Russia also agreed (then) because ostensibly Serbia agreed, leaving aside that they did that only after being bombed.
So, in that case there was a more complicated ballet of invoking UNSC authority, albeit the ex-post-facto one.
By 2002 NATO had various discussion papers on how it applied R2P etc. So, yeah, they extended the mission to R2P in non-NATO areas, albeit close enough to their "sphere of influence". The most relevant bit is perhaps bypassing UNSC, as a last resort
Right authority
There is no better or more appropriate body than the United Nations Security Council to authorise military intervention for human protection purposes. The task is not to find alternatives to the Security Council as a source of authority, but to make the Security Council work better than it has.
When it comes to authorising military intervention for human protection purposes, the argument is compelling that the United Nations, and in particular the Security Council, should be the first port of call. The difficult question - starkly raised by Kosovo - is whether it should be the last.
Note that this is not an official NATO document, even if hosted there.
Of course, a (or the) major reason they don't do that R2P in Ukraine, as NATO's outgoing military commander recently said is that Russia has nukes.
"I am absolutely sure if the Russians did not have nuclear weapons, we would have been in Ukraine, kicking them out," Admiral Rob Bauer, the outgoing chief of NATO's Military Committee, said during an appearance at the IISS Prague Defence Summit in the Czech Republic on Sunday.
By the by, NATO is not the only defensive organization that managed to mission creep. The CSTO moved into providing internal security for their member countries against opposition groups etc., while invoking a threat from (rather fictitious) "external forces" and "collective peacekeeping". That's not too far from how the Warsaw Pact functioned in practice. But I guess one can at least argue that those mission creeps were just "on their own territory". Of course, with a mission creep of that variety, the proposal to leave such an alliance becomes rather chancy in itself.