Further, the map on Wikipedia which accompanies the list seems to show some areas not colored in, but it's unclear why (i.e. there is no legend entry for these).
It helps to compare the map on the linked Wikipedia page to the Wikipedia map on other UN treaties such as that of the Antarctica Treaty System. Both maps have land areas that are colored gray. These gray areas are nearly identical in the two maps; the two exceptions are Greenland and the Falkland Islands (Las Malvinas), which are gray in the Antarctica Treaty System map but are colored in the UNFCCC map.
The areas that are covered gray on both maps include Antarctica and islands in the immediate vicinity of Antarctica. The Antarctica treaty bans the colonization of Antarctica. Also covered in gray on both are few islands in the far south that are not covered by the Antarctica Treaty such as the South Georgia Islands (south Atlantic Ocean) and the Kerguelen Islands (south Indian Ocean). Except for a few military personnel and scientists, nobody lives there, and nobody has taken up permanent residence. None these count as countries.
There are four gray-colored areas that might count as such, none of which have signed the UNFCCC:
Mainland China would object massively, with veto power, if Taiwan (the Republic of China) signed any UN-based agreement.
- New Caledonia
There have been three Brexit-style referenda in New Caledonia regarding its status as a largely autonomous French territory versus being an independent country. The "stay" vote won by a small majority in the first two referenda. The "stay" vote won by a massive majority (96%) in the third referenda, but that's because the "leave" coalition wanted the third (and last) referendum postponed due to COVID-19. It wasn't postponed, so the "leave" coalition boycotted the election.
- Western Sahara
This disputed territory is at war with itself and with Morocco. Some countries recognize Western Sahara is an Independent nation. The UN does not. The UN instead acknowledges that Western Sahara is a trouble spot.
- Puerto Rico
Sentiment is divided in Puerto Rico regarding whether it should become an independent nation, become a new state of the United States, or remain a semi-autonomous commonwealth but with no representation in the US federal government.
Which countries or territories are not parties to the UNFCCC?
None of the four listed above (Taiwan, New Caledonia, Western Sahara, and Puerto Rico) are parties to the UNFCCC. I'll let the OP decide which, if any of those count. In addition to these four gray colored regions that might or might not count as a country, there are several other regions that are not colored gray in that Wikipedia map but arguably should have been. These include Kosovo, South Ossetia, Abkhazia, and Northern Cyprus, none of which have signed on the be a party to the UNFCCC.