There are plenty of examples in which a coup is staged by a paramilitary organisation:
- The group known variously as "Daesh" or ISIS controlled much of Syria.
- An association of Tuareg militia controlled northern Mali.
- The Houthi movement has control in Yemen.
Large countries are not immune:
- A communist paramilitary group expanded their control from some small regions of China in 1930 to take over the country by 1949.
What is unlikely is for an organised crime syndicate to formally take over a country:
- The syndicate doesn't have enough foot soldiers. Even a very poor country like Mali has a military with about 10000 people, and more would be expected to join in the case of war. Organised crime gangs are much smaller. Most Mafia "families" have fewer than 10 "made" men. There is little organisation above the level of "family" (no boss of bosses).
- You don't have local support. The Tuareg were able to take control in northern Mali, because they had support of a significant part of the local population. This is because they had an ideology (Islamist) that people could attach to. A mafia don't.
- There is no money in it. The mafia are all about making money, from protection, drugs, extortion. There's no money to be made in running a country as poor as Mali.
- Big countries won't allow it. The Tuareg revolution was put down by the French Military. Unless you already control the western countries, your action will be put down by the powerful militaries of the rich nations.
A mafia boss wants to be sipping wine in a Sicilian villa, knowing that he has corrupted the local and national governments and so is immune from prosecution and will get first bid on all government contracts. He doesn't want to be sitting in an office in Timbuktu, dealing with running a country himself.