Your suspicions are correct in at least some cases - the withdrawal of recognition seems to mostly be as a result of Moroccan diplomatic and economic pressure. In Morocco and the African Union - Prospects for Re-engagement and Progress on the Western Sahara, McNamee et. al note the consequences for South Africa recognising SADR:
After 1984, Morocco expended a huge amount of diplomatic and financial
muscle on trying to get SADR out of the OAU.
[...]
On 15 September 2004, just over two years since the AU was founded,
Moroccan officials were left reeling after the ‘shock’ recognition of
SADR by South Africa.
[...]
Morocco swiftly withdrew its ambassador to Pretoria and condemned
South Africa’s ‘partial, stunning and ill-timed’ decision, noting in
its official response ‘the pioneer support it has continuously
provided to the legitimate struggle of the South African people for
its dignity and freedom,’ referring to the ANC’s struggle against
apartheid.
South Africa has not withdrawn its recognition, but this is hopefully illustrative of the sort of pressures Morocco puts on states that formally recognise SADR. This isn't, however, the only reason that may lead states to withdraw recognition; the case of Kenya is discussed:
Kenya, for example, suspended its decision to recognize SADR in 2006,
apparently so it could act as a mediating party.
Kenya, however, re-recognised SADR in 2014.
Examining the most recent cases of 'de-recognition', South Sudan withdrew recognition in 2018 after a meeting between the Moroccan and South Sudanese foreign ministers. El Salvador took a similar stance in 2019 during a visit by the Moroccan foreign minister to the country, the foreign minister of El Salvador stating in a tweet that:
It is the right decision for El Salvador. We are no longer recognizing
a country that does not exist, we are strengthening our ties with the
Kingdom of Morocco and we are opening the doors to the Arab world,
This last point probably refers to the fact that the Arab League also supports the Moroccan claim to the territory. Pressure from the League as a whole is also likely to be a factor.
On the other hand, Haiti withdrew recognition in 2013 ostensibly to "encourage the settlement of the issue by the UNSC and support efforts made by the UNSG and his personal envoy to reach a final, mutually-acceptable political solution by the parties".
Jamaica also took a similar line when it withdrew its recognition in 2016, the Minister of Foreign Affairs and Foreign Trade stating in 2019 that "Jamaica maintains the position it took in 2016 and reiterates its decision to withdraw its recognition of the self-proclaimed 'SADR,'. We reiterate our support for the political process underway aimed at reaching a realistic and pragmatic solution."
So broadly, there seem to be two reasons for withdrawal of recognition; firstly, pressure from Morocco and the Arab League as can be inferred either by withdrawals occurring after meetings with Moroccan officials, or subsequent support for the Moroccan claim, and secondly, a desire to appear neutral and support the process for the settlement of the issue through the United Nations.
Of course, this second reason could be used as a way of saving face and not being seen to submit to Morocco's demands, but there is unlikely to be evidence of this. In particular, it seems unlikely that Morocco could apply significant economic pressure on a state such as India, which withdrew recognition in 2000.
Finally, it should also be noted that according to McNamee et al., the 'de-recognition' of a state is fairly weak:
But under the AU Charter, just like the UN Charter, there is no
provision for de-recognition. In technical and legal terms, therefore,
it is immaterial how many African countries derecognise SADR.
