(This answer focuses on only the India aspects of this question. Individually, other countries in SE Asia have less people, with smaller internal markets and also have different strengths/weaknesses)
What makes it difficult for Western countries to move all (or most of) their factories from China to Southeastern Asia
This seems to be genuinely misunderstanding China's role and position, with regards to manufacturing.
China is not just acting as a cheap labor pool and subcontractor for "smart Western companies".
It has a highly educated workforce and a whole network and subnetwork of highly competent manufacturers that depend on each other and, besides copying and pirating IP willy-nilly, has long reached the point where it is actually quite innovative in many fields, with significant R&D. Large Chinese companies have the capital to innovate on their own, without being given orders by "the world".
(This doesn't apply to all fields, China is still struggling with jet engines or microchip fab technology, for example.).
China has 5x the GDP of India and a quickly growing middle class meaning that it has a large internal market for its manufacturers to sell stuff to. This lure of a big market is also what got many Western companies to invest in China, despite various concerns.
It is also a highly unequal society, but many poor rural workers do have the skills necessary to work in high/medium tech factories for low wages (not all, remember that China's 2017 per capita GDP is about Costa Rica's - there are lots of dirt poor Chinese farmers):
Now, some of these aspects do apply to India, as far as Information Technology goes. They don't seem to apply in manufacturing however where the broad base seems lacking. India's rural poor, as a whole, also seem less educated and less capable to take on roles in advanced manufacturing.
China is at #66 on World corruption index, while India is at #85.
China is also ahead on "infrastructure ranking". It sits at 77.9%, very close to Australia, while India is at 68%, near South Africa.
Bottom line: it takes time and persistence to build a competitive advantage in any domain. That's true for finance (NY, London), films (Hollywood, Vancouver), IT (India), manufacturing (China, Japan, Korea). Low labor costs is only an ingredient, and not the most important one when you truly want to innovate.
Last, India's politics under Modi doesn't make it exactly the most reassuring place to invest in. Nor is impossible to see something caused by his brand of Muslim baiting developing into a public relations disaster in the future for firms which do invest there.
p.s. Note that this answer, and more specifically, the bit about Modi's politics, is in no way an endorsement about how China goes about doing things in terms of employment practices or human rights.