Back during the German elections, I remember a bunch of parties saying they would be unwilling to form a coalition with AfD. That made me wonder--what if AfD had won the most votes without getting a majority, but no party was willing to form a coalition with them? Could some number of the minority parties that would add up to a majority form a coalition despite none of them getting a plurality of seats? Similarly, such a scenario could happen in the UK right now, though it would require an unlikely (?) coalition of everyone against the conservatives.
The general requirement for a government in a parliamentary system is that they can win motions of confidence. Even if one party holds a plurality in parliament, if it cannot win a motion of confidence it cannot form a government. Contrariwise, if two or more minority parties agree to work together, they can form a majority in parliament that will win a confidence motion.
However, coalitions that form only in opposition to a third force tend to be unstable and may not be able to agree on a legislative program or consistent economic strategy.
By way of example, in Spain, the current government is lead by the PSOE and their allies. But the People's Party hold a plurality (but not a majority) in parliament. The PP lost power following a confidence vote earlier this year.
In the particular case of the UK, following a general election, the largest party would normally be allowed to make the initial attempt to form a government, and a coalition (or minority government) of minority parties would be considered only after the party with the plurality had failed.
After a no-confidence vote in the PM in parliament, there would then be 14 days for the various other parties to attempt to form a government before a general election is called.
Quite simply yes… and it actually happened in Germany before, between 1969 and 1972 (first Brandt cabinet) and between 1976 and 1982 (second and third Schmidt cabinets). In both cases, the CDU/CSU was still the largest group in parliament but it did not have an outright majority without the FDP, which chose to form a coalition with the SPD instead.