Why does Germany's traffic light coalition seem more likely than ever before?

Question

Even though the parties of the Germany's grand coalition - CDU/CSU and SPD regained majority support in this year's election, according to the commentators, the traffic light coalition has the greatest chance to form because the grand coalition has become unpopular. I wasn't able to find historical data to find when exactly this shift was supposed to happen, however according to the current surveys, the traffic light coalition has the biggest support and only 5 % of Germans see the grand coalition as their most favorable option.

Since the traffic light coalition has never formed to date at the federal level, it begs the question of what has changed. Some of the hypotheses that come to my mind:

1. The exit of Angela Merkel made the Union less appealing to other parties or their voters.
2. It's the simple rise in support of the non-grand coalition parties and a loss in support for the Union.
3. As the non-grand coalition parties somewhat shifted more to the mainstream, they have become more appealing to the grand coalition voters and parties.
4. People want the coalition to be distinctively left or right, so that they can vote non-extremist opposition in case their preferred party went sideways (see o.m.'s answer).
5. People want change - even the voters of the grand coalition parties sense a need for a "fresh air" or a need to dissolve power structures once in a while
6. People want change because the left and the right voters became more polarized and thus, the grand coalition would mean more friction and compromises.
7. People have disliked the grand coalition more than the representatives for a while, however it took some time for these preferences to reflect in the Bundestag.

Is there any evidence pointing to any of these (or other) hypotheses?

Answer 1

It is a gross simplification to order parties on a left-right axis, but when one does that it becomes

Linke - Grüne - SPD - FDP - Union (CDU/CSU) - AfD

Historically, the SPD and CDU were the largest parties, by quite a margin. They could be characterized as center-left and center-right, respectively. A coalition with just one of them would lean either to the left or to the right. A coalition with both of them is on neither side.

• A Grand Coalition may be a very good idea if a grand compromise needs to be forged on some fundamental issue. But having a Grand Coalition year after year means it comes down to the lowest common denominator and gridlock.
• Having a government coalition from the center-left to the center-right leaves an opposition consisting of the far-left and the far-right. They cannot form a new government together unless one side gets an outright majority, so what can a voter do if he or she wants change?

For this reason a prolonged Grand Coalition is considered unhealthy for the political culture, and moderately right and left coalitions being elected and un-elected in turn are considered healthy.

---

As mentioned, a gross simplification. The FDP does not fit neatly between SPD and CDU, and the Grüne have conservative aspects.

Answer 2

TLDR:
While there are other political reasons that speak against another formation of a Grand Coalition, it had already been unpopular after the election in 2017 so that if there were an attempt to form another Grand Coalition after this election, heads would roll... probably figuratively.

The Grand Coalition had already been unpopular after the last election in 2017, especially amongst SPD-voters, which is why the SPD had declared immediately afterwards that there would be no grand coalition.

The coalition negotiations that followed were extremely tedious:
The election had been held on the 24th of September. Since the SPD didn't want to form another Grand Coalition, CDU, Greens and FDP tried to form a Jamaika Coalition which failed on the 20th of November, when the FDP cancelled the talks.
The tedious part that followed consisted of the SPD party leadership trying to convince its base to jump on board of another Grand Coalition, which they barely managed to achieve on the 21st of January with 56.4 % voting in favor after rather contentious discussions.
The new government was formed on the 14th of March, roughly half a year after the election, making this a longest government formation in our history.

Answer 3

Note: When I refer to parties, I will be counting CDU/CSU as one except where I spell out CDU without CSU.

As usual, I believe the situation cannot be understood without knowing where we came from.

A brief historical overview

Until 2005 (and with the exception of 1966–1969), ruling coalitions in Germany could always be clearly identified as centre-left (if the chancellor was SPD) or centre-right (if the chancellor was CDU). Prior to 1983, this was simply because there were only three parties in parliament of which one (the liberal FDP) was far smaller so it was a natural small partner for one of the large parties. Indeed, ruling coalitions at the time were basically 'who did the FDP campaign to support?' Between the elections of 1983 and 2002, there were four parties meaning that the FDP (liberal, i.e. centre) was mostly associated with the CDU/CSU (conservative) while the Greens (progressive, varying degrees of left) were mostly associated with the SPD (socialdemocratic) which formed two nice blocks that could be tallied up on election night to determine a winner. Majorities were clear enough that the PDS (further left than SPD and Greens, former socialist party of East Germany) did not change the equation between 1990 and 2005.

After the 2005 election, the old system fell apart because the greater share of the PDS prevented a simple two-party coalition along the old block lines (CDU/CSU/FDP had 287 seats, SPD/Greens 273, majority at 308). The tl;dr is that nobody wanted a three-party coalition and nobody wanted to work with the PDS, so a grand coalition of the two largest parties was formed. As the CDU/CSU had more votes than the SPD, Merkel became chancellor.

In 2009, things seemed to 'go back to normal': CDU/CSU and FDP won an outright majority. However, they lost that majority in the 2013 election (the FDP failing to get elected to parliament at all). While there was a majority for SPD/Greens/Left party (former PDS), the SPD had once again excluded working with the Left party. Likewise, the generally left-wing Greens did not agree to coalition negotiations with the CDU/CSU and thus another grand coaition was the only option.

2017

In 2017, the AfD entering parliament as an anti-refugee party to the right of the CDU/CSU changed things up again. Nobody desired to work with the AfD in any way. Neither a coalition of SPD/Left/Greens nor SPD/FDP/Greens would have a majority. That only left two options with the CDU/CSU as top party. A so-called Jamaica coalition (CDU/CSU/FDP/Greens) was negotiated but the negotiations failed. Another Merkel-headed grand coalition became the only remaining option with the SPD grudginly accepted to prevent a snap election.

After the grand coalition of 2005, the SPD had lost almost one third of its vote share and never recovered to anywhere near 30 %. This had been seen as a failure to promote socialdemocratic policies during the grand coalition. Therefore, they negotiated harder prior to the next two grand coalitions and were able to implement a lot of their policies. However, while the implemented policies proved popular, they seemed to be attributed mostly to Merkel's party as shown by the failure of the SPD to gain any ground in polls (while the CDU/CSU remained roughly where they were).

2021

Merkel announced after re-election in 2017 that she would not seek a fifth term as chancellor, meaning that the election of 2021 would essentially be a clean slate with no incumbency bonus. The SPD, with the experience of the previous three grand coalitions did not want to form another one – and quite honestly hadn't really wanted to form one in 2017 either.

Throughout the election campaign polls were rather volatile with the Greens being ahead in late spring or early summer, the CDU/CSU pulling back before the SPD gained ground and managed to displace the CDU/CSU by election night. There are probably a number of causes for why the election went this way, so I won't try and pick out any singular one.

Nevertheless, the outcome was:

• the SPD is clearly the strongest party in parliament
• the CDU/CSU has suffered one of their strongest losses ever
• still nobody wants to work together with the AfD in any way, shape or form
• a left-of-centre SPD/Greens/Left coalition does not have a majority
• possible options given the election results are: Jamaica, traffic light or grand coalition

What will come from this? Well, political wisdom based on German experience since 1983 says that the SPD has most dibs on the position of chancellor. They could choose either the CDU/CSU or Greens and FDP as their partner. In a constellation unheard and unthought of at a federal level since 1980, there are two essentially equivalent options from the point of view of the small parties: FDP and Greens together can either give the SPD or the CDU/CSU a majority.

From the SPD's point of view, with a fresh election victory, they really would prefer to send the CDU/CSU into opposition; not only due to their prior grand coalition experiences. Thus, the SPD clearly favours a traffic light coalition.

From the small parties' point of view, a number of current events have made them rather unhappy with the CDU/CSU after their inital talks. For example, the CDU is in the middle of reorganising and a mild leadership crisis after the lost election. The CSU, especially via their head, Söder, are more or less openly criticising the CDU from within-but-not-quite. And finally, confidential information about the contents of the bilateral talks with Greens and FDP were leaked to the press which is not exactly a trust-building action.

Bearing this in mind, practically all circumstances favour a traffic light coalition; the only remaining obstacle being whether a coalition agreement can be negotiated that can pass with all three parties in that coalition.

One final thought: you linked to polls asking which coalition voters favoured. However, the voters have no further say in this. After the voters provided the parties with the election results, the parties have to thrash out what they want to do with this result. Surely the parties will be influenced by polling a little, but only the parties themselves will decide. This could be via delegates (the typical case) or via members' poll (what the SPD did in 2017 and maybe also 2013). Combined, the four in question only have 1.155 million members; meaning that this will be a decision of less than 2 % of the electorate.