Note: When I refer to parties, I will be counting CDU/CSU as one except where I spell out CDU without CSU.
As usual, I believe the situation cannot be understood without knowing where we came from.
A brief historical overview
Until 2005 (and with the exception of 1966–1969), ruling coalitions in Germany could always be clearly identified as centre-left (if the chancellor was SPD) or centre-right (if the chancellor was CDU). Prior to 1983, this was simply because there were only three parties in parliament of which one (the liberal FDP) was far smaller so it was a natural small partner for one of the large parties. Indeed, ruling coalitions at the time were basically 'who did the FDP campaign to support?' Between the elections of 1983 and 2002, there were four parties meaning that the FDP (liberal, i.e. centre) was mostly associated with the CDU/CSU (conservative) while the Greens (progressive, varying degrees of left) were mostly associated with the SPD (socialdemocratic) which formed two nice blocks that could be tallied up on election night to determine a winner. Majorities were clear enough that the PDS (further left than SPD and Greens, former socialist party of East Germany) did not change the equation between 1990 and 2005.
After the 2005 election, the old system fell apart because the greater share of the PDS prevented a simple two-party coalition along the old block lines (CDU/CSU/FDP had 287 seats, SPD/Greens 273, majority at 308). The tl;dr is that nobody wanted a three-party coalition and nobody wanted to work with the PDS, so a grand coalition of the two largest parties was formed. As the CDU/CSU had more votes than the SPD, Merkel became chancellor.
In 2009, things seemed to 'go back to normal': CDU/CSU and FDP won an outright majority. However, they lost that majority in the 2013 election (the FDP failing to get elected to parliament at all). While there was a majority for SPD/Greens/Left party (former PDS), the SPD had once again excluded working with the Left party. Likewise, the generally left-wing Greens did not agree to coalition negotiations with the CDU/CSU and thus another grand coaition was the only option.
2017
In 2017, the AfD entering parliament as an anti-refugee party to the right of the CDU/CSU changed things up again. Nobody desired to work with the AfD in any way. Neither a coalition of SPD/Left/Greens nor SPD/FDP/Greens would have a majority. That only left two options with the CDU/CSU as top party. A so-called Jamaica coalition (CDU/CSU/FDP/Greens) was negotiated but the negotiations failed. Another Merkel-headed grand coalition became the only remaining option with the SPD grudginly accepted to prevent a snap election.
After the grand coalition of 2005, the SPD had lost almost one third of its vote share and never recovered to anywhere near 30 %. This had been seen as a failure to promote socialdemocratic policies during the grand coalition. Therefore, they negotiated harder prior to the next two grand coalitions and were able to implement a lot of their policies. However, while the implemented policies proved popular, they seemed to be attributed mostly to Merkel's party as shown by the failure of the SPD to gain any ground in polls (while the CDU/CSU remained roughly where they were).
2021
Merkel announced after re-election in 2017 that she would not seek a fifth term as chancellor, meaning that the election of 2021 would essentially be a clean slate with no incumbency bonus. The SPD, with the experience of the previous three grand coalitions did not want to form another one – and quite honestly hadn't really wanted to form one in 2017 either.
Throughout the election campaign polls were rather volatile with the Greens being ahead in late spring or early summer, the CDU/CSU pulling back before the SPD gained ground and managed to displace the CDU/CSU by election night. There are probably a number of causes for why the election went this way, so I won't try and pick out any singular one.
Nevertheless, the outcome was:
- the SPD is clearly the strongest party in parliament
- the CDU/CSU has suffered one of their strongest losses ever
- still nobody wants to work together with the AfD in any way, shape or form
- a left-of-centre SPD/Greens/Left coalition does not have a majority
- possible options given the election results are: Jamaica, traffic light or grand coalition
What will come from this? Well, political wisdom based on German experience since 1983 says that the SPD has most dibs on the position of chancellor. They could choose either the CDU/CSU or Greens and FDP as their partner. In a constellation unheard and unthought of at a federal level since 1980, there are two essentially equivalent options from the point of view of the small parties: FDP and Greens together can either give the SPD or the CDU/CSU a majority.
From the SPD's point of view, with a fresh election victory, they really would prefer to send the CDU/CSU into opposition; not only due to their prior grand coalition experiences. Thus, the SPD clearly favours a traffic light coalition.
From the small parties' point of view, a number of current events have made them rather unhappy with the CDU/CSU after their inital talks. For example, the CDU is in the middle of reorganising and a mild leadership crisis after the lost election. The CSU, especially via their head, Söder, are more or less openly criticising the CDU from within-but-not-quite. And finally, confidential information about the contents of the bilateral talks with Greens and FDP were leaked to the press which is not exactly a trust-building action.
Bearing this in mind, practically all circumstances favour a traffic light coalition; the only remaining obstacle being whether a coalition agreement can be negotiated that can pass with all three parties in that coalition.
One final thought: you linked to polls asking which coalition voters favoured. However, the voters have no further say in this. After the voters provided the parties with the election results, the parties have to thrash out what they want to do with this result. Surely the parties will be influenced by polling a little, but only the parties themselves will decide. This could be via delegates (the typical case) or via members' poll (what the SPD did in 2017 and maybe also 2013). Combined, the four in question only have 1.155 million members; meaning that this will be a decision of less than 2 % of the electorate.