There are a bunch of different questions here. I'll try to write this answer in such a way that it'll be comprehensible to people who aren't familiar with Dutch (or even European-style) politics, so bear with me if I explain some of the basics. I also apologize for the length of this answer, it seems things have gotten a bit out of hand :-/ I'll try to make it shorter if I have some more time in the future.
What are their chances of governing?
In Dutch politics, "governing" means forming a coalition which holds a majority of the 150 seats in the house of representatives ("tweede kamer").
Currently, the PVV polls at about 33 seats. The only other two major parties that are likely form a coalition with the PVV are the VVD (25 seats) and CDA (14 seats), which is currently 72 seats and not a majority (all of this depends a bit on which poll you look at, but this is the average).
It's possible to add a fourth of fifth party (SGP, 3 seats, or perhaps the new VNL if it gets 1 or 2 seats), but adding smaller parties is difficult. The small parties will have to compromise on too many issues, so are somewhat reluctant to enter a coalition with three larger parties as this will result with a loss of popularity with their voters (like what happened with the Lib Dems in the UK after the formed a coalition with the Tories). The last time there was a four-party coalition was in 1972 (Den Uyl, which actually had a whopping five parties).
It's also possible that the polls will shift in the next three months and that the PVV/VVD/CDA combination will get a majority. The VVD and CDA already formed a coalition with the PVV in 2010 (Rutte I*) which didn't work very well. It was a stormy coalition and in in 2012 Wilders left the coalition somewhat suddenly after disagreements over the 2013 budget. this was perceived as something of a immature douchebag move by both the VVD and CDA (and indeed, most other parties). In Dutch politics you're expected to compromise ("polderen") or work out some deal ("you can have this policy if I can have that policy", commonly called "trading cows"), and not just stop negotiating.
The VVD – Wilders' old party and the mainstream party that's closest to the PVV – has a reasonably "normal" relationship with the PVV (more normal than other parties anyway) but there are many differences between the parties. For starters, the PVV proposes much more left-wing social politics than the VVD (more on that later) while also being much more right-wing on other topics. PVV leader Rutte has indicated that "forming a coalition with the PVV will be complicated", adding that "we will certainly not form a coalition unless Wilders apologizes for his 'fewer fewer' remarks". (source).
So, the VVD might want to form a coalition, but meh. They're not exactly excited about it.
For the Christian-Democrats (CDA) things are even more controversial. CDA leader Buma has consistently and unambiguously indicated that forming a coalition with the PVV is simply not an option (source), and the 2010 arrangement was wildly unpopular with its members and voters.
Then again, things tend to get a bit more flexible after the election is over and the parties have to face the reality of forming a coalition. Can the PVV become a coalition partner? It's certainly possible. There hasn't been a single post-war coalition where the largest party wasn't a coalition member, so it would be an extraordinary situation if the PVV manages to become the largest party and doesn't end up in the coalition; but then again, we live in strange times, and forming a coalition with the PVV has some unique challenges.
My prediction: either the PVV won't form a coalition, or they will form a short-lived coalition where they will fail miserably and will do poorly the next elections (basically a repeat of what happened in Balkenende I and the LPF, which can be seen as PVV's spiritual predecessor in many ways).
Did any of the mainstream parties announce they'd rather form a very big coalition than govern together with the PVV? Outlooks?
This was already covered in the above to some degree, but to add:
- SGP, 50Plus, VNL – Might theoretically be possible, I can't find any clear statements from those parties about this. These are small parties (8 seats among them at the most), but one of them may play a pivotal role as a fourth party in bringing a majority to a VVD/CDA/PVV coalition.
- Labour (PvdA) hasn't explicitly said something about it, and they don't really need to. Much of Wilders' efforts are spent on insulting the PvdA, which is blamed for "letting the Muslims in" and all the wrongs of the country. It doesn't even matter what the PvdA wants, because PVV will never want to form a coalition with PvdA.
- D66, GroenLinks, SP, ChristenUnie, PvdD – Same as PvdA applies.
With a party so often criticized, pushed into the extreme right corner (whether or not they belong there is a different topic), so heavily fought, how can this success be explained? What did the PVV do right, what did the other (mainstream) parties do wrong?
The PVV isn't just extreme-right, it's also extreme-left. If you look at its much ridiculed single page party programme we see some pretty left wing topics:
- Lower health care costs.
- Lower rents (this is literally all it says, it doesn't say how, but I presume by some government intervention in the free housing market).
- Lower pension ages. Higher pensions.
- Better home care and elderly care.
These are major talking points for the PVV and not especially right-wing; these are all points broadly shared with the left-wing parties. This is why governing with the VVD will be extra complicated.
At any rate, why the PVV does so well is a complicated topic. As Philipp's answer already indicates, far-right nationalists are on the rise all over Europe as well as in the United States (Trump). There are also two questions here really:
- Why has the PVV been mildly popular for ten years?
- Why is the PVV much more popular now (to the point of being the largest party)?
I think Philipp's answer already nailed the second question, so I'll not repeat that here. As for the first question, this is more complicated. Thus far, I haven't heard of a comprehensive narrative which explains this very well. But here are some things to think about:
There has been a succession of popular populist parties: LPF in the early 2000s, then the Socialist Party in the mid 2000s, and since then mostly the PVV. It turned out that 22% of the people who voted for the Socialist Party (SP) in 2006 voted PVV in 2009 (source). This may seem strange at first since the SP is very left-wing, but the PVV and SP actually share a number of topics (see above) and perhaps more importantly: they offer "change" from the "established elite". Some people will always be unhappy.
Wilders is very good at drawing attention to himself, not infrequently in the form of controversies. He often uses language which is highly unusual in the house of representatives (e.g. calling people crazy), which not only draws a lot of attention from the media, but is also a way to communicate clearly with his voters ("look at me speaking plain language among these out-of-touch academic elites").
In addition, many of his communications are "sound bites" through Twitter. I've seen discussion programmes spend an entire hour discussing a single Tweet from Wilders.
In short, he's been doing what Donald Trump is doing, except he's much better at it (e.g. he actually plans this, rather than having it happen due to his personality).
The only other party which has a somewhat similar view on immigration (Wilders' signature topic) is the VVD, but because Wilders is so good at grabbing all media attention the VVD doesn't get that much chance to actually tell the voters that.
Many of Wilders' points are simple, easily understood, and lack compromise. He's also good at conveying this in plain language (his party programme is just one page, instead of the usual ~50 pages.)
Lack of internal party disagreements. There is no doubt that Wilders rules the party. Dissidents are not given a chance to speak their mind. This is different from all the other parties which are much more democratic and where internal disagreements flare up (we can blame the sensationalists media for this).
It's the only party that draws in on xenophobic feelings. Yes, I know this is something of a taboo word, but it's almost certainly a factor. Several of his statements about Muslims and Eastern-Europeans are factually provably untrue, and can be explained only by a dislike of those populations (which we call xenophobia). The only question here is whether this is a minor factor or a large factor for his popularity (to which I have no answer).
Again, this is not a comprehensive explanation, but these are probably some important factors. This page is also an interesting read; it shares some points with the above, but also has some other views.
* Technically the PVV wasn't part of that coalition but a "tolerating partner", but for the purpose of this answer we can pretend it was.
Note: all polling data from https://www.allepeilingen.com/