There are many ways to do proportional representation, and the choices affect the chances of fringe party candidates.
A way I'm most familiar with is Regional PR. If you split the nation into smaller but sizable geographical areas of roughly the same population, you can have each region select, say 15-25 representatives. This means that a fringe party must, regionally, be supported by 4 to 6 per cent of the voters to win a single seat. Electing MPs regionally is done nearly everywhere, I think. Basically to guarantee that each part of the nation has a voice.
An alternative, or an additional constraint, may be that a party must have a certain level of support nationwide. Do keep in mind that the use districts naturally leads to a certain kind of minimum overall support requirement in that getting 2 per cent of the vote in all the districts (in my example) won't earn the party any seats.
Such a system may still favor largish parties, because parties with, say, less than 10 per cent popular support often would win a total that would proportionally give them 1.5 or 1.6 seats, which (according to whichever algorithm is chosen) may lead to only a single MP. In that sense may be one third of the votes to that party were "wasted", as less would have sufficed to get that one seat. On the other hand, for a larger party that would percentagewise win 6.6 seats may actually get 7 (under d'Hondt) and anyway at least 6 seats, so will "waste" a smaller fraction of its votes. As I said, details like this depend on the choice of system, and some other systems don't have such bias. I think of this as a feature rather than a bug of d'Hondt, but opinions differ :-)
PR may still lead to their being two parties, or at least two loosely defined blocks, but it leaves room for some interesting coalitions. Human beings having tendencies to argue with each other, there is a need and also room to for several major parties. PR does seem to lead a certain kind of gravitation towards the political center when no single party has a chance to gain a majority by itself. This means that at least some party leaders must be in speaking terms with each other (necessary to rule together). I can say that in my native Finland the campaigns are usually less noisy than, say in the US. Consider the scenario of 3 major parties, some two of which are needed to form a majority coalition. If one of them indulges in excessively dirty campaign tactics, alienating their rivals, then guess who will be the odd one out in post election talks forming the coalition.
Most of the time the fringe parties remain in the sidelines. The voting power of a handful of MPs can gain only so much. But,
PR may lead to a "fringe" party getting a disproportionate amount of power, if special circumstances apply. An example that springs to mind is the current parliament in Sweden. For a longest time the elections in Sweden were about whether the left or the right block would have the majority. Then a populist party grew large enough to deny either block a simple majority. Nobody wants to play with them, so this has resulted in an impassé. Or, perhaps more accurately, together with the Swedish politicians being unable to trust parties from the opposite block, this resulted in a shaky situation of a minority government needing to bargain for its continued existence every now and then.
Anyway, PR, is not without problems. Listing a few:
- When all the governments are coalitions, who is accountable? The parties within a coalition have the option to blame their coalition partners for things that didn't go too well. Again, we can argue whether this is a bug or a feature :-)
- With districts necessarily larger, some smaller nooks may be left without an MP. Or, there is no such thing as "my MP/congressman" – a mythical representative at least Americans often refer to.
- Some of the implementations of PR involve parties listing their candidates in preferential order (if party A wins 6 seats from district X, the following 6 persons, often long time party mainstays, will be elected). In other implementations voters vote for individual candidates within a party, and if party A again wins 6 seats in district A, then those six candidates within a party who got the most votes will get the nod. Pros and cons here also.