If you are talking about the American system, then the solution is simple: The US is a two-party country, so stop trying to pretend the other parties matter. They don't, except to the extent that they are able to act as spoilers for the major parties. Nobody is (intentionally) gerrymandering for or against them, so there's nothing to measure in the first place. Let's assume, for the sake of argument, that you're talking about a different system in which small parties actually do win elections.
Let's assume, for the sake of simplicity:
- Each constituency consists of exactly 1000 voters.
- There are two big parties, which each get about 35% of the vote (~350 votes) in most constituencies, leaving 30% (300 votes) to divide between the remaining candidates.
- Each big party wins half of the "typical" constituencies. Each big party gets approximately 0 wasted votes from the constituencies it wins, and approximately 350 wasted votes from the constituencies it loses, for an average of 175 wasted votes per constituency.
- There are a small number of atypical ("regional") constituencies too, discussed below.
Instead of small parties, consider regional and quasi-regional parties. A regional party only runs candidates in a small number of constituencies, usually those in a specific geographic region. A quasi-regional party runs candidates in all constituencies, but is only competitive in a specific region.
A quasi-regional party runs in every constituency, but only wins in a few. In all other constituencies, all of their votes are wasted. We could imagine that they receive 20% of the vote in a typical constituency, meaning that they have 200 wasted votes per typical constituency. That's more than the big parties, so on average, they will tend to look like they are actually the victims of gerrymandering. This will continue to hold even if they win a small number of regional constituencies, assuming the total number of constituencies is large enough. You could even aggressively gerrymander those regional constituencies. Although this does not hit the 8% magic number by itself, it does average (200 - 175) / 1000 = 2.5%, and with a little imagination, you can further tweak the numbers to get to 8%, or any desired threshold.
(One could argue that this party really is the victim of gerrymandering. But if that 20% is distributed evenly throughout the whole country, except for a small region where it is dominant, then it's difficult to see how you could draw a more equitable map without doing exactly the sort of "carving out individual voters" nonsense that is characteristic of real-world gerrymandering.)
On the other hand, a "true" regional party (one which only runs in seats it has a strong chance of winning) may end up running primarily in constituencies which, simply because of regional differences, tend to vote strongly in favor of that party (e.g. they get more than 40% of the vote, putting them ahead of the big parties). Such a regional party will still have quite a few wasted votes. But they run in a smaller number of constituencies overall, so their absolute number of wasted votes will be smaller than big "national" parties, as you describe. This may make it appear that they are benefiting from gerrymandering where none exists.
As a result of these problems, we may draw a fairly straightforward conclusion: You can't fix this by adjusting the threshold, because it's giving us both false positives and false negatives. Instead, you need to make some more fundamental adjustment to the formula, so that it can properly account for these issues.
