Why does this [seemingly unsustainable] continue when the two parties could instead compromise to create something with lasting power and less resentment?
TL;DR: The biggest problem is that the partisans actually have different ideas about the direction in which to go.
Taking healthcare as an example. Conservative Republicans want to shift healthcare spending such that individuals have more of a role and insurance companies and the government have less of a role. Liberal Democrats want to shift healthcare spending such that government has more of a role. Conservatives want lower taxes. Liberals want more taxes for more spending.
There are insufficient moderates in the two parties (plus independent Angus King of Maine) to form a governing majority in the Senate (sixty votes). So any bill requires either conservative Republicans or liberal Democrats. And to get them on board, the moderates have to agree to either increase the role of government or decrease it.
There are other issues than just the role of government. Pro-life Republicans want to ensure that no federal dollars go to pay for abortions nor even contribute to facilities that support abortions. Democrats want to ensure that all women's health facilities are fully funded, including those that support abortions. Republicans don't want to increase the overall funding towards such places at all. Democrats don't want to advantage facilities that do not perform abortions over those that do. There is no compromise between those positions.
Pew Research quoted Keith Poole of the University of Georgia as saying:
“With almost no true moderates left in the House of Representatives, and just a handful remaining in the Senate, bipartisan agreements to fix the budgetary problems of the country are now almost impossible to reach. [And] given that trends in polarization have continued unabated for decades and appear to be related to underlying structural economic and social factors…it is unlikely that this deadlock will be broken anytime soon.”
Now, some have blamed this on gerrymandering, partisan primaries, etc. But the truth is that voters have become more polarized as well. The Pew source shows people identifying as liberal increasing from 15% in 2000 to 20% in 2012. And conservatives increase from 32% to 34%. That's just in twelve years, well into the process.
Real Clear Science reported:
While most everyone has been saying it, science now supports it: Political partisanship is the worst it's been in over half a century, and it's increasing at an exponential rate.
Or their sister site, Real Clear Politics reported:
Finally, note that ticket splitting was essentially extinct in 2012.
In 2016, ticket splitting between the Senate and presidency was extinct. Every state that Donald Trump won voted in a Republican Senator. Every state that Hillary Clinton won voted in a Democratic Senator. There were some House seats (35) where the results split with the presidency, but no Senate seats.
Until voters start voting in a less partisan way, it is unlikely that politicians will. If they do, their voters will likely punish them. For example, of the twenty-five Democrats in the House who voted against healthcare reform in 2010, only three are left. The last time I noted that, a bunch of people posted that voting against Obamacare was not a moderate position. The problem is that it was. They were Democrats, agreeing with Republicans on an issue. If that's not moderate, then what is?
A large part of the problem is that too many people see compromise as those nasty people on the other side voting with them for what is a partisan goal. The truth is that compromise means that you have to give up something that you want in order to get someone else to give up something that they want. But neither side is willing to accept such compromises.
Conservatives think that when they try to compromise with liberals, that the liberals just keep taking and taking. Liberals think the same about conservatives. For example, on healthcare reform, the individual mandate was a Republican suggestion in the 1990s. Liberals feel that conservatives pulled the rug out from under them. Conservatives are essentially rejecting their own compromise suggestion.
Of course, there are very few supporters of the individual mandate from 1994 still in Congress in 2017. There have been three wave elections since then. The Republicans in Congress now are entirely different people than those in 1994. The terms of the debate shifted. It's also worth noting that while the individual mandate was suggested by Republicans, it was never really a conservative proposal. It was what the moderate Republicans were suggesting as a compromise.
Know what a moderate Republican was called in 2009? A conservative Democrat. Most moderates registered as Republicans lost in 2006 and 2008, some replaced by Blue Dog Democrats. Then the Blue Dogs lost in 2010 and 2014, some being replaced by moderate Republicans again, although not necessarily in the same districts. Blue Dogs tend to run in Republican-leaning districts. Moderate Republicans tend to run in Democrat-leaning districts.