This graphic from a FiveThirtyEight article shows dozens of Congressmen were removed from office during the first four decades of the 20th century due to having election results overturned. Why was this so common at the start of the century, and what happened such that disputed elections haven't resulted in congressional representatives being booted from office since then?
Apparently, the House itself is the ultimate arbiter of whom to seat after an election.
From the FiveThirtyEight spreadsheet (thanks Michael Seifert), the most common reason for these elections to be overturned was that they had been challenged; the loser in the election had made a formal case to the chamber to be seated instead of the winner. Apparently they were often able to convince the body in this kind of case.
The article Partisanship and Contested Election Cases in the House of Representatives, 1789-2002 has lots of detail about these cases, looking at how partisan they were. Here is one trend worth noting:
... contested elections were the vehicle by which the Republican Party sought to preserve a party organization in the South during the late-nineteenth century. That is, in the face of evenly divided national electorates from the mid-1870s through the early-to-mid-1890s and a growing Democratic-led disfranchisement of African Americans in the South, Republicans turned to contested elections as an institutional equalizer, a tool to maintain a sufficient number of southern seats to retain majority control of the House. Thus, for a period of two decades, contested elections played a crucial role in southern politics and House politics specifically, as well as national party politics more broadly.
Your chart shows contested elections immediately following this period. The various election-winning, seat-losing Ds in the FiveThirtyEight spreadsheet might have lost out in similar circumstances.