Polygamy was a target of a major political movement
No answer to this question is complete without mention of the notable Morrill anti-Bigamy Act of 1862, the Edmunds Act of 1882 and the Edmunds-Tucker Act of 1887.
The express purpose of those acts was to restrict the religious practice of plural marriage and other activities by members of the Church of Jesus Christ of Latter-day Saints, who at that time had largely emigrated to the Utah Territory. The acts criminalized polygamy and enforced sanctions against those who practiced it or taught it, even to the extent of disincorporating the church and endeavoring to make its founders and practitioners felons.
The notion of polygamy being evil or reprehensible began to be more widespread and vehement in the United States some decades earlier as a direct response to the founding of this church and the practice of its religion, which did involve a number of cases of and teachings about polygamy. The indignation against polygamy became galvanized into a mainstream political movement when the 1856 Republican National Convention advanced the notion that it was a "barbaric relic" as a contemptible twin to human slavery, that had to be eradicated.
The founder of the Church of Jesus Christ of Latter-day Saints, Joseph Smith, had just been assassinated twelve years prior in 1844 by a mob in Illinois, in part spurred on by rumors of polygamy and myriad libelous claims against him.
After his death, the saints spent two years preparing to move West. They were headed for what was then Mexican territory to find a refuge beyond the borders of the United States where they could practice their religion without being mobbed and murdered for their beliefs.
The geopolitical aspect of this change did not last long; the Mexican Cession through the Guadalupe-Hidalgo Treaty of 1848 brought them once again within the borders of the United States, subject to its politics and policies.
What is ironic about the "twin relics" claim of the Republican Convention is that Smith was assassinated not long after he began campaigning for president of the United States; notably his platform included the abolition of slavery.
He was the first US presidential candidate to be assassinated. Some historians opine that his anti-slavery stance may have contributed significantly towards the impetus for his assassination. The later assassination of Abraham Lincoln over this same subject lends credibility to that hypothesis. Abraham Lincoln is widely hailed as the father of the Republican party.
On the basis of this understanding, did the two alleged "barbarisms" really belong together?
Relation of anti-polygamy movement to Christianity
The claim in at least one other answer that polygamy was disfavored in Christian societies ignores some very important facts about the Christian religion:
- Abraham was a polygamist (Genesis 16).
- Jacob was a polygamist (Genesis 29).
- Moses may have been a polygamist (Numbers 12).
- Hundreds and thousands of reputable practitioners of ancient Christianity, including during the mortal ministry of Jesus Christ, were polygamists.
- The law that God gave to Moses permitted polygamy, but forbade adultery and pagan marriages.
The idea that polygamy is unchristian or incompatible with Christianity or with any of the Abrahamic religions frankly denies the very Foundation of Christianity. To do so one must discredit the "father of the faithful" (Romans 4:11) and deny the covenants God made with the patriarchs and are ubiquitously affirmed in those volumes of Scripture.
Edit: Precursor bigamy legislation in the United Kingdom
There is a noteworthy law enacted in the United Kingdom, the Offences against the Person Act of 1861, that precedes the Morrill Act by one year, which states that bigamy shall be a criminal offense, together with many other criminal reforms. This is not the oldest anti-polygamy or anti-bigamy law on the books; legislation against polygamy appears to be as old as the late Roman Empire. (The Romans were actually a rare exception to the norm of polygamy in nearly all ancient cultures. Excepting times of war or
the privilege of potentates, we expect monogamous marriage to be the most common form of marital union due to simple mathematics).
The table on Wikipedia that you reference in your question shows the 1862 act as nearly the oldest "notable" example in modern times.
Modern Treatment of Polygamy
Fast forward to today: the State of Utah recently decriminalized adultery, fornication, and sodomy.
For a long time, fornication was a misdemeanor in Utah, while polygamy was still classed as a felony.
That has changed recently too, although apparently polygamy is still not treated with parity compared to extramarital relations, and perhaps never has been (enforcement of laws in the US against adultery, fornication, etc. has been a rarity). With the rapidly changing legal landscape, and questions continuing to arise about what will and will not be enforced, it is not yet clear exactly how the treatment of polygamy will compare to the treatment of sex crimes in the future. Based on the historic and present designations and enforcement rates, it appears that the old stigma is still at least somewhat in force, and polygamy may still be being treated as "more barbaric" than many forms of extramarital sexual licentiousness.
You are correct that polygamy is victimless since it is consensual and involves as strong a vow as monogamous marriage to maintain the relationship and commitment to provide for spouse(s) and children. Extramarital sex lacks these protections for partners and children and so are far from victimless.
Footnote on comparison to incest
There is not much comparison between incest and polygamy other than their illegality.
One reason why incest is scientifically wrong is the extremely high likelihood of children inheriting debilitating genetic anomalies due to inbreeding. This has been proven to be caused by parents sharing a deleterious mutation.
In the days of Adam and Eve, marrying one's sibling carried practically no genetic risks because few or no genetic errors had yet accumulated.
The Mosaic law forbade incest, but this cannot be used to condemn the children of Adam and Eve because Moses was 2500 years and many generations removed from Adam and Eve, and there was sufficient genetic differentiation by that time to make the risks of inheriting a redundant defect nontrivial, hence the introduction of a new law regarding it.