In part because to many being "American" means being white and Christian. Historically, the right kind of white and Christian. "Hyphenated-American" is a reminder that we are a nation of immigrants, not just white Christians. It's a reminder that there are very much two Americas.
As each immigrant wave comes to America they undergo a sort of hazing period by the previous immigrants who are now "American". Each of the groups you mention (except maybe the Swedes) underwent, or are still undergoing, this. The Irish, black people (doesn't matter where you're from, you're seen as "black"), "Mexicans" (again, doesn't matter where you're from, you're seen as "Mexican"), "Asians", Italians, Germans, Poles and other Slavs, Jews... It goes beyond race. LGBTQ, Catholics, "Muslims" (which can be any number of central Asian religions)... And the Native American holocaust we perpetrated.
"Hyphenated-American" labels have their own problems. They can be used as politically correct tools of discrimination via othering, lumping groups of people together to over-simplify and dismiss them. Many black Americans point out that they aren't from Africa, their heritage is from the Carribean, or South America, or Europe. Or their families were slaves and their culture and history destroyed and forgotten. They're just "black". "African-American" lumps together all black people no matter what they consider their heritage to be. Many "Asian-Americans" point out that lumping "Asians" together wipes out the cultural distinctions between the diversity of Asian culture: Chinese, Japanese, Korean, Vietnamese, Thai, Mongol, etc. Instead it allows them all to be lumped together as "Asians" who all act the same according to whatever caricatures and stereotypes go along with that, "all look same". "Muslims" cover 1.8 billion people with a huge diversity of sects and beliefs, but "Muslim" in the US can be used as shorthand for "radical Islam".
So even these labels are fragmenting into more refined versions according to the push and pull of cultural unity vs distinctiveness. These refinements can be used to educate the rest of the population and break their ideas of cultural homogeneity.
Most of these are still ongoing. Some groups have transcended and gained the privilege to be "American". Mostly older white immigrant groups (Irish, Italian, German, Slavic) and Christian sects, particularly the Catholics. The rest continue to live with individual or systemic discrimination and the knock-on effects of old discriminatory policies.
For example, Oregon has a long history of persecuting black citizens and Asian immigrants. Here in Portland, Oregon those scars are still there in how the population of the city is laid out and segregated with the black population still concentrated in one quarter (and rapidly being pushed out by gentrification) and why Portland, Oregon is still the whitest city in the US.
As another example, this 115th Congress is the most diverse... which means it's 80% white representing just 60% of the population. In 1984 it was 94% white representing 80% of the population. On the religious end, 90% of the 115th Congress is Christian while only 70% of the country is. Much of this comes from grossly underrepresenting Atheists and non-religious people: 23% of the population, only 1 representative in Congress.
For myself, as a third-generation American from Irish, German, and Italian grandparents, and someone who studies history, I've seen how my own racial heritage went from persecuted to accepted to "American". As an atheist I grew up during an era when atheism went from the next worst thing to "devil worshiper" to acceptable. But Atheists aren't yet "American". God is still on our money and our pledge (introduced in the 1950s). The 23% of the population that is "unaffiliated" gets 1 national representative.
So to some, saying "let's just call ourselves American" says "let's pretend none of that ever happened" or "let's pretend that's all in the past" or "let's pretend we're all white and Christian". Let's pretend we're "post-racism" when clearly we are not.
Saying you're a "Hyphenated-American" is not divisive. Instead it highlights the existing and historical divisions caused by historical and ongoing discrimination in the US. Rather than sweep the problems under the rug and accept the status quo, a status quo in favor of "Americans", and pretend we're united as "Americans", it reminds us that there's still a lot of work to do to bind these wounds. And it probably will never end.