The IRS is one of the government agencies that regularily steps on the wrong toes - those of people who have money, influence and connections. At the same time it doesn't have any active programs that directly benefit any of those people, because it doesn't do the spending side.
As a result, it has few people who speak or lobby in its favour and many who actively work against it. On the abstract, most people agree that the IRS is doing a necessary and beneficial job - but when it comes to the specific actions, especially those that affect us, we aren't exactly fans.
And while spending on the IRS would have a positive ROI, the same is true for the other side, especially considering the very rich: Spending to avoid paying taxes, be it on tax advisors, corporate lawyers to structure your company or investments in a way that avoids taxes, or lobbying against new taxes, for the reduction of old taxes, or the reduction of enforcement also has a positive ROI.
This is not limited to the IRS, but a global phenomenon. When the Cum-Ex scandal became public, damage control went into overdrive, and as a result many of the penalties stand uncollected to this day. In my country, a unit of our IRS equivalent that uncovered massive tax fraud among the rich was disbanded instead of promoted. And so on.
tl;dr: Nobody particularily likes the IRS. Most people consider it a necessary evil. Thus it has few advocates and many enemies, especially among those who pay a lot of taxes. But people who pay a lot of taxes also have ways to influence legislation.