First, the ADA is a lot more than tax breaks. It legally requires "reasonable accommodations" for disabled people. There are a lot of details and caselaw on what exactly that means, but the short of it is the law is intentionally vague so each case is mostly unique.
Their (not my) argument is it presents risks to employers.
Consider yourself a hiring manager. You bring in two interviewees, Alice and Carol. Both are about equal. Alice, while walking through the shop, draws everyone's eyes. You suddenly get the dark thought that she may get unwanted attention if she works here. Bob and Dan are good workers, and never had a complaint. But the way they looked at her... you're worried.
Sure, you should do something about Bob and Dan. That's not OK. But they haven't crossed a line permitting immediate, drastic action, so any fix there will take time.
You could really flip a coin between the candidates. But if it landed Carol, you don't need to worry about issues before you can address the Bob/Dan problem. No downside. So it landed Carol.
Congratulations, you just discriminated against good looking people.
This should be rare, even in their estimation. But there is no way to know the boss didn't randomly pick between candidates (or more likely value something unrelated instead). So even with laws against such a practice, there is no possible enforcement on individual cases short of mind reading, unless they get dumb and write it down. Disparate Impact can be used at scale to suggest its happening (like any other discrimination), but holders of this view typically view Disparate Impact analysis as not acceptable for legal cases.
To put a finer point on it, the belief is, if the employer knows a candidate is disabled (or has kids and is female, or may have them soon, or medical history, etc), the employer would prefer to hedge against hiring them for the risk issues may arise in the future.
I am describing their beliefs as I understand them; I am not making the argument.
Newsweek has a writeup:
https://www.newsweek.com/did-americans-disabilities-act-hurt-some-people-disabilities-71541