In some states, election day is a holiday.
The counter argument you are looking for is that the federal government shouldn't dictate how states implement their elections. If the states wish to declare their own civic holidays on election day they are free to do so, without the federal government requiring it and the corresponding loss of productivity from anyone else who doesn't want the holiday.
Other states solve the issue differently, for instance Colorado, Oregon, and Washington hold their elections via the postal system. Legislators can argue that the loss of economic activity due to the closing of businesses adversely affect the state, and other avenues of solving the problem of lack of participation are available.
Another alternative solution proposed would be to move the current election day to always land on November 11th, which happens to be Veterans Day and is already a federal holiday that is widely observed. This can be done without taking the step of declaring election day itself a holiday around the same time in November when elections are generally held, and a lot of people already have the day off of work so there will be less work hours lost than having a separate holiday.
Additionally, the left-leaning Slate published an article pointing out that, even though many people would get the day off, other certain types of establishments that don't generally close for federal holidays still won't benefit because they won't observe the holiday. (Inc.com also published a similar argument.) Restaurants, retail, healthcare, all are businesses that generally get inundated with customers on other federal holidays, causing those industries an undue strain on the same day where their workers would need time off to go to polls also.
Still further, the problem of low-income voters not voting may be an XY-problem. Perhaps it's not because they can't get the time off to vote, but rather they don't believe voting would do them any good if they did, making the exercise not worthwhile in their eyes. If their employer closes in deference to a holiday, they may be adversely affected financially against their will.
One final observation is that there is more to an election than just the day of voting. The only holiday we're talking about is the day of the actual election, but counting primaries and run-offs there could be more days throughout the year that require voter participation. These elections are arguably more important, since they determine who gets to appear on the ballot. Creating a federal holiday for the election won't encourage people to participate in choosing who is available in the pool of candidates to pick from.