Vote by mail does not preclude secret ballots
I am a resident of Oregon, which was the first state to move to an entirely vote-by-mail system following the passage of Ballot Measure 60 in 1998.
A good vote-by-mail process can preserve a secret ballot. In practice, Oregon's process does a reasonable job of this. Unfortunately, the codified process does not enforce this, and a few modifications would make the preservation of secrecy much stronger.
For reference, here is Oregon's official Vote by Mail Procedures Manual.
Here's the short version:
Ballots are submitted in two envelopes:
- The outer envelope is the Return Identification Envelope. This includes the name and address of the voter, and a signature promising that the sender is who they say they are.
- The inner envelope is the Privacy Envelope. This is unmarked.
Ballots may be submitted directly to secure voting dropsites, of which there are many, circumventing the postal service.
Ballots are batch processed in three phases:
- The opening board:
(a) checks the signature on the return envelope against the signature on file for that voter,
(b) removes the return envelope, and
(c) removes the secrecy envelope.
- The inspection board sorts ballots into those that can be machine counted and those that can't. They fix non-machine-countable ballots where intent is clear.
- The ballots are counted.
Every step of the process may be observed/overseen by any member of the public that agrees to follow some obvious rules.
Each of the three steps at the opening board are done in batches of ~200 ballots at a time (source), which provides a reasonable amount of secrecy. That is, 200 return envelopes are removed, then the secrecy envelopes are removed from those 200 ballots. This means that nobody sees a ballot with its return envelope at the same time, and batches of 200 preserve reasonable anonymity.
Unfortunately, the batch size is not codified in the official process. More fundamentally, secrecy would be much stronger if 1(b) and 1(c) occurred in batches at different stations, so that no one person sees both the ballot and its return envelope.
Hopefully this makes it clear that ballot secrecy and vote-by-mail are not fundamentally opposed, as the question implies. Rather, ballot secrecy can be preserved by a good vote-by-mail process.
To more directly answer your question, however, yes, people make arguments against vote-by-mail on the basis of privacy (and also voter fraud). Unfortunately, I can't track down a copy of the official "Against" arguments for Oregon's ballot Measure 60, but political activist Bill Sizemore was and is one of the vocal opponents. You can read his thoughts on vote by mail here.
In general, however, the vote-by-mail system is quite popular in Oregon across the political spectrum, and Oregon has consistently high voter turnout relative to the national average, which is usually attributed to the vote by mail. In practice, I believe our process preserves a reasonable amount of secrecy. With the minor changes I suggest above, I think it would preserve it to at least the degree of most in-person voting systems.