This abstract scenario could only occur in a system that is ripe for abuse. Now, despite there being regulations to prevent abuse, how well those regulations are followed, and how "ripe" many state and municipal processes are might be a matter for debate.
In almost any voting system, there are laws, rules and regulation for maintaining the security of the ballots or records, and keeping a chain of custody log. For example -
Questions about the authenticity of ballots arose during the 2011 Supreme Court recount process due to holes in some ballot bags, gaps in their closure or issues with security tags. A hole in a ballot bag or a missing security tag is not enough evidence alone to discard the ballots inside. The ability to put a hand into a ballot bag is not by itself evidence of fraud.....
... After the polls close, election workers print out a tape which lists the tabulated vote totals. The poll workers remove the voted ballots and place them into a secured container or bag. The bag is secured using a tamper evident numbered seal. Ballot containers have all potential openings secured in such a manner that no ballot may be removed, nor any ballot added, without visible interference or damage to that ballot container. The seal number is recorded on the Inspectors’ Statement and Ballot Container Certificate by the poll workers. Election officials are required to maintain a chain of custody record that documents the movement and location of election ballots from the time of delivery of the ballots to the municipal clerk or board of election commissioners until the destruction of the ballots is authorized under § 7.23 Wis. Stats.
Even if the container or bag is somehow opened later, or if the chain of custody is broken, election officials have the original print-out tape from the machine, as well as the electronic memory device from the machine. This enables election officials to determine the election night vote count.
Wisconsin Elections Commission: Authenticity of Ballots and Responsibility for Conducting Recounts
If they had a verified machine count, and taped printout of each and every vote, and then they witnessed the contents being tampered with in that way, and a hand-verification of the contents showed a discrepancy of one vote/ballot, they'd be able to use the original recount, in all probability.
When doing a recount, if the seals appeared tampered with or the control numbers do not match the logs, they are supposed to consider that container to be "spoiled," and not include the counts, though in the past, they've chose to accept those situations as minor, malice-less human error, rather than deal with the scandal of an election being possibly tampered with. Again, a Wisconsin example -
Kloppenburg's campaign has raised questions about torn ballot bags in Waukesha County. But Geske said a rip in a ballot bag would not be enough for a court to throw out the votes in that bag without more evidence to draw a reasonable inference that fraud had been committed.
Keep in mind, the combination of a partisan county clerk "finding" thousands of votes after initial reporting, and the ballot bags not being securely sealed, from that same county during the recount, were not enough to have ballots disallowed. It seems like you'd have to have someone dressed like a villain being seen to tamper with a bag to actually cause action.
Initially, Kloppenburg appeared to defeat Prosser by 204 votes. Two days after the election, Waukesha County Clerk Kathy Nickolaus announced she had not included about 14,300 votes from Brookfield, 76% of them for Prosser, in the initial tally she gave to the news media.
Wisconsin State Journal Archive: Prosser wins recount in Wisconsin Supreme Court race
In terms of at what point a bag of ballots would be deemed to be "spoiled," I think that would depend on whether a bag was outside the chain of custody, and the seal(s) compromised, and just found in that condition, in which case you'd have no way of verifying that the count was valid, and, if election fraud was going on, a matching vote-tape from a machine would just be a circular "verification." In a formal recount, I'd think the bag would have to be disallowed, entirely, because, a dozen other ballots or ten thousand, any or all of the ballots would be questionable. But, that's under an ideal and strict standard. As already shown, the scandal of having that happen in an election causes those overseeing verification to dismiss or hand-wave problems with safeguards of ballot authenticity.
If we had your scenario where someone inserting a single ballot was seen, but we couldn't tell which one it was, as long as the rest of the security procedures were in place and verifiable, none of the ballots would be considered "spoiled," because a single illegitimate vote would still be within the standards for margin of error, both machine and human, of a normal recount or verification. Therefore, throwing in one illegitimate ballot into the election mix would not have an actual statistical impact. That single vote would not be worth nullifying others, and, indeed, the visible nature of such a crime might indicate that this would be the goal/reason for the insertion of that ballot.
This is the same reason why voter impersonation fraud, statistically, does not exist. There is almost zero impact in terms of altering an election.