Counterfactual: Random Voters
Let's posit a counterfactual world where voters randomly determine whether they will vote or not. Each voter is independent of each other and who they vote for is independent of whether or not they vote.
In this case, it's trivial to say that (so long as an appropriate sample size is met) more voters do not really influence the outcome of the vote.
The real world
This isn't true in the real world.
The first reason is voter turnout: the most important factor in determining whether people vote is the social pressure to do so. This can be "manufactured" by using direct marketing (people knocking on doors), telephone calls, mailing, etc. - but it works even better when people in your own social group pressure you to vote. This pressure is significantly increased when voters feel that the vote is going to be close, meaning that their votes are particularly important.
There can be institutional factors which provide incentives to vote. Powell and Jackman provide a decent list in this article. And it's worth noting that Powell and Jackman are well known in the field of voter turnout.
If you have access to academic journals, here are some articles about voter turnout that you might be interested in:
A second reason would be that our voting preferences are not distributed evenly. The most important factors in who we vote for are all demographic (age, race, religious background, your preferred measure of socioeconomic status, etc.). This is the basis of the 'funnel model' popularized by "The American Voter". If you want to read more about voter behavior, this is the single best resource available. It was re-published in the 2000's and can be had on Amazon.
In an extreme case, even an election or referendum with 100% voter turnout would not be unbiased - it would be biased in favor of registered voters.
Many people never register to become voters. I was unable to find any literature on why people do not register to vote, but my suspicion is that it is not random.
There are also examples of large swaths of the public being unable to vote by law. For example, minimum voting ages prevent the vote from being representative of people in those age categories.