This is really asking multiple questions stacked together: How do you fairly vote among a large set of options? How do you weight the votes of distinct electoral groups voting on the same issue? How does that weight change when one electoral group has a presumably much stronger interest in the outcome?
Voting Among Many Options?
I suspect you were hinting at this answer, but your first sub-question probably lends itself to some sort of Condorcet or alternative-voting system. All voters rank their options numerically in order of preference. For example, in an election with 3 political outcomes (Puerto Rico style) and 3 timelines for independence, Victor ranks his votes thus:
- Commonwealth
- Independence in 5 years
- Independence in 1 year
- Statehood
- Independence immediately
And all the other voters similarly rank their choices. The calculation is done with a series of implicit elections between the options. If there are five options, then there are 10 separate binary elections. The winner will vary with the specific formulation of Condorcet used, but often the winner will be whichever option wins the most binary elections - this option can be said to be most preferred to the others. Sometimes the tiebreaker, if necessary, will be whichever option had the strongest result in its poorest showing (e.g. whose worst outcome was the closest to winning?)
Of course, there's also instant runoff voting, where each vote counts once but is transferable. If no option has a majority of number 1 votes in the first round, then the lowest-ranked option is eliminated and those who ranked it first get their votes transferred to their second-best choices. The process repeats, with options eliminated and their voters transferred to lower-ranked votes, until some option gets a majority. Possible tie-breaker (very rarely needed in a group of any size) would be which option had more #1-ranked votes.
Voting Among Many Groups?
It's actually quite common in history to weight the scales for various groups in the process of voting or selecting representatives. There are many methods for this, including:
- Openly weighting voters or their representatives, such as the French Estates General or Swiss half-cantons.
- Allowing certain groups to vote earlier and so influence later voters, as with Iowa and New Hampshire.
- Increasing the number of representatives, districts, or votes in certain areas relative to others, as with Japanese electoral districts or the US Senate and small states.
- Manipulating electoral district boundaries to concentrate distinctly identifiable groups into fewer districts and so minimize the number of electoral victories, or the opposite and disperse those groups among as many districts as possible to deny them a plurality anywhere (called packing and cracking, respectively).
- Special formulas or designated seats for different electoral groups, such as quotas for women, confessional allocations for religions in Lebanon, and the Swiss magic formula guaranteeing political party balance.
Of course there are more varieties and far more examples, but the point is that it takes many forms. However, these tend to be the product of negotiation and settlement to maintain a peaceful atmosphere long into the future. Moreover, they tend to involve far more sides and parties, for example, a handful of Christian confessions and a few Muslim ones, with various complex alliances and competitions between these groups.
Voting On Independence?
What you're talking about is potentially a single election that happens once, and in which there are essentially two main sides (broadly, local versus center), so the dynamics are quite different. Moreover, an independence referendum generally stems from a sense that the central government does not adequately respect the region and its prerogatives. So the question of negotiating the ground rules for a referendum is in large part overlapping with the cause for separation. It may be hard to agree to a delicate weighted balance, especially since a binary negotiation is more overtly zero-sum, unlike a multi-party situation where every group gets some of the preferences and each group has more opponents than members (e.g. historical Lebanon, where the Christian and Muslim confessions were to some extent competitive with each other).
All of which is to say there are lots of ways to do it, but most of those situations have different dynamics. So likely none of them are very instructive here.
But one feature of independence referenda is that typically the larger region being seceded from does not engage in a vote. Wikipedia has assembled a significant list of independence referenda and you'll note this trend generally borne out. But see the French plebiscite approving the Evian accord and letting Algeria secede; however, this merely ratified the agreement that the government had already negotiated and signed.
The central government tends to hold for itself the right to approve or deny moves toward autonomy or independence. First, because it would be embarrassing for an anti-autonomy government if even a small number of non-residents of the upstart region supported the claim. Second, because there's a good argument to be made that those non-residents have no real say since they are much less affected by the choice.
So as with Catalonia and Scotland, both scheduled to vote on independence before the end of 2014, it's more likely that a California referendum would only be voted on by residents of California. This is also what the Puerto Rico referendum looked like, with no vote in the mainland.
The federal government would then be responsible for figuring out how to respond to the vote, if at all, and would be unlikely to call for a plebiscite or national referendum. Note that the US has never had a single national referendum on any issue, aside from the presidential elections. Even movements to amend the Constitution move through state legislatures and Congress, so the regular voting population is only indirectly involved in the decision.
