In the US, various states have different rules on how parties may hold their primaries: some require direct ballots, some require caucuses, some states hold "open" primaries and some are "closed. But to me this seems perplexing as in principle political parties should be free to choose their candidates in whatever manner they see fitting, with no intervention by the state authorities.
So why do governments in the US get to regulate how political parties conduct their business?
A substantial factor were white primaries during the defining civil right struggles and the case law that followed from that. From a law paper on the topic "Developments in the State Regulation of Major and Minor Political Parties" (which is 74-pages long, by the way):
If political parties were truly private organizations, they could exclude whomever they wished from political participation—a result that would conflict with the "White Primary Cases" in which the Court protected racial minorities' right to participate in party primaries.
[in footnote] Cf. Gray v. Sanders, 372 U.S. 368 (1963) (finding state action when the state enforces exclusion of voters); Terry v. Adams, 345 U.S. 461, 469-70 (1953) (holding that the use of a discriminatory preprimary election administered by a private association, which determined the primary winner, constituted state action under the Fifteenth Amendment); Smith v. Allwright, 321 U.S. 649 (1944) (determining that political party's exclusion of African-American voters in primary constitutes state action); United States v. Classic, 313 U.S. 299 (1941) (holding that Congress can regulate fraud in primary elections); Nixon v. Condon, 286 U.S. 73 (1932) (finding that party's exclusion of black voters when authorized to determine voter qualifications constitutes state action).
And from Wikipedia's United States v. Classic:
Many observers assumed that the court had already ruled in Newberry v. United States, that primary elections could not be regulated under the powers granted to Congress under Article I, Sec. 4 of the Constitution. But writing for the majority, Justice Harlan Fiske Stone argued that the Newberry court had been deeply divided on the issue and no majority had ruled one way or the other. Utilizing the reasoning by Chief Justice Edward Douglass White and Justice Mahlon Pitney in their concurrent opinions in Newberry, Stone argued that the Constitution's protection of the right to vote cannot be effectively exercised without reaching to primary elections and/or political party nominating procedures.
Though broadly noting that the constitutional right to vote extends to a party primary even when it "sometimes or never determines the ultimate choice of the representative," the Court offered no standard for determining whether a primary "was made an integral part of the election machinery." However, in Morse v. Republican Party of Virginia, the Court clarified that this extends to virtually all primaries, noting that "Virginia, like most States, has effectively divided its election into two stages, the first consisting of the selection of party candidates and the second being the general election."
So basically the Supreme Court decided that there can be no free elections in the US without free primaries. Morse v. Republican Party of Virginia, (517 U.S. 186, 205-07 (1996)), which cemented/clarified this is a relatively recent decision, surprisingly.
However, the Fifteenth Amendment is relatively narrow. It
prohibits the federal and state governments from denying a citizen the right to vote based on that citizen's "race, color, or previous condition of servitude".
So based on it the government(s) don't get to regulate everything relating to primaries or nominations. In particular, it doesn't say who can or cannot be nominated in a primary. And the case law in this respect is much, much less straightforward (one reason why that law review is so long). I'm not going to get into the details here... but I will mention one case that limited the state's right to dictate every minute detail of how primaries have to be organized: Republican Party v. Faulkner County; without getting into the gritty case details, the principles derived in this case were (quoting from the law review again):
Faulkner is significant because the Eighth Circuit used associational rights as a substantive limit on the state's ability to dictate a party's nominating procedures. A state can violate a political party's associational rights if the state enacts a primary election system that unduly burdens the party's association with its members. Moreover, the court reached this conclusion by considering the actual impact of the nominating procedure on the party's campaign activities. Thus, Faulkner suggests that courts balancing the burdens imposed by an election scheme must consider evidence demonstrating that a scheme actually interferes with the party's ability to organize and campaign effectively. If the party can produce sufficient evidence, they can object to a state-dictated candidate selection method and the state must have a compelling justification for its electoral scheme.
The decision in Faulkner was based on broader principles previously set out by the Supreme Court in Eu v. San Francisco County Democratic Central Committee:
"a State cannot justify regulating a party's internal affairs without showing that such regulation is necessary to ensure an election that is orderly and fair."
So basically, in US jurisprudence, there is a balance of interests to be taken into account: fair elections stand in balance with unnecessary interference in party's internal affairs, the latter stemming from the right to political association.
There is a practical reason not yet mentioned by the other answers, that is: the state needs to print the ballots for the general election in November. Therefore, the state has to determine which candidates merit to be put on the ballot. Therefore, the state must make some kinds of laws about which parties are officially recognized, how each party may/must choose its nominee, how an independent candidate may earn a place on the ballot, and so on. States have chosen different approaches to solving these problems, but they cannot do nothing.
Generally, this is because of the 10th Amendment, which basically says that the Constitution only grants the Federal Government the rights that it says it does. Anything not explicitly listed in the constitution as a power the Federal Government has, the power devolves to the State (or the individual). As such, primaries are not mentioned in the constitution (nor political parties at all, the founding fathers didn't want party politics at all).
Most questions along the lines of "Why does the state have this power and not the federal government" are pretty much because the power was not mentioned in the Constitution, and thus, are state powers.
The reason why the states have this rule is because the states are technically 50 separate "countries" that collectively conduct their foreign policy with each other and non- US nations through the powers of the Federal Government.
It sounds like we're talking about Presidential primaries here, so it's because the Constitution explicitly assigns to the states the right to chuse in what manner the state's electors shall be chosen and primaries are part of that process.
In the US, why does the government have the right to regulate how political parties hold their primaries?
It doesn't. This is not a legitimate enumerated power of the US government, nor of State governments.
This is what the Constitution does say about the authorized manner of conducting a presidential election:
Each State shall appoint, in such Manner as the Legislature thereof may direct, a Number of Electors (Article II, Section 1, Clause 2)
In other words, only State legislatures may determine the manner of choosing electors, and the electors are unconstrained by any other artifice; the Constitution does not allow such constraints. The electors are appointed in the manner given by the State legislature and are unconstrained per the Constitution.
In the US, various states have different rules on how parties may hold their primaries: some require direct ballots, some require caucuses, some states hold "open" primaries and some are "closed.
As we see above, each State does have the Constitutional prerogative to determine how its own electors are appointed. What it does not have is any Constitutionally authorized oversight of the nomination processes of any political party. None. You can search the Constitution and there is no enumerated power or delegation of authority into parties nor is there any allowance for government intervention into, or oversight of them, beyond the protection of individual rights.
political parties should be free to choose their candidates in whatever manner they see fitting, with no intervention by the state authorities.
You are correct and that is what the original United States Constitution expressed (actually it was even more permissive; it makes no mention of parties at all. Parties could choose to hold a nomination or none at all. This has always been beyond the government's prerogative to control). Prohibitions against partisan control were significantly weakened with the 12th Amendment in 1804, since which time there have begun to be general elections in which the public votes for presidential candidates and States have thenceforth been in the business of propping up candidates to appease the national parties, and now even the amended version of the Constitution is ignored as partisan-government incest abounds.
So why do governments in the US get to regulate how political parties conduct their business?
Because the political parties now have nationalized oversight over how elections are even conducted, and how votes are allowed to be allocated. This is evident in the introduction of bound delegates per partisan rules, the existence of unconstitutional partisan "tickets" formed at national party conventions, and the existence of general elections by the public for presidential candidates. The Constitution allows none of this.
Remember, when the Constitution was originally ratified:
Under the original Republic, State legislatures determined how to appoint each State's electors, independent of all other power mechanisms. They could have done a statewide popular vote, a straw poll, or had their legislators nominate and choose electors, held nonpartisan caucuses, or anything else allowed under the Constitution. Once those electors were chosen, their own judgment was the test for who became the next president and vice president. If you voted in a State popular election for your elector(s), your elector(s) would bear your representation and vote according to his/their own judgment and conscience. That would be all you had to do with any presidential election, unless you had further involvement with a national party's nomination process. There was no presidential primary and no general election by the public for the office of president. The only election you would normally vote in was to choose your elector(s), if your State legislature had invited your participation in that electoral process. If and only if you were an elector, then you could vote for two presidential candidates, the runner-up of which would be vice president.
Nowadays, we vote at least three times in each presidential race. Once on caucus night to get our delegates to the party convention. If you are a delegate, you vote a second time to nominate your party's candidate(s). (This gets overruled if there is a signature-gathering option.) Then you vote again in a partisan primary. Finally, you vote in a party-controlled general election.
The original Constitution was much simpler and much less vulnerable to incestuous control of the parties by government, and of the government by parties.
