In hindsight, the answer is yes, it's constitutional to restrict electors' votes at least by means of state-law punishments, which may include their replacement, according to SCOTUS Chiafalo v. Washington and Baca (2020):
Chiafalo v. Washington, 591 U.S. ___ (2020), was a United States Supreme Court case on the issue of "faithless electors" in the Electoral College stemming from the 2016 United States presidential election. The Court ruled unanimously that states have the ability to enforce an elector's pledge in presidential elections. Chiafalo deals with electors who received US$1,000 fines for not voting for the nominees of their party in the state of Washington. The case was originally consolidated with Colorado Department of State v. Baca, 591 U.S. ___ (2020), a similar case based on a challenge to a Colorado law providing for the removal and replacement of an elector who does not vote for the presidential candidate who received the most votes in the state, with the electors claiming they have discretion to vote as they choose under the Twelfth Amendment to the United States Constitution. [...]
The [Colorado] electors appealed to the Tenth Circuit, with oral arguments held in January 2019. [...] The court ruled in favor of the electors in a 2-1 vote in August 2019, agreeing that Baca's removal as an elector violated the Twelfth Amendment. The majority opinion, written by Circuit Judge Carolyn Baldwin McHugh and joined by Circuit Judge Jerome Holmes, stated that "The text of the Constitution makes clear that states do not have the constitutional authority to interfere with presidential electors who exercise their constitutional right to vote for the President and Vice President candidates of their choice." [...]
Instead of seeking an en banc review at the Tenth Circuit, Colorado filed a petition for writ of certiorari to the Supreme Court on October 16, 2019. Colorado's petition identified the circuit split between the Tenth Circuit's decision and that of the Washington Supreme Court in Chiafalo, seeking the Supreme Court's involvement to resolve the split.
The Supreme Court issued its rulings in both Chiafalo and Baca on July 6, 2020. Chiafalo was a unanimous ruling of the court, affirming the Washington court's decision that states may enforce the pledge of an elector in the presidential election; Baca was decided per curiam (with Sotomayor recused) reversing the Court of Appeal's judgement "for the reasons stated in Chiafalo..." Justice Elena Kagan wrote the majority opinion which all but Justice Clarence Thomas joined. Kagan wrote "Today, we consider whether a State may also penalize an elector for breaking his pledge and voting for someone other than the presidential candidate who won his State's popular vote. We hold that a State may do so...The Constitution's text and the Nation's history both support allowing a State to enforce an elector's pledge to support his party's nominee — and the state voters' choice — for President." Thomas wrote a concurrence that was partially joined by Justice Neil Gorsuch, adding that "nothing in the Constitution prevents States from requiring Presidential electors to vote for the candidate chosen by the people." In Baca, Thomas concurred in the judgment without an opinion.
According to some commentators cited by Wikipedia, SCOTUS surprised with its broad sweep allowing faithless electors not only to be fined (Chiafalo) but also outright replaced (reversing the 10th Circuit) in Baca.