The size of the Supreme Court can be changed by passing a law
The size of the Supreme Court is set by statute: Title 28 § 1 of the United States Code. Under the necessary and proper clause, Congress has the power to pass legislation about how "all other Powers vested by this Constitution in the Government of the United States, or in any Department or Officer thereof" are carried into execution. Supreme Court justices are officers of the United States.
It doesn't need to involve removal of a sitting justice
As you mention, there is precedent that Congress may change the size of the Supreme Court. There is historical precedent for reducing the size of the court—but not for removal of any justices: the changes only affected whether appointments would be made to replace them. That approach (reduction by attrition) avoids the question you raise of "how would the justices to be removed be decided".
Impeachment: an established process to remove a federal judge
Supreme Court justices are considered civil officers of the United States, which are subject to removal through the process of impeachment per Article II § 4 (U.S. Constitution Annotated, LII, Cornell Law School).
Eight federal judges (none on the Supreme Court) have been removed after being impeached and convicted.
This removal process is not connected to the process of changing the legal size of the Court.
Could anything other than impeachment remove a judge? Unclear
Article III of the US Constitution precribes the duration for which federal judges hold office:
The Judges, both of the supreme and inferior Courts, shall hold their Offices during good Behaviour, and shall, at stated Times, receive for their Services, a Compensation, which shall not be diminished during their Continuance in Office.
There is no constitutional distinction here between a justice of the Supreme Court and another federal judge. The phrase “during good Behaviour” is widely interpreted as giving federal judges lifetime tenure.
It has been speculated, but not tested, that Congress could pass a statute allowing removal of a federal judge upon a judicial finding of bad behavior
However, I found an article, “How To Remove a Federal Judge,” Saikrishna Prakash and Steven D. Smith, The Yale Law Journal 2006, that argues that if an appropriate statute were passed by Congress, a judicial finding of misbehavior ought to be legally sufficient grounds for removal.
Prakash and Smith suggest removal of a federal judge could occur via something like the following procedure:
Our research suggests that under the historically established meaning of "good Behaviour" tenure, Congress could have enacted a statute authorizing the President or perhaps even a private party "' to bring an action in court to determine whether a judge had departed from "good Behaviour." A court so finding could then have ordered the judge's removal-subject of course to appellate review
(page 125)
This kind of procedure never seems to have been attempted; Prakash and Smith suggest that Congress preferred not to try this approach because it would require politically involving other actors than Congress. In contrast, impeachment is initiated by Congress and is not subject to judicial review, so if the Senate voted to remove an impeached Supreme Court justice, there wouldn’t be any way to appeal the removal.