Would the Supreme Court have had the power to challenge this decision of the Lok Sabha?
Broadly, yes, but it depends on a case by case basis.
In the indian constitutional setup, the Parliament is not sovereign. Under the doctrine of power of separation, it is well-defined that the Judiciary cannot interfere with Parliamentary proceedings nor can the Legislature or the Executive interfere with a Judicial proceeding.
For example, Article 122 of the indian constitution - Courts not to inquire into proceedings of Parliament - does clearly say:
(1) The validity of any proceedings in Parliament shall not be called in question on the ground of any alleged irregularity of procedure.
(2) No officer or member of Parliament in whom powers are vested by or under this Constitution for regulating procedure or the conduct of business, or for maintaining order, in Parliament shall be subject to the jurisdiction of any court in respect of the exercise by him of those powers.
(Article 212 - Courts not to inquire into proceedings of the Legislature - spells out the same for the State Legislatures).
Note though that the power to interpret the constitution vests solely with the Judiciary. The Indian Judiciary has repeatedly emphasised that Parliament is not supreme, and it cannot make laws or act in any manner contrary to the fundamental rights of an individual or against the Basic Structure of the Constitution. Thus over the years, it has developed judicial doctrines to constrain privileges like Article 122 and 212, and figured out a way for itself to intervene, when it felt that fundamental rights or constitutional supremacy was under threat or legal clarity was required on certain issues of public importance.
We have seen this in action with courts reviewing a Speaker's decision (or indecisiveness) on disqualification requests, on holding sessions or floor tests, on questions on certifications of money bills etc.
Very specifically, in 2005 some Member of Parliament were expelled in a similar "cash for query" scandal.
In 2007, they approached the judiciary claiming that Parliament did not have the power to expel them. The court agreed to determine if the Houses of Parliament are competent to expel their respective Members from membership of the House and that if such a power exists, is it subject to judicial review and if so, the scope of such judicial review.
In Raja Ram Pal vs The Hon'ble Speaker, Lok Sabha the court summarised that as far as Judicial review of the Parliament was concerned:
(i) Parliament is a co-ordinate organ and its views do deserve deference even while its acts are amenable to judicial scrutiny;
(ii) Constitutional system of government abhors absolutism and it being the cardinal principle of our Constitution that no one, howsoever lofty, can claim to be the sole judge of the power given under the Constitution, mere co-ordinate constitutional status, or even the status of an exalted constitutional functionaries, does not disentitle this Court from exercising its jurisdiction of judicial review of action which part-take the character of judicial or quasi-judicial decision;
(iii) Expediency and necessity of exercise of power or privilege by the legislature are for the determination of the legislative authority and not for determination by the courts;
(iv) Judicial review of the manner of exercise of power of contempt or privilege does not mean the said jurisdiction is being usurped by the judicature;
(v) Having regard to the importance of the functions discharged by the legislature under the Constitution and the majesty and grandeur of its task, there would always be an initial presumption that the powers, privileges etc have been regularly and reasonably exercised, not violating the law or the Constitutional provisions, this presumption being a rebuttable one;
(vi) Fact that Parliament is an august body of co-ordinate constitutional position does not mean that there can be no judicially manageable standards to review exercise of its power;
(vii) While the area of powers, privileges and immunities of the legislature being exceptional and extraordinary its acts, particularly relating to exercise thereof, ought not to be tested on the traditional parameters of judicial review in the same manner as an ordinary administrative action would be tested, and the Court would confine itself to the acknowledged parameters of judicial review and within the judicially discoverable & manageable standards, there is no foundation to the plea that a legislative body cannot be attributed jurisdictional error;
(viii) Judicature is not prevented from scrutinizing the validity of the action of the legislature trespassing on the fundamental rights conferred on the citizens;
(ix) Broad contention that the exercise of privileges by legislatures cannot be decided against the touchstone of fundamental rights or the constitutional provisions is not correct;
(x) If a citizen, whether a non-member or a member of the Legislature, complains that his fundamental rights under Article 20 or 21 had been contravened, it is the duty of this Court to examine the merits of the said contention, especially when the impugned action entails civil consequences;
(xi) There is no basis to claim of bar of exclusive cognizance or absolute immunity to the Parliamentary proceedings in Article 105(3) of the Constitution and
(xii) The manner of enforcement of privilege by the legislature can result in judicial scrutiny, though subject to the restrictions contained in the other Constitutional provisions, for example Article 122 or 212. Article 122 (1) and Article 212 (1) prohibit the validity of any proceedings in legislature from being called in question in a court merely on the ground of irregularity of procedure."
It also said:
(ii) The rules which the legislature has to make for regulating its procedure and the conduct of its business have to be subject to the provisions of the Constitution;
(iii) Mere availability of the Rules of Procedure and Conduct of Business, as made by the legislature in exercise of enabling powers under the Constitution, is never a guarantee that they have been duly followed;
(iv) The proceedings which may be tainted on account of substantive or gross illegality or unconstitutionality are not protected from judicial scrutiny;
... (vi) An ouster clause attaching finality to a determination does ordinarily oust the power of the court to review the decision but not on grounds of lack of jurisdiction or it being a nullity for some reason such as gross illegality, irrationality, violation of constitutional mandate, mala fides, non-compliance with rules of natural justice and perversity”.
While the constitution does not explicitly say that Parliament has the power to expel its members, the courts interpreted that the constitution does allow Parliament this power. It also, as outlined above, clearly stated that the courts do have limited power of judicial review over Parliamentary proceedings.
In my opinion, Modi's government is on a weak footing on this particular case as they expelled Ms. Mahua Moitra based on allegations that are yet to be proven - the CBI and ED have only started investigating the case, and are yet to find any evidence to prosecute her. (In the previous case of the 10 expelled MPs the evidence was pretty strong as they had been caught on camera, in a sting operation). Ms. Mahua Moitra is already back in the Parliament as she has been re-elected again - so she doesn't need any relief from the courts to reinstate her as an MP. However, she hasn't withdrawn her petition in the court and it is still ongoing because she has alleged constitutional impropriety that the court may determine need a closer look.
Sources:
Constitutional Supremacy vs Parliamentary Sovereignty
Before Mahua Moitra, the other cash-for-query case: What happened to 10 of the MPs expelled in 2005
Raja Ram Pal vs The Hon'ble Speaker, Lok Sabha
