This is known as the Hastert Rule. Hastert, a Republican, was the Speaker of the House from 1999-2006. Though in fact the rule was considered in effect and well known during the prior Speaker's tenure: Newt Gingrich (1995-1999; also Republican). Hastert himself said it was never really a rule one way or another, for what it's worth.
Typically speaking the rule is only informal, in the sense that it is not explicitly coded up in any sort of binding way within the rules of the conference or the House. It is more of a governing principle than a rule, guided by musings of Hastert and Gingrich that questioned in what sense you are the leader of your party if you require the other party to get something done. And a standard aspect of Congressional politics in either House tends to be to abide by the party majority's decision. Meaning a Representative may be opposed to some bill, or certain aspects of it, and will negotiate and advocate for what they think should or shouldn't be in the bill while it is being written. But once the party reaches a majority opinion that the bill should be finished off and passed, then by and large all members are expected to fall in line and vote "yes" on it. With only such exceptions as are needed as part of the bargaining process, typically as part of reelection strategy for members in more competitive districts or districts with opinions strongly aligned against a given bill. Usually this too is only an informal agreement, though it has appeared explicitly in caucus/conference rules before, at least as regards certain specific topics (such as selection of who will be Speaker, though the current House itself demonstrated how this is not inviolable). Nevertheless, if the "majority of the majority" rule is eschewed, then this toe-the-party-line policy tends to mean that the item in question will only pass if the (often significant) majority of the "yes" votes come from the opposition party. Which may have undesirable optics to the Speaker.
The wiki page previously linked contains several examples of how the rule was received or applied by Speakers since and including Gingrich. Most such speakers were Republican; the Democratic party usually decried the rule as creating needless strife and dysfunction, and an undemocratic excision of the minority party from the legislative process. Though even then some such speakers made numerous exceptions (like Boehner), while others (like Ryan) seemed to stick to it as an absolute.
In practice it is hard to tell if the rule is really being invoked directly or only indirectly. The role of the successful modern Speaker/Leader/Whip in the House is to count votes without ever actually taking a vote. Each member's views and likely votes are handled through private conversations, negotiations, and just general knowledge of their behaviors. As each party offers incentives for members to toe-the-party-line, it becomes very difficult to know if "united party opposition" is truly a united party opposition or simply a majority party opposition, with the rest following the party line. Votes against a measure are usually the result of the negotiations process: members in competitive or highly opinionated districts (on a given issue) will often vote against certain measures deemed risky to their reelection, but the party leaders will have already calculated for this before risking votes, all as part of the calculus of retaining majorities while also passing legislation. Failures are rare, though a recent one is the failure to repeal Obamacare during the Trump administration; one might argue the difficulties in selecting a Speaker for the current House are another. As such, while Pelosi publicly decried the Hastert rule as both Minority Leader and Speaker, it seems reasonable that it was de facto relevant at least on occasion. What we know is that certain Republican leaders have been more publicly open about the Hastert Rule being a guiding principle, with Ryan even publicly promising to apply it on at least immigration bills. It has consistently not been part of the publicly presented governing philosophy for the Democrats, though it seems reasonable to suspect that it has been used in a de facto way on at least a few occasions in order to reach a conclusion.
