With plenty of caveats, the speaker of the house is who schedules
legislation for a vote
This is not accurate. The House Rules Committee handles this, not the Speaker of the House by himself or herself. But, since the House Rules Committee is also controlled by the majority party, the incentives are more or less identical.
There has been no formal change in the rules on this in recent history, to the best of my knowledge.
So, you are broadly correct in the general gist of what you are suggesting and these incentives are longstanding.
Historically, there was more willingness to entertain votes on bills that did not have majority support from the party, particularly during the 1980s when Congress had a de facto three party system of Republicans, Northern Democrats and Southern Democrats. In that situation, Republicans advanced their agendas in the long run by offering some accommodations to Southern Democrats who often supported their positions, and visa versa, and there was some ideological overlap on a liberal-conservative scale between Republicans and Democrats in the House, something that is no longer the case.
The Rules Committee didn't always have an ironclad grip on which bills would receive votes in the House, but this has been the case for quite a long time, probably since the New Deal in the 1930s, at least.
The level of professionalism and structure in how Congress operates increased greatly after the U.S. Civil War and increased significantly again starting with the New Deal. In both cases, this was because the scope of the U.S. government's role relative to that of state governments increased greatly at those points in time.
Prior to the U.S. Civil War, the federal government was much more minimal in the functions it undertook to perform. This thin agenda reduced the perceived prestige involved in serving in Congress and the need to Congress to efficiently deal with legislation since there wasn't much of it. The need for tight organizational control in the pre-Civil War Congress was also reduced because it had fewer members then.