tl;dr They never said anything about it.
The framers likely didn't envision the courts playing as a large role as they do today.
Thomas Jefferson (not technically one of the 55 framers, but influential on them) went so far as to say that the new universally accept notion judicial review itself was nonsensical:
If this opinion be sound, then indeed is our Constitution a complete felo de se [act of suicide].
For intending to establish three departments, coordinate and independent, that they might check and balance one another, it has given, according to this opinion, to one of them alone the right to prescribe rules for the government of the others...The Constitution on this hypothesis is a mere thing of wax in the hands of the judiciary, which they may twist and shape into any form they please.
While this was not the opinion of the Federalists, suffice it to say, in general the courts were not seen as politically significant as they are today.
As far as I know, no framer discussed the possibility -- for or against -- of increasing the size of judiciary for political purposes. There were controversies over appointments, but not the size itself.
Yes, the size of Supreme Court changed over time. That is was because the structure assigned Supreme Court Justices to circuits, making their number proportional to the number of circuits need to address cases.
But not until 1936 was there serious suggestion of altering court size to produce favorable outcomes.
So in short, we have no idea what the framers thought about this.