Former US Senator and Virginia Governor Harry F. Byrd Sr. (Dem) once said "I stand for strict economy in governmental affairs". Is this a well known phrase? From where does it originate?
It's probably impossible to find the origin of either the phrase or the underlying concept because they are "folklore" probably as old as elected governments, and perhaps even before. Just to stick to the US, they go back at least 100 years, e.g. to the Coolidge administration (but who was probably hardly the first politician to emphasize that angle).
To many Americans, Calvin Coolidge embodied the frugality they sought in their lives. The image he presented in numerous photographs and films was that of a simple man who endorsed plain living. Pictures of him as a rural Vermonter working in the fields of his family's Plymouth Notch farm emphasized traditional values and thriftiness and allayed popular anxieties about excess and indulgence. It was an image that served him well and that he actively promoted in his electoral and public-relations campaigns.
In keeping with his image, Coolidge's great policy concern was economy in government. He assumed office in August 1923 upon the death of Warren G. Harding and served as president for six years. During that time he concerned himself with such measures as paying off the national debt, eliminating waste, and cutting taxes to stimulate capital investment.
And not a politician himself, but probably influential or at least representative in that regard of the zeitgeist was Francis Wayland, a 19th century Baptist minister and president of Brown University, who wrote several books e.g. Elements of Political Economy (1837) emphasizing among other things thrift in taxation and in government spending.