Generally it is not really polled among the voters as the people who are upset with the President for golfing are the hardliners from the opposition party while the people in support tend to be a combination of the hardliners of the President's Party, centerists and swing voters, and moderates of the opposition party.
The people who think it's inappropriate for the current president to play golf are likely to be cool with it when it's their guy in office and at least two Presidents (Kennedy and Trump) both were critical of their predecessor's amount of golf, only for them to play some rounds upon taking office. Kennedy did try to hide it, but "the guy who was President before me played too much" ranks as one of the rare things they would probably have agreed on.
Closer to center, most Americans recognize that the White House these days is a really fancy jail cell someone and his family in for the crime of being elected to lead the nation, so they tend to be a bit more understanding that the current President can get Cabin Fever from the security restrictions on his movement and that the job is really stressful and the President looks physically less healthy on exit then on entry (Obama was not grey haired when he took office on 1/20/09, but come 1/20/17 he had gone straight past grey to white haired.) and that even on vacation, the President is still surrounded by security and a big red "launch nukes" button. Most people tend to be accommodating to the president having me time from time to time, while in office.