The core concept here is procedural fairness—or rather, the appearance of procedural fairness.
Procedural fairness involves whether impartial and open procedures are used when decisions affecting the well being of others are made. Is the decision-maker impartial? Is the game rigged? Procedural fairness is crucial for the health of a democracy because when people have faith in the system, they are much more willing to accept outcomes that are disadvantageous to themselves.
A public servant who holds strongly partisan political views could, in theory, allow their politics to seep into their work, and twist a part of government that is supposed to be neutral to give their party an unfair advantage. Of course, they could also do their job impartially and not allow their personal views to influence their work—and most of them do. And even if a public servant is a "bad apple", the mechanisms of government may have built-in checks and balances that limit the damage they can do, thus ensuring procedural fairness.
But if a public servant puts their political views out into the public sphere, some people will inevitably think that they're not being impartial. This has nothing to do with if they're actually being impartial, or if procedural fairness is actually being upheld. It's all perception. And this perception of corruption does its own kind of damage.
Consider, as an example, the case of FBI agent Peter Strozk, who expressed some strongly anti-Trump opinions in private text messages while simultaneously helping to carry out an investigation into President Trump's ties to Russia. The texts leaked, and Trump and his allies raised hell. They accused the FBI of deep corruption, saying that Strozk and others were conspiring against Trump and deliberately seeking to keep him out or kick him out of office by any means necessary. The counterarguments that Strozk was just one figure in a larger investigation, or that there was no evidence of him breaching standard FBI procedures, did not satisfy them.
In other words, Trump argued that procedural fairness was missing in the FBI's investigation. And whether you believe that's true or not, a lot of people did believe it was true, and probably still do. Which means that if they ever see any evidence against Trump coming from the FBI, they will automatically dismiss it regardless of its merits, because they believe that the FBI is corrupt.
And those were private text messages. Can you imagine what would have happened if Strozk tweeted those opinions?
To specifically address the points from Zeus's comment:
it could be easily argued that gagging public servants doesn't actually help the situation
Yes, public servants refraining from sharing their political opinions does not actually ensure procedural fairness. That is done through other mechanisms. Their silence merely safeguards the perception of procedural fairness. But the perception is important too. People will not trust a fair game that looks rigged, any more than they will trust a game that is actually rigged.
like all humans, they will still act according to their biases, just quietly. Transparency is always better.
If the mechanisms of government are well-designed, with appropriate checks and balances, then the operation of that government will be detached from the personal opinions of the people carrying it out to the greatest extent possible. It's never going to be 100%, but after a certain point, being "transparent" about personal views will just lead people to imagine that those personal views are more important than they actually are. When a public servant calls the Whig president an idiot, and pledges undying loyalty to the Bull Moose party, many Whig voters either won't know or won't care that the servant's job is designed to make that opinion irrelevant. They just hear that a person who is supposed to carry out the Whigs' marching orders is rebelling.
And that's before we even consider 'rights'.
It would be perfectly reasonable to argue that preventing public servants from expressing their opinions as private citizens is a violation of their rights to free speech. However, the question as asked is "why should public servants be apolitical", not "is it legal or ethical for the government to force public servants to be apolitical". I am not weighing in on whether the silence of public servants should be forced or voluntary; I am just trying to show why that silence is a good idea.