In democracies like the UK, where ministers have considerable executive power, these ministers are still restricted (by law or convention) in their ability to get directly involved with the work of civil servants in their ministry, let alone replace key civil servants with "their own" people. Why is that? I understand the argument that this would provide opportunity for corruption, but if we assume for the sake of argument that we could somehow set up a system that prevents corrupt nomination, are there other good reasons to block this? For example, let's say a very socially minded Lord of the Exchequer is appointed, but the entire ministry is geared towards right-wing Neo-liberalism. Would it be unbecoming if he felt the entire staff needs a "shake-up"?
This is part of the political traditions of the UK.
A benefit is that the civil service can hire some of the most able graduates from the top universities on the understanding that they won't lose their job if and when the government changes.
It has the advantage that a civil servant can give advice that a politician doesn't like without worrying about retaliation by the politician. The civil servant can be honest and doesn't have to "suck-up" to the politician to get a promotion.
It means that the civil servants are experts. The civil servants can give advice based on years of experience. If the incoming administration appoints a new civil service, they would lack this experience.
It has the disadvantage that there is a group of extremely powerful government forces that the democratic process cannot remove.
There is a tradition that your scenario of "a neo-liberal civil service" should not occur, as the civil service is supposed to be politically neutral.
That is, there is a tradition that the civil service does not set the political agenda, but works to make the plans set by the Ministers as effective as possible. However the
documentary comedy, Yes Minister portrays a civil service that is obstructive and concerned with maintaining the status quo.
Loss of institutional knowledge.
This plagues the US quite severely because a lot of even mid-level positions are replaced with each administration.
For a regular job, half a year is considered wasted before a new recruit becomes effectively productive in an otherwise productive/already-working environment. Aka this new employee (or here civil servant) can just ask their colleagues whenever something not in the training manual comes up or is unclear (which is a lot).
If you replace an entire department all at once, there is nobody to ask, leading to an insane loss of productivity, stuff just being left behind, simple shortcuts forgotten and so on.
This means that effectively, every 4 years (if a switch happens), large parts of the administration grind almost to a halt and only slowly pick up their work again. By the time everyone is comfortable in their roles they might be kicked out again.
On top of this, plans beyond 3-4 years (or whatever the term length is) need to be watered down and simplified so that a) it's OK for the current ruling government b) doesn't get cancelled by that positions replacement and c) can be effectively continued by a random person with little to no handover.
In contrast to this, Germany for example has only few politically appointed/recalled positions at the very top of ministries/departments, meaning that even after a government change, life just goes on and new directions/priorities are slowly introduced from the top down and running plans/projects are amended rather than forgotten or cancelled.
This also means that more positions are filled by people who got there based on their merits rather than political opportunism, leading to more people with work and/or academic experience in the relevant field(s) in more senior positions.
Edit: This was a bit one-sided. The massive downside of less appointees is that elected officials have significantly less (or slower) influence on day-to-day business, removing the state as an institution a bit from democratic control. Whether this is a bonus (radical politics have to go through the legislative system first) or a disadvantage (the state can feel detached from the elected officials) is an open debate.
I am not sure how internationally relevant this is, but in Germany one reason is the status of civil servants as "Beamte", which broadly means they have a right to employment until retirement. Either another post of similar standing has to be found, or payment has to be continued (lifelong!) despite the person being in "provisonal retirement".
If a minister could relieve them at will, the costs for his administration could grow beyond any reason.
(Note there is a small group of leading servants called Parlamentarische Staatssekretäre or Staatsminister that are appointed a temporary status as Beamte that will expire at the end of their term. Those can be exchanged freely by their ministers.)