Let's presume there is no voting strategy. So everyone picks the one that makes the most sense to them.
But that's not how people actually vote. Libertarian voters are entirely capable of recognizing the issue you propose and choosing to vote for the Republican over the Democrat. This pushes the Republican party to be more libertarian on economic issues (where Republicans are more like libertarians) and moral issues (where Republicans are less like libertarians). And it pushes Democrats the same way, although the similarities are different. There's even a term for this: Duverger's law:
In political science, Duverger's law holds that plurality-rule elections (such as first past the post) structured within single-member districts tend to favor a two-party system
In your model, it's clear that a libertarian party would make the system less libertarian. But that may be because your model doesn't reflect reality particularly well. If you are trying to make a statement about reality rather than one that holds only in your thought experiment, you have to work with realistic measures.
This is also true of Duverger's law. For example, the United Kingdom has plurality (first-past-the-post) voting and single member districts. But it has at least six parties with multiple members elected to parliament. So we have to revise our understanding of Duverger's law so that it only affects parties in each district. So the United States with its national presidential district is restricted by Duverger's law, but the UK without overlapping districts is not as much (although generally only one or two parties are competitive in each district).
And this has a name as well: All models are wrong, but some are useful. Basically, all models are simplifications of reality and therefore at least partially wrong. But a model may still be predictive of reality and therefore useful. In this particular case, Duverger's law seems more predictive of behavior in the US than your model is.