You probably have a number of factors going on.  It is surprising difficult to find any solid studies however (I'll try to find some later).

Mostly answering wrt the US Libertarian position, since I don't know if too many other countries have a significant explicit party in this domain.

##### Racism (by some of its members)

[The Hill oped - blames it on racism](https://thehill.com/opinion/civil-rights/3680007-the-libertarian-party-is-collapsing-heres-why/)

He probably has a point in practice, but I would not say racism is a feature of Libertarianism per se.

>In May, the party was taken over at its national convention by the so-called Mises Caucus, a far-right group, some of whose members have been associated with racist and antisemitic ideas. The caucus is named after the libertarian economist **Ludwig von Mises, whose philosophy was pretty crude (as I explained in the book) but who firmly condemned racism.**
>
>On Martin Luther King Jr. Day this year, the Libertarian Party of New Hampshire tweeted (in a later deleted post) that “America isn’t in debt to black people. If anything it’s the other way around.” Caucus members have called for violent repression of antifa and Black Lives Matter protesters.

##### [ party platform - Positions that do not appeal to too many people](https://www.lp.org/platform/).



>Parents should have control of and **responsibility for all funds expended for their children’s education.**
>
>Retirement planning is the responsibility of the individual, not the government. **Libertarians would phase out the current government-sponsored Social Security system and transition to a private voluntary system.**
>
>To further that end, we call for the **repeal of the income tax, the abolishment of the Internal Revenue Service and all federal programs and services not required under the U.S. Constitution.**
>
>We oppose the administration of the death penalty by the state.

It's not hard to see how some of these are **not** going to be very appealing across the US ideological landscape.  Reform of pensions for example is a political live-wire *everyone* avoids.

Seems to me that US Libertarians are stuck in the same kind of rut Germans Greens wrt ["fundis vs realos"][1], with the fundis being in the ascendant.  People often talk about big-L Libertarians - those are the people who are not getting elected.

However, as an influence on politics in general, especially in the US, small-l libertarians, advocating for a smaller state, do have some influence within the Republican party.

Which is, to some extent, not that dissimilar how Green parties struggle to get elected on their own ticket, but manage to get quite a few of their principles adopted by mainstream parties.

  [1]: https://www.theguardian.com/politics/2005/sep/21/uk.libdems20051