The high level reason is fairly simple. DNC poses more of a threat to libertarian ideals and way of life than GOP.
The main philosophical concern of Libertarians is reduction of violence (or a threat of violence) and coercion in political life. All of the other things that characterize libertarians stem from that high level point.
In practice, the main source of said coercion since at least 1930s comes from the government; therefore in practice, this means that achieving this philosophical goal requires limiting the power of the government.
Between the two parties, the DNC is clearly 100% for increasing the power of central government - both as a goal and as means to achieving other goals. Whereas GOP is - while far from anarcho-capitalists - significantly less so (they still tend to grow government power, and wish to use it in politics for their ends - which is why they are Republicans and not libertarians - but their drive to do so is significantly less both in practice AND on ideological/philosophical level).
Even in areas where GOP does seem (or is) to be supporting government interference, it is VERY frequently in areas that many libertarians either don't view as undue interference, OR where they disagree with the underlying premise by both parties, or both GOP and DNC are not all that different. To whit, the four main "government interference" items that GOP is usually accused of are: abortion, gay marriage, immigration, and military intervention abroad. Let's look at each one in detail:
First of all, libertarians don't support EITHER side here. Most of them want marriage equality via the means of turning marriage into a private contract between consenting adults, with ZERO involvement by the state.
More importantly, while virtually no libertarian I'm aware of supports prohibition of gay marriage (e.g., if they can't get their preferred outcome, they prefer DNC's approach over GOP's), many of them simply view it as an issue to small in importance compared to other ones when choosing between DNC or GOP.
I don't have polls to back this up, but a simplified version of what I just said would be: A libertarian is likely to vote FOR "pro-gay-marriage" specifically - e.g. in a referendum - but vote against politicians ("D") who are for gay marriage for reasons that have nothing to do with gay marriage but with the politicians' other policies.
It may surprise you, but for the vast majority of people opposed to abortion, that opposition has nothing to do with "wanting power over a woman's body". Nearly 100% of people opposing abortions view the aborted fetus as a live human being, and therefore view abortion as nothing short of killing a human. When viewed from that angle, opposing abortion is nothing more than opposing killing, and therefore is fully within the very few powers that the state SHOULD have, namely protecting lives via law enforcement.
Note that some have such view for religious reasons, some have such view for purely scientific ones (e.g., for a fetus in a stage late enough that it would have survived in nICU if delivered prematurely, it's hard to make an argument that merely being attached to a placenta and not to nICU life support somehow turns the fetus from a live human being to "perfectly fine to surgically excise part of mother's body". So saying that "but for GOP it is a matter of theocratic domination" is a strawman.
This is another one where purely practical considerations matter.
Yes, in an ideal world, libertarians prefer unlimited immigration (citations to position statements trivial to find so I'll skip them).
However, that ideal world includes one very important stipulation, that the government won't compel anyone currently residing in the country to help provide for the living of the people who choose to immigrate.
As it is, that's not the case, and the immigration model as it currently stands serves to strongly enhance the welfare state, since all those immigrants get to be taken care of by the state (health, education, other welfare payments) if they are allowed to reside in USA legally without restrictions. You don't see Democrats offering to trade amnesty for "no welfare/social payments to any newly legalized people, ever", do you?
OK, this one is pretty much one of the main areas of disagreement between many libertarians and the hawkish parts of GOP.
So why don't libertarians vote for DNC because of this? Reasons vary between different people, but some of them are:
It's not like Democrats (as governing party, not as antiwar demonstrating individual voters) are terribly more peaceful compared to GOP. JFK was more of a hawk than Nixon, LBJ was deeply and fully committed to Vietnam War even if he didn't start it [ed. note JFK did](LBJ subscribed to Domino theory); Clinton military intervened in Serbia and Kosovo, and shot cruise missiles at Sudan and Afghanistan; Obama intervened in Syria, Libya, continued in Afghanistan and re-introduced troops into Iraq and greatly expanded the drone program of George W. Bush.
And having left-wing government never stopped USSR from invading Afghanistan or being an aggressive military superpower. [Ed. note: Marxist theory calls for revolutionary terror as a "necessity".] Plus, for all their condemnation of US foreign adventurism since Bush, the socialist government of France happily got themselves involved in the war in Mali and before that, started the war in Libya.
Some people simply take a more pragmatic view that defense of the country is the proper (only one of 2 proper) jobs of federal government, even if they are libertarian. Therefore they may object to specific foreign policy steps but don't have any inclinations to vote DNC just because they are "less militarily inclined". Remember that Eisenhower was a Republican.
Some simply take the long view. ANY large and powerful government is more liable to be more militaristic, that's just how history shows the world works. Therefore, going back to the beginning of this answer, to them stopping Democrats from growing the power and size of government is actually synonymous with long term having less militaristic government. See USSR or Cuba.
There are plenty of other topics where libertarians disagree with GOP but DNC would not be much of an improvement in practice based on observed history (war on drugs/drug legalization, prostitution legalization, SWAT and aggressive police tactics, censorship etc...)
To view it another way, look at the stated ideal end goal (leaving aside that in reality the end state would differ from that end goal).
GOP wants the country to be the way USA was in 1950s, sans segregation/racism.
DNC wants the country to be the way France is now.
Which of the two countries do you think a random libertarian would prefer to live in, 1950s USA sans segregation/racism; or 2000s France?