Overview
Why are Democrats more concentrated more at a neighborhood by
neighborhood level?
First of all, the trend is real.
Image Source: CDJB
Migration and economics drive this trend.
Partisan segregation holds true, however, without regard to population density and also appears to be supported by migration.
Both Democrats and Republicans are highly segregated from each other spatially, with Democrats tending to live in more concentrated urban areas, and Republicans tending to live in exurbs, small towns, and rural areas.
The current position of the Democratic and Republican party also flows from the two parties reversing their political ideologies over the last few decades.
Migration As A Cause
A leading scholarly journal article on this issue is Will Wilkinson, "The Density Divide: Urbanization, Polarization, and Populist Backlash" (June 2019).
It concludes, as its executive summary states:
Basically, the analysis is that liberals have moved to cities, while conservatives have remained in, or moved to small towns, exurbs, and rural areas.
The same author, in less formal writings, has also hypothesized that in recent times, mass media driving culture (e.g. Fox News and the County Music industry) have homogenized the conservative areas culturally, facilitating their unified politic organization by "Southernizing" them.
Another bellwether new article also supports the migration hypothesis and demonstrates it at an unprecedented neighborhood by neighborhood level. The article and its abstract are as follows:
Segregation across social groups is an enduring feature of nearly all
human societies and is associated with numerous social maladies. In
many countries, reports of growing geographic political polarization
raise concerns about the stability of democratic governance. Here,
using advances in spatial data computation, we measure individual
partisan segregation by calculating the local residential segregation
of every registered voter in the United States, creating a spatially
weighted measure for more than 180 million individuals. With these
data, we present evidence of extensive partisan segregation in the
country. A large proportion of voters live with virtually no exposure
to voters from the other party in their residential environment. Such
high levels of partisan isolation can be found across a range of
places and densities and are distinct from racial and ethnic
segregation. Moreover, Democrats and Republicans living in the same
city, or even the same neighbourhood, are segregated by party.
Brown, J.R., Enos, R.D., "The measurement of partisan sorting for 180 million voters." Nat Hum Behav (March 8, 2021). https://doi.org/10.1038/s41562-021-01066-z
Naturally, of course, in addition to self-segregating, living in areas that have partisan isolation encourages people to adopt the political views of the people they interact with every day.
Risk takers (who collectively help local economies) likewise favor cosmopolitan urban areas. Timur Sevincer et al. , "Risky Business: Cosmopolitan Culture and Risk-Taking" Journal of Cross-Cultural Psychology, forthcoming.
Much of the shift in the political makeup of particular U.S. states is driven by migration as well.
Economics As A Cause
Another key factor is to see economics as driving the change.
There is a well established, nearly universal scaling relation in economics that relates economic productivity to population density. The more populous the city, on a per land area basis, the more economically productive it is per capita. If "you double the size of a city, you generally increase its productivity per capita by between 2 and 5 per cent."
As the economy has shifted, rural areas have depopulated or stagnated growing only slowly in population (half of rural counties actually lost population between 2010 and 2020).
Some of this is due to automation of farming and manufacturing, and offshoring of manufacturing to benefit from cheaper labor and reduced labor and environmental regulation. Some of this is due to economic opportunity being greater in urbanized areas. The availability of remote work has not yet been significant enough to make it even feasible to reverse this trend with remote work. The very high fixed costs of land, equipment, skills, and personal networks involved in beginning agricultural operations have also discouraged migration into that sector of the economy which is a major driver of rural life.
The general shift from rural to urban areas has been more or less relentless and continuous for almost every decade since the Census Bureau first started to keep accurate records, for most of the same economic reasons.
To a large extent this is because farm and manufacturing and mining productive has soared and so rural areas need fewer people, yet produce more products. This represents a remarkable improvement in agricultural productivity in the last half century or so. Between 1950 and 2000, during the so-called "second agricultural revolution of modern times", U.S. agricultural productivity rose fast, especially due to the development of new technologies. For example, the average amount of milk produced per cow increased from 5,314 pounds to 18,201 pounds per year (+242%), the average yield of corn rose from 39 bushels to 153 bushels per acre (+292%), and each farmer in 2000 produced on average 12 times as much farm output per hour worked as a farmer did in 1950.
While a significant number of jobs in exurbs, rural areas and small towns are service sector jobs, those jobs are driven by support for the primary industries (usually agriculture and manufacturing) that power these economies.
In the Rust Belt (formerly manufacturing driven but now deindustrialized), for example, declining jobs have driven stagnant populations and declining housing prices. This is also destroyed the personal wealth of those who have stayed there.
The end result is that counties that voted for a Democrat in either of the last two Presidential elections had about twice the per capita GDP as counties that voted for a Republican Presidential candidate, on average, confirming the nearly universal in time and space and culture economic relation.
Migration (both international and domestic) happens overwhelmingly to places that have more economic prosperity as measured by per capita GDP, driven by work opportunities. Those who move the other way are often those who failed economically in the city for whatever reason, or who are retired and don't need to work and stay in or return to the communities where they grew up. These migration patterns make dense urban areas more diverse as people come from all over to join their economy, and less diverse in rural areas that experience brain drain and have only their historic populations, for the most part.
Affluence and economic security, in the long run and on a cross-cultural basis, are strongly linked to more liberal and more secular values. Economic scarcity and insecurity, in the long run and on a cross-cultural basis, are strongly linked to more conservative and more religious values. See, e.g., the World Values Survey, also for U.S. data here (importance of religion by state) and here (state GDP).
This has been exacerbated by a long term deindustrialization of the U.S. economy since the 1970s in favor of a post-industrial "information economy", that has hit men without college educations who historically made good livings in manufacturing, agriculture, mining, skilled trades, and construction, for example, very hard. Even the military, in the post-Cold War era, dramatically reduced the number of jobs for this demographic by reducing its active duty military personnel levels. Coal mining based economies have underperformed everywhere in the world that they are present. See, e.g., Elena Esposito & Scott Abramson, "The European coal curse" Journal of Economic Growth, Pages 77–112 (March 2021) ("former coal-mining regions are substantially poorer, with (at least) 10% smaller per-capita GDP than comparable regions in the same country that did not mine coal").
This demographic has captured almost none of the last 50 years of economic growth, at a time when more educated adults and women have seen their economic opportunities dramatically improve from what they were in the early 1970s and have captured a significant share of economic growth. (In the case of women, see, e.g., John Iceland and Ilana Redstone, "The declining earnings gap between young women and men in the United States, 1979–2018" Journal of Social Science Research (September 28, 2020) https://doi.org/10.1016/j.ssresearch.2020.102479).
For example, almost all of the inflation adjusted post-tax, post-transfer payment income growth in the United States in the time period from 1979-2011 has accrued to the top 1% of the income distribution. A 2006 study suggests that this is largely due to incredible income gains in geographically concentrated growth in the IT and financial industry (largely confined to ten of the roughly 3000 U.S. counties).
Men without college educations have high rates of unemployment and low incomes in an era when income and wealth inequality have risen dramatically. This has denied these men, and their families, economic security.
This economic stagnation and insecurity has caused divorce rates to soar to unprecedented levels for men in this demographic, and has created economic struggle leading to child abuse and neglect proceedings at high rates sometimes causing these men to lose their children (see, e.g., Christopher Wildeman, Frank R. Edwards, and Sara Wakefield, "The Cumulative Prevalence of Termination of Parental Rights for U.S. Children, 2000–2016" 25(1) Child Maltreatment 32-42 (May 21, 2020) doi: 10.1177/1077559519848499 (open access).
It has driven widespread alcohol, opiate and methamphetamine abuse problems which have led to many deaths of despair and a falling life expectancy in recent years. See, e.g., Hui Zheng, Paola Echave "Are Recent Cohorts Getting Worse? Trends in U.S. Adult Physiological Status, Mental Health, and Health Behaviors across a Century of Birth Cohorts" American Journal of Epidemiology, kwab076 (March 18, 2021).
https://doi.org/10.1093/aje/kwab076
These dire straights have further alienated members of these communities from urban, affluent and educated communities from which they have been excluded which have very different values as a result of their economic and social circumstances.
David Brooks has connected the dots leading to this alienation in a piece entitled "The rotting of the Republican mind" which despite its provocative headline actually shows a great deal of empathy to how economic conditions can drive a conservative mindset and alienation of urban elites.
Thus, it makes sense that Democrats, since they are now the more liberal and secular party, thrives in urban areas, while the Republican party, which is the more conservative and religious party, thrives in rural areas.
Some political scientists have hypothesized that the reality of "rubbing shoulders" with other people, many unlike you, in urban life on a day to day basis also fosters liberal attitudes, but evidence supporting this hypothesis, apart from the general density ==> productivity and affluence ==> liberalism relationships has not been established very solidly, while evidence in favor of migration and this more general economic driver of the difference has grown much stronger in recent years.
Realignment
The Democratic party was historically a rural party, and the Republican party was historically more urban. But the process of realignment has reversed this trend and made the present parties very pure ideologically by historic standards and shifted different demographics to different parties. The chart below illustrates changes over the last twenty-five years or so.