Economic migration (temporary or more permanent) is considered the main factor. This is quite apparent in the Eastern EU countries. Note that the official EU terminology is "mobility" not migration, inside the Single Market.
Bruegel has roughly the same info as a graph over time.
Something a bit similar happens to Ukraine's workforce, even though they're not in the EU, but in a certain partnership.
I did find some expert opinions:
Eastern Europe is set to have all but one of the world’s fastest-shrinking populations over the coming decades, according to official estimates.
The number of people in Bulgaria will tumble the quickest, says the United Nations, from 7.08 million now to 5.42m by 2050.
A swathe of other eastern European countries will suffer a similar fate: from Latvia and Lithuania in the north to Romania and Moldova further south.
Experts say falling fertility rates and economic emigration are combining to reduce the number of people in some eastern European countries.
The fertility rate of European Union states has fallen from 2.6 births per woman in 1960 to 1.6 in 2015.
Poland, whose population is predicted to fall by six million over the next three decades, had a fertility rate of 1.32 in 2015. It is one of the lowest levels in the European Union - only Portugal’s was smaller.
It has prompted a rather drastic — if comic — remedy: the country's health ministry released a video urging its citizens to 'breed like rabbits'.
Emigration has also played a major role in the falling populations. [...] This has especially been the case for the most-recent countries to join the EU, such as Bulgaria and Romania.
Brussels-based economic think-tank Bruegel claimed in a recent report that the population of young people in Latvia and Lithuania dropped by up to 25% between 2008-2014 as a result of emigration.
“Emigration is certainly a major factor,” Bernd Parusel, a migration expert with the European Migration Network told Euronews.
“Especially young people tend to leave, to study or work in other EU member states or even farther abroad.
The drop in fertility itself is probably correlated to that age profile of those who leave, i.e. the older people who stay probably aren't as likely to have as many children.
If that's not convincing enough, I'll try to find some studies that actually correlate depopulation with the aforementioned processes.
Some IMF researchers had a paper on this (which I found via an article in The Economist) which apportions migration effects on population growth. Alas their data only goes up to 2012 (despite the paper being published in 2016):
The scale of emigration from CESEE countries since the early 1990s has been
staggering. During the past 25 years, nearly 20 million people (5½ percent of the CESEE
population) are estimated to have left the region (Figure 2). By end 2012, Southeastern Europe (SEE)
had experienced the largest outflows, amounting to about 16 percent of the early-1990s population.
Emigration has also been persistent—annually, reaching as high as ½–1 percent of the 1990s
population—and has tended to pick up following each new wave of EU expansion in 2004 (Estonia,
Hungary, Latvia, Lithuania, Poland, Czech Republic, Slovak Republic and Slovenia), 2007 (Romania
and Bulgaria), and 2013 (Croatia). The non-EU SEE (SEE-XEU) countries have seen easing access for
their citizens for travel to Western Europe, and thus stay and work.
Emigration significantly lowered population growth in sending countries, in some
cases worsening already negative demographic trends. Between 1990 and 2012, outward
migration from SEE shaved off more than 8 percentage points from cumulative population growth.
While these trends were partly offset by strong population growth in some CE-5 (CE-5 refers to the
Czech Republic, Hungary, Poland, Slovak Republic, and Slovenia) and SEE countries, emigration has
aggravated already pronounced negative demographic trends in the Baltics and some
Commonwealth of Independent States (CIS) countries. As a result, local populations in most
countries in the region have been stagnant or shrinking.