To Protect Domestic Job Markets
One reason that country's dislike free movement for workers is because there is a risk it will negatively impact their own domestic job markets. Immigrants may displace citizens, which is a problem for governments which both pay social benefits (like unemployment benefits) and require the support of citizens to stay in office.
Interviews with union leaders in 4 European countries identified several factors which can influence this choice (Krings, 2009):
Unemployment: Countries with high unemployment are less likely to support freedom of labor agreements. When unemployment is high, people may be more sensitive to job competition. However, these concerns don't appear to be validated. A study by the UK Department of Pensions found no connection between immigration and unemployment claims (Gilpen et al., 2006).
Wage and Standard of Living Differences: Large wage or standard of living differences may be associated with less support for free movement of labor. When these differences are large between countries, there are stronger incentives for people to immigrate. However, in these cases immigrants will generally take jobs that are unattractive to domestic workers (Werner, 1990).
When these factors aren't present (that is, when unemployment is low and there are no large wage or standard of living differences between countries) there is likely to be greater union support for free movement. Kring's (2009) analysis of UK and Irish union leaders found this to be the case.
There is such an agreement within the EU
-> And the wage levels are definitely not similar within the EU, Bulgaria has wages ~10 times lower than Luxemburg.