The Conservative government, even Labour, have talked about reducing immigration because they know it's what a lot of the electorate want. This is why one of the Conservative party's 2010 election pledges was to reduce net immigration "from hundreds to tens of thousands".
Xenophobia has been a long running problem in Britain. After the second world war the economy was booming, but there were labour shortages. So the government sent recruiters to the Caribbean to find people who would come to work in London. That however invoked a bristling racism, and by the late 1970s the National Front became an influential if minor political force, exploiting economic decline to play on working class fears.
The economy is still used to justify this sort of sentiment, which means that parts of the UK left behind by progress have grown bitter. By the end of the 1970s the Winter of Discontent proved that the left wing establishment had lost control of the economy and their members. Far left union agitation led to economic paralysis, and thus Thatcher's 1979 victory. She then went about dismantling socialism, and in doing so gutted entire communities in the industrial heartlands; predominantly in middle and northern England, built around mines, mills, factories. The poverty, desperation, and anger from that time lasts to this day.
These impoverished and uneducated communities have found it hard to survive and thrive against rising living costs and increasing competition for jobs. They feel that government does not care, and indeed is prioritising immigrants over the white working class.
But that's not the end of it. Another problem is the perception of a fundamental conflict of values between white British and immigrant communities, and this segregation is getting worse. Indeed in some places communities live side by side without mingling, which further fuels distrust. Many of the older generation feel alienated by how much Britain has changed since they were young.
And even many of those neither xenophobic, nor poor, nor uneducated, nor old, are worried about the consequences of unregulated immigration. It has been suggested that it's not that Brits think immigrants are bad per se, but that the system lets in too many of the wrong kind; immigration of skilled versus unskilled workers.
What this all adds up to is the fact that the benefits of immigration for many are besides the point, or completely irrelevant. For many immigration is perceived as an existential threat for different reasons that are often lumped together; to their prosperity, community, or very identity. Whether any of that is true or not is besides the point, and some want society to be more homogenous even if it makes them poorer.