The simplest answer is that your question is not correct in fact: you do not live in a “progressive” society.
are supposed to be completely irrelevant and we should
Should statements are usually indicative that whatever “ought” to be, isn’t.
What's the reason behind this conflict?
For hundreds of years within capitalism and protocapitalism there were immediate material rewards to the reproduction of “race” and “gender” as socially oppressive structures. These included latifundia slavery and the family as reproducer of wage labour, for example.
However long run processes of mechanisation and skill dilution, alongside struggle by workers and bourgeois who opposed these oppressions, have resulted in a situation where cultural lip service must be paid to the principle that all persons are created equal in liberal democratic states with high levels of mechanisation. The risk of missing out on a particularly skilled worker due to her race is too great to afford. The risk of union or media ire for discriminating too obviously against a woman for her gender is unaffordable. Moreover, with changes to the labour force, a household is now expected to supply 1.6 workers or more (both parents, one part time).
These changes can’t and don’t erase history or culture. And they don’t erase economic incentives towards racism such as the prison labour complex.
What prevents politics from being completely neutral and not taking the person's physical characteristics into account?
Because people aren’t neutral. People’s cultures aren’t neutral. Gender and race are deeply embedded categories and relationships that exist overtly and covertly and which are continuously reinforced (in old and new ways) by economic incentives.
“Oughts” don’t cause “is” without time, effort and energy. And the expenditure so far hasn’t overcome hundreds of years of economic and cultural reinforcement. Debate amongst scholars shows significant doubt that these relationships can be eliminated, as opposed to ameliorated, within a wage labour society.