If one agrees that feminism is adequately described by the statement "Feminism holds that men and women should have equal rights", then a majority of people in liberal democracies, including religious people, are feminists. According to this poll carried out in April 2013 in the United States by the Economist magazine and the polling organisation YouGov, 57% of respondents and 67% of female respondents would describe themselves as feminists when given a definition of feminism very similar to that.
However, the same poll states that only a minority - 28% of respondents and 38% of female respondents - answered yes when asked "Do you consider yourself to be a feminist, or not?" I have read of other polls in other developed countries that gave a slightly higher percentage describing themselves as feminist but still not a majority. I don't know enough about opinions in less developed countries to discuss them.
Clearly there are many people who do not accept that feminism is adequately defined by the objective of equal rights between the sexes, which they generally like; they feel that feminism also includes other aspects, which they dislike.
There is a sense in which feminism cannot be solely defined by its desired outcome, equal rights, which is shared by many other political philosophies which have no special focus on women. For it to be worthy of the name, feminism must have a special interest in women and an analysis that says injustices suffered by women because of their gender are sufficiently important to have a movement dedicated to ending them.
My impression is that much of the explicitly political non-religious opposition to feminism comes from those who for one reason or another dispute that the ending of gender-based injustice is an important political objective. They think not that feminism is wrong, but that it is a distraction.
Some left wingers see feminists as rich women complaining about not getting the very top levels of class privilege while remaining complacent in the face of much greater class-based injustice. Some black and ethnic minority activists make much the same diagnosis but with "white women" replacing "rich women" and "race-based injustice" replacing "class-based injustice". Another strand of opposition to feminism comes from those who believe that classifying people as men or women forces them into an over-simplistic picture of binary genders.
There is opposition to feminism that is based on a belief that the feminist movement has taken on repressive attitudes. In this article from the UK Guardian the radical feminist Julie Bindel worries that feminism is in danger of becoming "toxic".
There are also those who do not accept that in modern society gender-based injustice is suffered by women (or not entirely by them, in a softer version). For instance, they argue, a higher proportion of women than men go to university, women usually get custody of children in divorces, men generally die earlier, and much higher proportions of men do the most dirty and dangerous jobs.
An additional source of opposition to feminism relates to feminist-inspired moves for the watering down of the presumption of innocence and other legal protections for men accused of rape. For example this document from the group "Women Against Rape" urges that there should be no prosecutions for false allegations of rape, ever. Concern over this specific issue prompts much non-religious based opposition to feminism as it is seen by many.
Edit, 23rd February 2021: The link to the document from the campaign group "Women Against Rape" I referred to in the above answer has now gone dead. However it can still be read on the Internet Archive / Wayback Machine here. The time of first publication seems to have been June 2011.
The document is an open letter addressed to the then UK Director of Public Prosecutions from Lisa Longstaff of the campaign group "Women Against Rape" and co-signed by 30 other organisations and 15 individuals. It begins, "We strongly believe that the prosecution of women for alleged false allegations of rape is not in the public interest." It then lists several reasons why the signatories of the letter held that opinion. At the time the letter was written, the post of UK Director of Public Prosecutions was held by Keir Starmer. Note that his becoming first an MP then leader of the Labour party (and also becoming Sir Keir Starmer) all lay in the future.