Has anyone attempted to scientifically determine whether MSNBC or Fox News is more biased?

Question

I'm wondering if there is any study or other serious attempt which compares Fox News to MSNBC to see which one was more 'biased' towards their respective sides of the political spectrum.

I am not looking for opinion here, I'm looking for serious analysis that presents a criteria that can be (at least reasonably) objectively measured, has measured the two respective media outlets, and have presented results as to which outlet best met or failed those criteria.

I'm open to any criteria that can be used to measure bias, from number of factual inaccuracies in reporting, to how often they call the other side immoral and evil, to any other criteria that could be imagined as being a measure of, or result of, bias for their preferred political view and/or a sign of lack of trustworthiness in their reporting due to such a bias. All I really care is that it can be objectively measured and concrete results presented.

Answer 1

I'd say anyone who tells you they have "scientific" information on media bias isn't being objective. But it is entirely possible to present a "systematic" information on media bias, and at least one online source with that goal is https://mediabiasfactcheck.com. They have an explicit methodology, which says:

When determining bias, there isn’t any true scientific formula that is 100% objective. There are objective measures that can be calculated, but ultimately there will be some degree of subjective judgement to determine these. On each page we have put up a scale with a yellow dot that shows the degree of bias for each source. Each page also has a “detailed report” section that gives some details about the source and an explanation of their bias. When calculating bias we are not just looking at political bias, but also how factual the information is and if they provide links to credible, verifiable sources. Therefore, the yellow dot may indicate political bias or how factual a source is, or in many cases, both.

Mediabiasfactcheck has a high level summary graphic which may be helpful: For example, MSNBC looks like this:

Fox News looks like this:

But mediabiasfactcheck.com goes into detail as well, and includes a systematic categorization of different kinds of journalistic bias.

If what you are looking for is unbiased news, recognize that's not the goal of either organization. Some news sources actually make reduction of bias and emotionally loaded reporting a policy goal. Take for example: Reuters News looks like this:

Note well: its impossible to to express an opinion on news source bias without having some bias, and getting accused of being biased. Mediabiasfactcheck.com is no exception: some other organizations rate mediabiasfactcheck.com as biased. Oh well...

Answer 2

Many studies have been conducted to test exactly this question. This study from 2012 asked participants 4 factual questions about international events, and 5 questions about domestic ones (questions like "It took a long time to get the final results of the Iowa caucuses for Republican candidates. In the end, who was declared the winner?")

Nobody did very well on the survey-- even the most well-informed group averaged less than 2 correct answers in each category. The highest average were people who got their news primarily from NPR. Fox News performed the worst in both domestic issues and international ones, even worse than the "no news" group. MSNBC was the second-lowest, coming in just under "no news" for international issues and just above "no news" for domestic issues.

Studies like this one usually produce similar results. Some researchers call this "the Fox News effect", though a similar case could be made for calling it "the MSNBC effect". So the best empirical evidence we have suggests that Fox is worse than MSNBC-- but not by much.

Answer 3

I'll take you through a Thought Experiment to show why this isn't really possible.

Imagine you were trying to judge whether networks A, B, and C were 'biased' on the coverage of, say, "Should the US offer aid to the rebels of [country name]?"

So, you might think, well, they should have facts from all sides. Present all the facts. Have people on to discuss any point of view on the issue. Give equal time to each side. Don't exclude any news that sheds either side in a positive light. Etc.

... now let's change those 'rebels' to a genocidal group that utilizes a child-army and practices slavery.

Suddenly, things change. If your network gives facts/opinions/equal-time to the genocidal/slaver group, it suddenly feels a lot less 'unbiased' and starts to feel a lot more like they're giving legitimacy to something horrible.

Do you now see the discrepancy? You're now in a position of having to figure out, for this particular issue: how legitimate is each side? Because you can't cover both sides 'equally' and have it be balanced if only one side has a legitimate cause. It's not like someone would expect News Co. XYZ to be balanced by giving equal time to a NASA Spokesman and a rep from the Flat Earth Society.

Let's take an actual issue: Brexit.

News Organization A covers Brexit by giving the facts from both sides, giving equal time to both proponents and dissenters. Organization B has more pro-Leave airtime/arguments; Organization C has more pro-Remain. So which ones are biased?

Well, how legitimate is each side? You could say Organization A is non-biased... except it means you've given each side roughly equal legitimacy. Is that fair? How do you tell? In any sort of non-subjective manner?

Now multiply this by the number of issues that people argue about. And... add in a dash of an uncomfortable truth: generally most popular political topics do indeed have at least some legitimacy on both sides (which a lot of people don't like admitting, since it makes it harder to viscerally hate the people that disagree with them.)

So... how would you design that experiment now? You can't - because "balance" depends on your subjective measure of each side of each issue.

Answer 4

AllSides.com has a nice list of bias ratings along with a community rating of how accurate that label is.

From their Media Bias Ratings page:

Our Media Bias Ratings are determined using multiple methods and represent the average judgment of Americans. They are based on blind surveys of people across the political spectrum, multi-partisan analysis, editorial reviews, third party data, and tens of thousands of user feedback ratings. Our scientifically-generated ratings are fluid and subject to change as new information is gathered and biases change over time.

MSNBC rating (2020/02/03):

Fox News rating (2020/02/03):

Answer 5

Along the lines of the excellent answer from Burt_Harris, you might be interested in Ad Fontes Media

They rank a wide array of news (... and, not-really-news) sites.

The reason I reference the answer from Burt_Harris is because I agree that it's difficult to have a strictly scientific approach to this, but Ad Fontes Media has a systematic approach, meant to produce reliable results. For detailed information, see their methodology page.

Answer 6

EDIT

some other answers suggest political bias is not objective, probably because they are confounding "preference" with "observation of preference (bias)". Ambiguity in natural language obscures the issue, but objectivity is possible. Preference is subjective, but measurement of preference (i.e. observed bias) is objective. For example, the question is red favorable to green? probes subjective qualial preference, whereas Do young adults in Europe prefer red over green?, though difficult to measure, probes objective bias. Likewise, favorability of party X is subjective, but observed favor for party X is objective. Also, objectivity is not confined to non-stochastic phenomena. In cases where objective phenomena are difficult to measure, clever scientists will search for plausible proxy metrics that are easier to measure, as is the case with the following:

ORIGINAL POST

Please see this publication from UCLA, which estimates ADA scores for major media outlets. According to the introduction:

...we count the times that a media outlet cites various think tanks. We compare this with the times that members of Congress cite the same think tanks in their speeches on the floor of the House and Senate. By comparing the citation patterns we can construct an ADA score ... These findings refer strictly to the news stories of the outlets [and not] editorials, book reviews, and letters to the editor from our sample

The publication also contains a section Previous Studies of Media Bias

It's worth considering possible shortcomings of the article's described methods. Firstly, you have to be willing to accept that think-tank citation ratios that coincide with those of congressional members are an acceptable proxy for media outlet bias. This requires that we accept other assumptions prima facie, like a consistent manner of think-tank citation behavior across the political spectrum, etc.

Still, this is, to date, the most serious analysis I've seen. As regarding objectivity, the described methods seem sufficiently detailed so as to be repeated, and hence the results independently measurable and verifiable. I've intentionally omitted their conclusions - if you want to know, read the article ;)

EDIT

Here are two other studies I found:

1. WHAT DRIVES MEDIA SLANT? EVIDENCE FROM U.S. DAILY NEWSPAPERS
2. Fair and Balanced? Quantifying Media Bias through Crowdsourced Content Analysis

The first tries to compare "unique phrase usage" analysis of media outlets to members of Congress. I'm skeptical of the second, which tries to recognize bias with machine learning techniques, trained with data procured by a set of volunteers. They might have sufficed with just generalizing the volunteers' results since the AI can only ever (likely) perform as well and no better than it's training set provided by flawed humans.

And here is a data source of news chyron archives used in some more serious analyses.

Finally, an article probing the meta-concerns of determining media bias.

Answer 7

I am not looking for opinion here, I'm looking for serious analysis that presents a criteria that can be (at least reasonably) objectively measured

Here's an objectively measurable criterion: what percentage of people employed by the agency cut their toe nails once a week?

Asking "has anyone come up with an objective measure of X?" makes no sense. If X is itself not an objective measure, there is no objective way to assess if your objective measure is a meaningful measure of X.

The question you should really be asking in situations like this is "What would be the purpose of such a measure?" What makes lower bias preferable? This will tell you what you should expect from an objective measure of bias. The same applies, by the way, to "factual reporting". How exactly do you wish to weight each statement's truth value? What counts as a statement by a news organisation/how discrete do you go?

I don't think there's a good definition here, though. Gotcha side-by-side comparisons of analogous articles by the same outlet are the best you're ever likely to see on this.

Answer 8

Ad Fontes Media produces the following chart, which seems to be well researched:

MSNBC seems to be roughly on par with the FOX News channel as a whole in terms of partisan slant, though MSNBC is positioned as being a more reliable news source. Note that FOX news programming (separated from the punditry of the rest of the channel) is viewed as equally reliable to MSNBC, and slightly less partisan.

Answer 9

There have been studies trying to define media bias and make that measurable:

https://www.nature.com/articles/s41599-024-03143-w

https://www.sciencedirect.com/science/article/pii/S0957417423021437

https://www.sciencedirect.com/science/article/pii/B9780444636850000152

Why anybody in their right mind would waste their time and energy on clear cases like Fox News and MSNBC is beyond me.

Like just picking random articles from their website:

Apparently the opinion of an MSNBC columnist is openly for the democratic candidate or at least against Trump. https://www.msnbc.com/opinion/msnbc-opinion/jon-stewart-tried-warn-us-joe-biden-rcna160914

Now you can decide whether that is a "hard left" bias or just a reasonable position giving that Trump is coy with dicatorship and that his disavowals of Project 2025 look like this:

I have no idea who is behind it. I disagree with some of the things they're saying and some of the things they're saying are absolutely ridiculous and abysmal. Anything they do, I wish them luck, but I have nothing to do with them.

But they at least had the decency to wear their convictions on their sleeves and label it an opinion (though only after you've clicked it, otherwise it's among election news...).

While apparently quoting Dave Portnoy's opinion (whysoever that is "BREAKING NEWS" btw they both call their frontpage that...) on George Clooney's op-ed is "news" and not an opinion. Now fair enough that apparently IS Dave Portnoy's opinion and that is pretty obvious, but stretching what is essentially: "Lil late George". to an article of a reading time of 5-6 minutes quoting him calling Biden a vegetable a total of 22 times (headline included) is not just "reporting" on a nobody, but sounds more like an excuse for calling Biden "a vegetable" without raising the blood pressure of the legal department...

So is this really a case for where you need science to tell you that these news outlets have a bias or even a strong bias? Well no. That is quite obvious. Where it might be much more interesting is with Reuters, AP or other supposedly neutral sources, where they lean towards, which ideas or ideologies they explicitly or implicitly push, what they are well informed and inform well about and so on.

Also even more interesting than the bias ratings is the factfulness ratings. So how reliable is what they report. Like ideally even if I disagree with a news source they might still be worth a read if they do their jobs well and provide their biased perspective with well articulated and well researched articles, that maybe even illuminate blindspots.

Now how do they perform in that regard? https://www.politifact.com/article/2014/sep/16/fact-checking-fox-msnbc-and-cnn-punditfacts-networ/

For instance, currently 45 percent of the claims we’ve checked from NBC and MSNBC pundits and on-air personalities have been rated Mostly False, False or Pants on Fire.

At Fox and Fox News Channel, that same number is now 58 percent. At CNN, it’s 22 percent.

Now these numbers are from 2014 and they are from a fact checking website, so likely the claims that they checked were dubious to begin with. So one might take that with a grain of salt, nonetheless that's too high.

Like it's inevitable that news have a bias and it's inevitable that some news end up being wrong (if you deliver news fresh there is an intrinsic risk of being wrong or insufficient), that being said how often do they do that (fresh news) and how often do they just rehash existing news stories with a network specific spin? Still if your failure rate is slightly below or even above 50% that's just bad.

Again bias and fact checking are difficult endeavors where you not only have to assess a sources' bias but also account for your own and would need to find a benchmark to compare to which is likely also biased (even if it's just a little). But those are statements about nuances within what would otherwise appear to be neutral.

While if someone abuses the BREAKING NEWS WARNING that is reserved for idk natural disasters, nuclear war, catastrophes where you need to evacuate... you get the picture. For talking about shit some nobody has said. You should be able to asses for yourself that what you read isn't news but gossip...

If the headline is clickbait (provocative, emotional, ...) you should know it's gossip not news.

If it's one sided, it's obviously biased... If it's two sided or pretends to be it could still be biased, but at least they put in slightly more effort...

If the article doesn't match the headline, add that source to the block list (at least temporarily)...

Like with all due respect these are VERY VERY low bars for trustworthy news and if companies still fail them, don't honor them with your trust.

Seriously that's not supposed to be quality limbo ("how low can you go!"). If you're so far off from being a reliable source, you don't even qualify as comparison anymore.

Answer 10

I don't think that measuring bias could make any sense.

A lot of studies showed that it is much easier to fool a reader or a viewer if the beginning part of the news is credible and properly argued. It is a natural weakness of the human mind, once the reporter caught our trust we are more prone to believe what follows in the reporting. So measuring the bias in some news is tricky, because a news that is largely accurate, but at the end contains some biased details might be more effective in letting the public believe that those details are true.

So. The measure of how accurate a news is does not give a measure of its capability to fool the public.