Has anyone attempted to scientifically determine whether MSNBC or Fox News is more biased?

Question

I'm wondering if there is any study or other serious attempt which compares Fox News to MSNBC to see which one was more 'biased' towards their respective sides of the political spectrum.

I am not looking for opinion here, I'm looking for serious analysis that presents a criteria that can be (at least reasonably) objectively measured, has measured the two respective media outlets, and have presented results as to which outlet best met or failed those criteria.

I'm open to any criteria that can be used to measure bias, from number of factual inaccuracies in reporting, to how often they call the other side immoral and evil, to any other criteria that could be imagined as being a measure of, or result of, bias for their preferred political view and/or a sign of lack of trustworthiness in their reporting due to such a bias. All I really care is that it can be objectively measured and concrete results presented.

Answer 1

I'd say anyone who tells you they have "scientific" information on media bias isn't being objective. But it is entirely possible to present a "systematic" information on media bias, and at least one online source with that goal is https://mediabiasfactcheck.com. They have an explicit methodology, which says:

When determining bias, there isn’t any true scientific formula that is 100% objective. There are objective measures that can be calculated, but ultimately there will be some degree of subjective judgement to determine these. On each page we have put up a scale with a yellow dot that shows the degree of bias for each source. Each page also has a “detailed report” section that gives some details about the source and an explanation of their bias. When calculating bias we are not just looking at political bias, but also how factual the information is and if they provide links to credible, verifiable sources. Therefore, the yellow dot may indicate political bias or how factual a source is, or in many cases, both.

Mediabiasfactcheck has a high level summary graphic which may be helpful: For example, MSNBC looks like this:

Fox News looks like this:

But mediabiasfactcheck.com goes into detail as well, and includes a systematic categorization of different kinds of journalistic bias.

If what you are looking for is unbiased news, recognize that's not the goal of either organization. Some news sources actually make reduction of bias and emotionally loaded reporting a policy goal. Take for example: Reuters News looks like this:

Note well: its impossible to to express an opinion on news source bias without having some bias, and getting accused of being biased. Mediabiasfactcheck.com is no exception: some other organizations rate mediabiasfactcheck.com as biased. Oh well...

Answer 2

I'll take you through a Thought Experiment to show why this isn't really possible.

Imagine you were trying to judge whether networks A, B, and C were 'biased' on the coverage of, say, "Should the US offer aid to the rebels of [country name]?"

So, you might think, well, they should have facts from all sides. Present all the facts. Have people on to discuss any point of view on the issue. Give equal time to each side. Don't exclude any news that sheds either side in a positive light. Etc.

... now let's change those 'rebels' to a genocidal group that utilizes a child-army and practices slavery.

Suddenly, things change. If your network gives facts/opinions/equal-time to the genocidal/slaver group, it suddenly feels a lot less 'unbiased' and starts to feel a lot more like they're giving legitimacy to something horrible.

Do you now see the discrepancy? You're now in a position of having to figure out, for this particular issue: how legitimate is each side? Because you can't cover both sides 'equally' and have it be balanced if only one side has a legitimate cause. It's not like someone would expect News Co. XYZ to be balanced by giving equal time to a NASA Spokesman and a rep from the Flat Earth Society.

Let's take an actual issue: Brexit.

News Organization A covers Brexit by giving the facts from both sides, giving equal time to both proponents and dissenters. Organization B has more pro-Leave airtime/arguments; Organization C has more pro-Remain. So which ones are biased?

Well, how legitimate is each side? You could say Organization A is non-biased... except it means you've given each side roughly equal legitimacy. Is that fair? How do you tell? In any sort of non-subjective manner?

Now multiply this by the number of issues that people argue about. And... add in a dash of an uncomfortable truth: generally most popular political topics do indeed have at least some legitimacy on both sides (which a lot of people don't like admitting, since it makes it harder to viscerally hate the people that disagree with them.)

So... how would you design that experiment now? You can't - because "balance" depends on your subjective measure of each side of each issue.

Answer 3

Many studies have been conducted to test exactly this question. This study from 2012 asked participants 4 factual questions about international events, and 5 questions about domestic ones (questions like "It took a long time to get the final results of the Iowa caucuses for Republican candidates. In the end, who was declared the winner?")

Nobody did very well on the survey-- even the most well-informed group averaged less than 2 correct answers in each category. The highest average were people who got their news primarily from NPR. Fox News performed the worst in both domestic issues and international ones, even worse than the "no news" group. MSNBC was the second-lowest, coming in just under "no news" for international issues and just above "no news" for domestic issues.

Studies like this one usually produce similar results. Some researchers call this "the Fox News effect", though a similar case could be made for calling it "the MSNBC effect". So the best empirical evidence we have suggests that Fox is worse than MSNBC-- but not by much.

Answer 4

Along the lines of the excellent answer from Burt_Harris, you might be interested in Ad Fontes Media

They rank a wide array of news (... and, not-really-news) sites.

The reason I reference the answer from Burt_Harris is because I agree that it's difficult to have a strictly scientific approach to this, but Ad Fontes Media has a systematic approach, meant to produce reliable results. For detailed information, see their methodology page.

Answer 5

AllSides.com has a nice list of bias ratings along with a community rating of how accurate that label is.

From their Media Bias Ratings page:

Our Media Bias Ratings are determined using multiple methods and represent the average judgment of Americans. They are based on blind surveys of people across the political spectrum, multi-partisan analysis, editorial reviews, third party data, and tens of thousands of user feedback ratings. Our scientifically-generated ratings are fluid and subject to change as new information is gathered and biases change over time.

MSNBC rating (2020/02/03):

Fox News rating (2020/02/03):

Answer 6

Ad Fontes Media produces the following chart, which seems to be well researched:

MSNBC seems to be roughly on par with the FOX News channel as a whole in terms of partisan slant, though MSNBC is positioned as being a more reliable news source. Note that FOX news programming (separated from the punditry of the rest of the channel) is viewed as equally reliable to MSNBC, and slightly less partisan.

Answer 7

EDIT

some other answers suggest political bias is not objective, probably because they are confounding "preference" with "observation of preference (bias)". Ambiguity in natural language obscures the issue, but objectivity is possible. Preference is subjective, but measurement of preference (i.e. observed bias) is objective. Consider supremacy of the color red (a subjective qualial preference), in contrast to current preference (i.e. bias) for red amongst infants (difficult to measure, but still objective). Likewise, favorability of party X is subjective, but observed favor for party X, though hard to measure, is still objective. Also, objectivity is not confined to non-stochastic phenomena. In cases where objective phenomena are difficult to measure, clever scientists will search for plausible proxy metrics that are easier to measure, as is the case with the following:

ORIGINAL POST

Please see this publication from UCLA, which estimates ADA scores for major media outlets. According to the introduction:

...we count the times that a media outlet cites various think tanks. We compare this with the times that members of Congress cite the same think tanks in their speeches on the floor of the House and Senate. By comparing the citation patterns we can construct an ADA score ... These findings refer strictly to the news stories of the outlets [and not] editorials, book reviews, and letters to the editor from our sample

The publication also contains a section Previous Studies of Media Bias

It's worth considering possible shortcomings of the article's described methods. Firstly, you have to be willing to accept that think-tank citation ratios that coincide with those of congressional members are an acceptable proxy for media outlet bias. This requires that we accept other assumptions prima facie, like a consistent manner of think-tank citation behavior across the political spectrum, etc.

Still, this is, to date, the most serious analysis I've seen. As regarding objectivity, the described methods seem sufficiently detailed so as to be repeated, and hence the results independently measurable and verifiable. I've intentionally omitted their conclusions - if you want to know, read the article ;)

EDIT

Here are two other studies I found:

1. WHAT DRIVES MEDIA SLANT? EVIDENCE FROM U.S. DAILY NEWSPAPERS
2. Fair and Balanced? Quantifying Media Bias through Crowdsourced Content Analysis

The first tries to compare "unique phrase usage" analysis of media outlets to members of Congress. I'm skeptical of the second, which tries to recognize bias with machine learning techniques, trained with data procured by a set of volunteers. They might have sufficed with just generalizing the volunteers' results since the AI can only ever (likely) perform as well and no better than it's training set provided by flawed humans.

And here is a data source of news chyron archives used in some more serious analyses.

Finally, an article probing the meta-concerns of determining media bias.

Answer 8

I am not looking for opinion here, I'm looking for serious analysis that presents a criteria that can be (at least reasonably) objectively measured

Here's an objectively measurable criterion: what percentage of people employed by the agency cut their toe nails once a week?

Asking "has anyone come up with an objective measure of X?" makes no sense. If X is itself not an objective measure, there is no objective way to assess if your objective measure is a meaningful measure of X.

The question you should really be asking in situations like this is "What would be the purpose of such a measure?" What makes lower bias preferable? This will tell you what you should expect from an objective measure of bias. The same applies, by the way, to "factual reporting". How exactly do you wish to weight each statement's truth value? What counts as a statement by a news organisation/how discrete do you go?

I don't think there's a good definition here, though. Gotcha side-by-side comparisons of analogous articles by the same outlet are the best you're ever likely to see on this.

Answer 9

I don't think that measuring bias could make any sense.

A lot of studies showed that it is much easier to fool a reader or a viewer if the beginning part of the news is credible and properly argued. It is a natural weakness of the human mind, once the reporter caught our trust we are more prone to believe what follows in the reporting. So measuring the bias in some news is tricky, because a news that is largely accurate, but at the end contains some biased details might be more effective in letting the public believe that those details are true.

So. The measure of how accurate a news is does not give a measure of its capability to fool the public.