36

I'm wondering if there is any study or other serious attempt which compares Fox News to MSNBC to see which one was more 'biased' towards their respective sides of the political spectrum.

I am not looking for opinion here, I'm looking for serious analysis that presents a criteria that can be (at least reasonably) objectively measured, has measured the two respective media outlets, and have presented results as to which outlet best met or failed those criteria.

I'm open to any criteria that can be used to measure bias, from number of factual inaccuracies in reporting, to how often they call the other side immoral and evil, to any other criteria that could be imagined as being a measure of, or result of, bias for their preferred political view and/or a sign of lack of trustworthiness in their reporting due to such a bias. All I really care is that it can be objectively measured and concrete results presented.

  • 22
    a lot of those measures are going to be subjective and hard if not impossible to quantify. However, there are viewer surveys and polling available that asks about a respondent's sources. Those could potentially be used to collectively judge the informedness (or mis) of viewers. For example, i recall in 2004 a report that fox news viewers were more likely than pbs viewers to believe that Iraq had weapons of mass destruction (false). It was about a 12% difference, iirc. Those kind of stats are actually attainable, "who lies more" isn't. – dandavis Jan 28 at 19:21
  • 4
    To measure bias scientifically (e.g. statistically) a study would need to limit itself questions where a true underlying quantitative parameter being measured. Categorical measures, such as true/false questions aren't really measurable as to bias, as I recall the definition of the scientific terms. – Burt_Harris Jan 28 at 19:33
  • 1
    Right, we can calculate which viewers are better-informed, but throwing in 'more biased' requires that we find a centerpoint, which is a much bigger headache. – Carduus Jan 28 at 19:50
  • 1
    I think a better question might be which organizations present more "fact" statements vs "opinion" statements. Opinions and Facts have two concrete definitions and determining which is which is not difficult. From here, one can apply their own subjective definitions of "Left" vs. "Right" thought and maybe assuage which side is more "opinionated" for their side of the aisle. This is still a subjective exercise but it is a way of looking for scientific evidence to make a specific point. – RLH Jan 29 at 14:45
  • 1
    This is like two astronauts floating in deep space trying to determine who is moving and who is still. Without a universally agreed upon reference point you simply can't tell. Any investigation of bias will necessarily be biased by those doing the investigating. – eps Jan 29 at 17:23
42

I'd say anyone who tells you they have "scientific" information on media bias isn't being objective. But it is entirely possible to present a "systematic" information on media bias, and at least one online source with that goal is https://mediabiasfactcheck.com. They have an explicit methodology, which says:

When determining bias, there isn’t any true scientific formula that is 100% objective. There are objective measures that can be calculated, but ultimately there will be some degree of subjective judgement to determine these. On each page we have put up a scale with a yellow dot that shows the degree of bias for each source. Each page also has a “detailed report” section that gives some details about the source and an explanation of their bias. When calculating bias we are not just looking at political bias, but also how factual the information is and if they provide links to credible, verifiable sources. Therefore, the yellow dot may indicate political bias or how factual a source is, or in many cases, both.

Mediabiasfactcheck has a high level summary graphic which may be helpful: For example, MSNBC looks like this:

msnbc factual

msnbc graphic

Fox News looks like this:

factual on fox news

fox news graphic

But mediabiasfactcheck.com goes into detail as well, and includes a systematic categorization of different kinds of journalistic bias.

If what you are looking for is unbiased news, recognize that's not the goal of either organization. Some news sources actually make reduction of bias and emotionally loaded reporting a policy goal. Take for example: Reuters News looks like this:

reuters

reuters graphic

Note well: its impossible to to express an opinion on news source bias without having some bias, and getting accused of being biased. Mediabiasfactcheck.com is no exception: some other organizations rate mediabiasfactcheck.com as biased. Oh well...

  • 17
    note that, while mediafactcheck rates both about the same re. Media Bias, when it comes to factual reporting they say: Fox: We also rate them Mixed factually and borderline Questionable, MSNBC: Although pundits and hosts have failed a few fact checks, straight news reporting is usually factual and sourced, therefore we rate them Mostly Factual for news reporting. – Italian Philosophers 4 Monica Jan 28 at 21:30
  • 58
    Being factual isn't enough. The facts reported and the facts not reported matter, too. For example, must US TV networks showed the House impeachment managers' making their case to remove President Trump live, but did not show Trump's lawyers making their case live. Is that bias? Or was that done because ratings were terrible for the live coverage last week? You'd have to be a mind-reader to know for sure. – Just Me Jan 28 at 21:47
  • 13
    For me, being "usually factual" isn't enough either. Too many reporters want to express their opinions, and without strong journalistic standards, it get's out of hand. Compare w/ Reuters Handbook of Journalism section on Freedom from Bias, which for lays out what it means to avoid bias, e.g.: Reuters journalists do not express their opinions in news stories, voiced video or scripts, or on blogs or chat rooms they may contribute to in the course of their work. – Burt_Harris Jan 28 at 23:30
  • 24
    Ugh, those graphics are nasty. It looks like the position of the spot on the arrows is marking the position in the truth-reporting table, and reading it like that (I did, at first, until I looked harder) suggests that Fox News has a rating of High*/*Very High and MSNBC has a rating solidly in Low. – Richard Ward Jan 29 at 15:15
  • 12
    @JustMe do you have evidence for your claim that most US TV networks didn't air Trump's lawyers? CSPAN, ABC, MSNBC and CNN certainly did – Shaz Jan 29 at 21:31
18

I'll take you through a Thought Experiment to show why this isn't really possible.

Imagine you were trying to judge whether networks A, B, and C were 'biased' on the coverage of, say, "Should the US offer aid to the rebels of [country name]?"

So, you might think, well, they should have facts from all sides. Present all the facts. Have people on to discuss any point of view on the issue. Give equal time to each side. Don't exclude any news that sheds either side in a positive light. Etc.

... now let's change those 'rebels' to a genocidal group that utilizes a child-army and practices slavery.

Suddenly, things change. If your network gives facts/opinions/equal-time to the genocidal/slaver group, it suddenly feels a lot less 'unbiased' and starts to feel a lot more like they're giving legitimacy to something horrible.

Do you now see the discrepancy? You're now in a position of having to figure out, for this particular issue: how legitimate is each side? Because you can't cover both sides 'equally' and have it be balanced if only one side has a legitimate cause. It's not like someone would expect News Co. XYZ to be balanced by giving equal time to a NASA Spokesman and a rep from the Flat Earth Society.

Let's take an actual issue: Brexit.

News Organization A covers Brexit by giving the facts from both sides, giving equal time to both proponents and dissenters. Organization B has more pro-Leave airtime/arguments; Organization C has more pro-Remain. So which ones are biased?

Well, how legitimate is each side? You could say Organization A is non-biased... except it means you've given each side roughly equal legitimacy. Is that fair? How do you tell? In any sort of non-subjective manner?

Now multiply this by the number of issues that people argue about. And... add in a dash of an uncomfortable truth: generally most popular political topics do indeed have at least some legitimacy on both sides (which a lot of people don't like admitting, since it makes it harder to viscerally hate the people that disagree with them.)

So... how would you design that experiment now? You can't - because "balance" depends on your subjective measure of each side of each issue.

  • Comments are not for extended discussion; this conversation has been moved to chat. – yannis Jan 31 at 23:44
17

Many studies have been conducted to test exactly this question. This study from 2012 asked participants 4 factual questions about international events, and 5 questions about domestic ones (questions like "It took a long time to get the final results of the Iowa caucuses for Republican candidates. In the end, who was declared the winner?")

Nobody did very well on the survey-- even the most well-informed group averaged less than 2 correct answers in each category. The highest average were people who got their news primarily from NPR. Fox News performed the worst in both domestic issues and international ones, even worse than the "no news" group. MSNBC was the second-lowest, coming in just under "no news" for international issues and just above "no news" for domestic issues.

Studies like this one usually produce similar results. Some researchers call this "the Fox News effect", though a similar case could be made for calling it "the MSNBC effect". So the best empirical evidence we have suggests that Fox is worse than MSNBC-- but not by much.

  • 21
    I'm not sure this really answers the question, as interesting as that study is. Being non-informative isn't necessarily the same as biased... – Fizz Jan 28 at 19:53
  • 30
    It's also not clear if watching Fox News leads to ignorance, or ignorance leads to watching Fox News, or some mixture there of. – CrackpotCrocodile Jan 29 at 6:10
  • 3
    @CrackpotCrocodile It can be a vicious cycle, at both ends of the political spectrum. People gravitate towards outlets that reinforce their preconceptions. Flat-Earthers can find plenty of websites that tell them that they're correct. – Barmar Jan 29 at 16:07
  • 3
    @Kevin On the other hand the study does not appear to specify how many got their news exclusively from fox or MSNBC. Considering the study only covered a little over a thousand people if most people got their news from multiple sources the sample size for people who got their news exclusively from one source may be a bit small to draw a conclusion. Likewise the total number of questions asked is rather small. Both may lower how much confidence we place on the exact numbers. – dsollen Jan 29 at 20:57
  • 5
    One issue with surveys of this nature, though, is that the questions being asked can themselves be biased towards one group or another, even if they remain entirely factual. One frequent tactic of media on both sides is constantly harping on a few talking points, including whatever facts may support it (but typically omitting most or even any that don't.) As such, it would be pretty easy to design a study that says just about any group you want is the 'most informed' by choosing the questions according to which group is more likely to use those facts to support one of their positions. – reirab Jan 29 at 21:02
6

Along the lines of the excellent answer from Burt_Harris, you might be interested in Ad Fontes Media

They rank a wide array of news (... and, not-really-news) sites.

The reason I reference the answer from Burt_Harris is because I agree that it's difficult to have a strictly scientific approach to this, but Ad Fontes Media has a systematic approach, meant to produce reliable results. For detailed information, see their methodology page.

  • 1
    Interesting, but a bit confusing. For example, "Fox News" online is listed as less biased than MSNBC, but "Fox News" on cable is more. That makes OP's question, at least as it is currently formulated, impossible to answer with this source. – Michael W. Jan 30 at 18:40
  • 3
    @MichaelW. : that's one way to look at it; another way is to realize that this source shows that "Fox News" is not a monolithic, monochromatic thing; the same is true of CNN (per that chart); in that sense, I feel that this source is very pertinent to the question, because it helps illuminate subtleties that are not made obvious by the question as-asked; IMHO, providing an answer that examines / challenges the assumptions of a given question is a great way for us all to learn from each other – landru27 Jan 30 at 22:57
  • 1
    I think you misunderstand me. I think this source, if taken as true, illuminates a flaw in the original question. "Is MSNBC or Fox News more biased" can only be responded with "which MSNBC and which Fox News?" – Michael W. Jan 31 at 22:41
  • Thanks @landru27, and welcome! Nice contribution. – Burt_Harris Feb 1 at 3:37
2

Please see this publication from UCLA, which estimates ADA scores for major media outlets. According to the introduction:

...we count the times that a media outlet cites various think tanks. We compare this with the times that members of Congress cite the same think tanks in their speeches on the floor of the House and Senate. By comparing the citation patterns we can construct an ADA score ... These findings refer strictly to the news stories of the outlets [and not] editorials, book reviews, and letters to the editor from our sample

The publication also contains a section Previous Studies of Media Bias

It's worth considering possible shortcomings of the article's described methods. Firstly, you have to be willing to accept that think-tank citation ratios that coincide with those of congressional members are an acceptable proxy for media outlet bias. This requires that we accept other assumptions prima facie, like a consistent manner of think-tank citation behavior across the political spectrum, etc.

Still, this is, to date, the most serious analysis I've seen. As regarding objectivity, the described methods seem sufficiently detailed so as to be repeated, and hence the results independently measurable and verifiable. I've intentionally omitted their conclusions - if you want to know, read the article ;)

EDIT

Here are two other studies I found:

  1. WHAT DRIVES MEDIA SLANT? EVIDENCE FROM U.S. DAILY NEWSPAPERS
  2. Fair and Balanced? Quantifying Media Bias through Crowdsourced Content Analysis

The first tries to compare "unique phrase usage" analysis of media outlets to members of Congress. I'm skeptical of the second, which tries to recognize bias with machine learning techniques, trained with data procured by a set of volunteers. They might have sufficed with just generalizing the volunteers' results since the AI can only ever (likely) perform as well and no better than it's training set provided by flawed humans.

And here is a data source of news chyron archives used in some more serious analyses.

Finally, an article probing the meta-concerns of determining media bias.

2

AllSides.com has a nice list of bias ratings along with a community rating of how accurate that label is.

From their Media Bias Ratings page:

Our Media Bias Ratings are determined using multiple methods and represent the average judgment of Americans. They are based on blind surveys of people across the political spectrum, multi-partisan analysis, editorial reviews, third party data, and tens of thousands of user feedback ratings. Our scientifically-generated ratings are fluid and subject to change as new information is gathered and biases change over time.

MSNBC rating (2020/02/03): MSNBC AllSides Bias Rating

Fox News rating (2020/02/03): Fox News AllSides Bias Rating

1

I am not looking for opinion here, I'm looking for serious analysis that presents a criteria that can be (at least reasonably) objectively measured

Here's an objectively measurable criterion: what percentage of people employed by the agency cut their toe nails once a week?

Asking "has anyone come up with an objective measure of X?" makes no sense. If X is itself not an objective measure, there is no objective way to assess if your objective measure is a meaningful measure of X.

The question you should really be asking in situations like this is "What would be the purpose of such a measure?" What makes lower bias preferable? This will tell you what you should expect from an objective measure of bias. The same applies, by the way, to "factual reporting". How exactly do you wish to weight each statement's truth value? What counts as a statement by a news organisation/how discrete do you go?

I don't think there's a good definition here, though. Gotcha side-by-side comparisons of analogous articles by the same outlet are the best you're ever likely to see on this.

1

Ad Fontes Media produces the following chart, which seems to be well researched:

enter image description here

MSNBC seems to be roughly on par with the FOX News channel as a whole in terms of partisan slant, though MSNBC is positioned as being a more reliable news source. Note that FOX news programming (separated from the punditry of the rest of the channel) is viewed as equally reliable to MSNBC, and slightly less partisan.

  • I ended up checking out that source as part of answering an SE.Skeptics question. The overall goal of the site looks very interesting, and the analysis isn't without some merit, though the data seemed far too sparse to be representative. Assuming that the project has been conducted in-good-faith, I'd consider it to be interesting in an academic sense as initial groundwork in the field, but not something that people should regard as providing reliable scientific analysis in a general context. – Nat Jan 31 at 17:35
  • 1
    I didn't fully investigate their methodology and such, but their results struck me as suspiciously clean, especially for the amount of data that they appear to have collected. So while I wouldn't necessarily disagree with their results, I'm skeptical of the methodology used to arrive at those results. – Nat Jan 31 at 17:39
  • Ridiculous. The Weather Channel has clear bias. It regularly shows that winds largely flow to the right across the globe on our standard north-up map orientation. A politically balanced view should obviously show an equal amount of left-flowing winds. 😊 – Ben I. Jan 31 at 17:45
  • That's just because people on the right are bigger blowhards... 😜 – Ted Wrigley Jan 31 at 17:51
-2

I don't think that measuring bias could make any sense.

A lot of studies showed that it is much easier to fool a reader or a viewer if the beginning part of the news is credible and properly argued. It is a natural weakness of the human mind, once the reporter caught our trust we are more prone to believe what follows in the reporting. So measuring the bias in some news is tricky, because a news that is largely accurate, but at the end contains some biased details might be more effective in letting the public believe that those details are true.

So. The measure of how accurate a news is does not give a measure of its capability to fool the public.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .