What is it that "97% of climate scientists" actually believe about global warming?

Question

It is widely known that reporting on polls must be done with careful consideration. I have often heard that there is a scientific consensus on global warming that can no longer be denied, that 97% of climate scientists agree. This consensus is used politically to justify government spending to avert catastrophic effects from global warming.

97 out of 100 climate scientists agree humans are causing global warming

Climate Change: Public Skeptical, Scientists Sure

Is there a scientific consensus on global warming?

"Only 13 percent of Americans got the correct answer, which is that in fact about 97 percent of American scientists say that climate change is happening, and about a third of Americans just simply say they don't know," he said.

What statement do 97% of the climate scientists agree with?

How many people were interviewed for the survey?

How were those people chosen?

Were these results based on all the people interviewed? (or a sub-sample)

What is the sampling error for the poll results?

(Bonus points for answering all 20 questions a journalist should ask, but not necessary)

Answer 1

Here's a link to some documentation on the survey.

It was an invitation-only survey, and 10,257 Earth scientists were invited (from a database listing all geosciences faculty and working scientists).

It was web-based, and 3,146 invitees participated. They broke the respondents down based on area of expertise and if they are actively publishing.

Regarding the question:

Do you think human activity is a significant contributing factor in changing mean global temperatures?

82% of all participants "Yes"

• 77% of non-climatologists/non-publishers
• 90% of all climatologists
• 90% of all earth science publishers
• 90% of all active publishers on climate change
• 97% (75 out of 77) of climatologists who have published more than 50% of their recent peer-reviewed papers on climate change.

Answer 2

At SkepticalScience.com, we see this infographic:

With regard to the National Council on Public Polls guidelines outlining the 20 questions that journalists ought to ask when reporting on polls, I will address all 20 questions here with regard to:

Examining the Scientific Consensus on Climate Change, 2009, Peter T. Doran and Maggie Kendall Zimmerman

1. Who did the poll?

Maggie R. Kendall Zimmerman, Earth and Environmental Sciences, University of Illinois at Chicago, performed the poll. Her results are called "THE CONSENSUS ON THE CONSENSUS: AN OPINION SURVEY OF EARTH SCIENTISTS ON GLOBAL CLIMATE CHANGE", and in 2008 it was Kendall Zimmerman's Ph. D dissertation. Though the 2008 version is not available online, we can read the 2011 revision here.

The December 31, 2011 revision is published by The Heartland Institute,

2. Who paid for the poll and why was it done?

According to the poll's introduction, it was done "to collect and assess information about the opinions and attitudes of professionals within the field of geosciences (earth sciences) regarding global climate change, and the climate 'consensus' debate, as well as to understand the rationale the participants use when forming their opinions by directly surveying a large number of earth scientists."

3. How many people were interviewed for the survey?

10,257 people were invited to complete the online poll, of whom 3146 responded. Of these, a sub-set of the most specialized and knowledgable respondents was defined who listed climate science as their area of expertise and who also have published 50% or more of their recent peer-reviewed papers on the subject of climate change. This sub-set is 77 in number.

4. How were those people chosen?

The field of 10,257 people was chosen by a 2007 work collecting the identities of all geosciences faculty at reporting academic institutions, researchers at state geologic surveys associated with universities, and research at US federal institutions such as USGS, NOAA, NASA, etc.

5. What area (nation, state, or region) or what group (teachers,lawyers, Democratic voters, etc.) were these people chosen from?

90% of respondents were from the USA; 6% from Canada, and less than one percent from various other countries.

6. Are the results based on the answers of all the people interviewed?

Of all 3146 respondents, 82% answered "yes" to Question 2: "Do you think human activity is a significant contributing factor in changing mean global temperatures?"

Of the 77 selected individuals, 75 answered "yes" to Question 2.

7. Who should have been interviewed and was not? Or do response rates matter?

This question is very broad and the subject to much debate. In addition to any other debates about this question, I posit that the Bradley effect may work to cause people whose opinion does not coincide with the consensus to avoid such polls.

8. When was the poll done?

The poll was done from April 3 through April 17, 2008.

9. How were the interviews conducted?

Invitees received an email containing a unique URL. If they clicked the URL, they could answer the questions. Once the URL had been accessed once, it was locked so that it could not be accessed again.

10. What about polls on the Internet or World Wide Web?

This study was conducted without falling for various problems commonly associated with web polling, such as one person / many entries and positive self-selection of respondents.

11. What is the sampling error for the poll results?

The pollsters addressed error in their report by an account of how specialized and knowledgable the respondents were. They write:

Those who claimed to have over 50% of their peer-reviewed publications in the area of climate change over the last five years (a response of 2 on Q5) are, for the purposes of this study, considered an expert population in the field of climate science. They are referred to as “active climate researchers” (ACRs) in this paper. It became apparent early on in the survey process that, because emphasis would be placed on the answers given by this particular group of respondents, it would be necessary to fact check those claims.

When a respondent was found to have claimed over 50% of their publications in the area of climate change, their email address was recorded, along with their survey completion ID and their response to Q5. The email address was matched to a name of the respondent, and those two pieces of information were used to run an Internet publication search. The participant’s personal web page was accessed, along with a search of their publications on ISI WOS. If it was possible to verify that the participant did indeed have a proportionally significant number of climate-related publications, the respondent was checked off a list. If it was apparent that the participant did not have a record of publishing on climate change in peer-reviewed publications, their ID was noted. This judgment was reserved for extreme cases which could be classified as blatantly fraudulent answers. It is important to note that while verifying respondent’s publication claims, their survey responses were not associated with their names and email addresses so as to remove any possible judgment bias (i.e. all those indicating they had published more than 50% of the peer-reviewed publications in the last five years on the subject of climate change were treated equally in this judgment regardless of how they answered in the rest of the survey).

Altogether, 267 participants responded that over 50% of their peer-reviewed publications in the last five years were on the subject of climate change. Of those 268, 243 were found to have made an accurate claim. In some cases, these participants had no publication history at all since their Ph.D. theses, in some cases they had some published papers, but nothing that would fall into the category of climate change, and in other cases, participants had perhaps one or two publications in the area of climate change, but had not published on the issue anytime in the past decade. It is impossible to know the motivation behind these inaccurate responses, but likely reasons include inadvertent answer selection that did not reflect their intended answers, misinterpretation of the question, or an attempt to mislead the researchers.

The online survey administration and data tracking site that was used for this survey, QuestionPro, repeatedly calculated standard error at 0.01 for each question data set. The error calculated using the specific example of false publication history claims is 0.76.

THE CONSENSUS ON THE CONSENSUS: AN OPINION SURVEY OF EARTH SCIENTISTS ON GLOBAL CLIMATE CHANGE, page 16.

12. Who’s on first?

This journalistic-integrity question is directed at writers who misinterpret a narrow margin with high error as an indication that the leader is really ahead; but if the errors contain the margin very well, it is not acceptable to report that "candidate A is ahead".

In the present case, there is no indication that the researchers attempted to extrapolate the sample to the population, instead evaluating the results on the basis of whittled-down respondent populations. Again, instead of interpreting survey results as a sample of the true population, the survey was interpreted as containing the binary opinion (on Question 2) of a large group which that would be selected for special respondents.

13. What other kinds of factors can skew poll results?

This survey had been built on the learnings from several previous surveys and is not likely to have been skewed by wording or method. Biases like the Bradley Effect that affect human polling behavior are notoriously difficult to remove from poll responses.

14. What questions were asked?

We have been primarily talking about Question 2, "Do you think human activity is a significant contributing factor in changing mean global temperatures?"

15. In what order were the questions asked?

The first question was "When compared with pre-1800’s levels, do you think that mean global temperatures have generally risen, fallen, or remained relatively constant?"

16. What about "push polls?"

The important question #2 does not seem to be a push-poll, following the neutrally-worded question #1.

17. What other polls have been done on this topic? Do they say the same thing? If they are different, why are they different?

The green pie-chart infographic refers to several other studies since the time of the Kendall-Zimmerman study. The latest study was Carlton 2015, in which 630 of 698 scientists in different fields answered "yes" to question 4 of the anonymous survey:

Q4 Do you think human activity is a significant contributing factor in changing mean global temperatures?

This represents 90% of the survey respondents, but it represents 97% of the survey respondents who answered Question 3, "When compared with pre-1800's levels, do you think that mean global temperatures have generally risen, fallen, or remained relatively constant?" with "Risen".

18. What about exit polls?

This is not relevant for scientific-consensus polling.

19. What else needs to be included in the report of the poll?

It may be relevant that in April 2008, George W. Bush was in his final year of his second term and Al Gore's "An Inconvenient Truth" had been out for two years.

20. So I've asked all the questions. The answers sound good. Should we report the results?

I think that the 97% figure should only be reported with all of these qualifiers:

• Of 77 scientists whose single specialty is climate science,

• who have published over 50% of their peer-reviewed papers for the last five years on climate issues,

• 75 (97%) answered "yes" to Question 2 on an anonymous online survey: "Do you think human activity is a significant contributing factor in changing mean global temperatures?"

Answer 3

What statement do 97% of the climate scientists agree with?

From the IPCC Fifth Assessment Report (2014 / current), p. 28:

Observed temperature trends over the period 1951–2010, which are characterized by warming over most of the globe with the most intense warming over the [northern hemisphere] continents, are, at most observed locations, consistent with the temperature trends in CMIP5 simulations including anthropogenic and natural forcings and inconsistent with the temperature trends in CMIP5 simulations including natural forcings only.

On to Question 19 from the "20 Questions a Journalist Should Ask" webpage:

What else needs to be included in the report of the poll?

A notable contributor to the IPCC has made clear that the summaries, conclusions, and recommendations of IPCC reports do not necessarily reflect the opinions of all the authors, but only of the lead authors. Read the statement by John Christy, dated 17 June 2010, here.

From the IPCC Fifth Assessment Report (2014 / current), p. 30:

Box TS.3 | Climate Models and the Hiatus in Global Mean Surface Warming of the Past 15 Years

The observed [global mean surface temperature, or GMST,] has shown a much smaller increasing linear trend over the past 15 years than over the past 30 to 60 years (Box TS.3, Figure 1a, c). Depending on the observational data set, the GMST trend over 1998–2012 is estimated to be around one third to one half of the trend over 1951–2012. For example, in [the observational data set called] HadCRUT4 the trend is 0.04°C per decade over 1998–2012, compared to 0.11°C per decade over 1951–2012. The reduction in observed GMST trend is most marked in [northern hemisphere] winter. Even with this ‘hiatus’ in GMST trend, the decade of the 2000s has been the warmest in the instrumental record of GMST. Nevertheless, the occurrence of the hiatus in GMST trend during the past 15 years raises the two related questions of what has caused it and whether climate models are able to reproduce it. {2.4.3, 9.4.1; Box 9.2; Table 2.7}

Fifteen-year-long hiatus periods are common in both the observed and CMIP5 historical GMST time series. However, an analysis of the full suite of CMIP5 historical simulations (augmented for the period 2006–2012 by RCP4.5 simulations) reveals that 111 out of 114 realizations show a GMST trend over 1998–2012 that is higher than the [measurements*]. This difference between simulated and observed trends could be caused by some combination of (a) internal climate variability, (b) missing or incorrect [radiative forcing], and (c) model response error. These potential sources of the difference, which are not mutually exclusive, are assessed below, as is the cause of the observed GMST trend hiatus. {2.4.3, 9.3.2, 9.4.1; Box 9.2}

* Instead of "measurements", the report reads "entire HadCRUT4 trend ensemble (Box TS.3, Figure 1a; CMIP5 ensemble mean trend is 0.21°C per decade)"