At SkepticalScience.com, we see this infographic:
With regard to the National Council on Public Polls guidelines outlining the 20 questions that journalists ought to ask when reporting on polls, I will address all 20 questions here with regard to:
Examining the Scientific Consensus on Climate Change,
2009, Peter T. Doran and Maggie Kendall Zimmerman
- Who did the poll?
Maggie R. Kendall Zimmerman, Earth and Environmental Sciences, University of Illinois at Chicago, performed the poll. Her results are called "THE CONSENSUS ON THE CONSENSUS: AN OPINION SURVEY OF EARTH SCIENTISTS ON GLOBAL CLIMATE CHANGE", and in 2008 it was Kendall Zimmerman's Ph. D dissertation. Though the 2008 version is not available online, we can read the 2011 revision here.
The December 31, 2011 revision is published by The Heartland Institute,
- Who paid for the poll and why was it done?
According to the poll's introduction, it was done "to collect and assess information about the opinions and attitudes of professionals within the field of geosciences (earth sciences) regarding global climate change, and the climate 'consensus' debate, as well as to understand the rationale the participants use when forming their opinions by directly surveying a large number of earth scientists."
- How many people were interviewed for the survey?
10,257 people were invited to complete the online poll, of whom 3146 responded. Of these, a sub-set of the most specialized and knowledgable respondents was defined who listed climate science as their area of expertise and who also have published 50% or more of their recent peer-reviewed papers on the subject of climate change. This sub-set is 77 in number.
- How were those people chosen?
The field of 10,257 people was chosen by a 2007 work collecting the identities of all geosciences faculty at reporting academic institutions, researchers at state geologic surveys associated with universities, and research at US federal institutions such as USGS, NOAA, NASA, etc.
- What area (nation, state, or region) or what group (teachers,lawyers, Democratic voters, etc.) were these people chosen from?
90% of respondents were from the USA; 6% from Canada, and less than one percent from various other countries.
- Are the results based on the answers of all the people interviewed?
Of all 3146 respondents, 82% answered "yes" to Question 2: "Do you think human activity is a significant contributing factor in changing mean global temperatures?"
Of the 77 selected individuals, 75 answered "yes" to Question 2.
- Who should have been interviewed and was not? Or do response rates matter?
This question is very broad and the subject to much debate. In addition to any other debates about this question, I posit that the Bradley effect may work to cause people whose opinion does not coincide with the consensus to avoid such polls.
- When was the poll done?
The poll was done from April 3 through April 17, 2008.
- How were the interviews conducted?
Invitees received an email containing a unique URL. If they clicked the URL, they could answer the questions. Once the URL had been accessed once, it was locked so that it could not be accessed again.
- What about polls on the Internet or World Wide Web?
This study was conducted without falling for various problems commonly associated with web polling, such as one person / many entries and positive self-selection of respondents.
- What is the sampling error for the poll results?
The pollsters addressed error in their report by an account of how specialized and knowledgable the respondents were. They write:
Those who claimed to have over 50% of their peer-reviewed publications in the area of climate change over the last five years (a response of 2 on Q5) are, for the purposes of this study, considered an expert population in the field of climate science. They are referred to as “active climate researchers” (ACRs) in this paper. It became apparent early on in the survey process that, because emphasis would be placed on the answers given by this particular group of respondents, it would be necessary to fact check those claims.
When a respondent was found to have claimed over 50% of their publications in the area of climate change, their email address was recorded, along with their survey completion ID and their response to Q5. The email address was matched to a name of the respondent, and those two pieces of information were used to run an Internet publication search. The participant’s personal web page was accessed, along with a search of their publications on ISI WOS. If it was possible to verify that the participant did indeed have a proportionally significant number of climate-related publications, the respondent was checked off a list. If it was apparent that the participant did not have a record of publishing on climate change in peer-reviewed publications, their ID was noted. This judgment was reserved for extreme cases which could be classified as blatantly fraudulent answers. It is important to note that while verifying respondent’s publication claims, their survey responses were not associated with their names and email addresses so as to remove any possible judgment bias (i.e. all those indicating they had published more than 50% of the peer-reviewed publications in the last five years on the subject of climate change were treated equally in this judgment regardless of how they answered in the rest of the survey).
Altogether, 267 participants responded that over 50% of their peer-reviewed publications in the last five years were on the subject of climate change. Of those 268, 243 were found to have made an accurate claim. In some cases, these participants had no publication history at all since their Ph.D. theses, in some cases they had some published papers, but nothing that would fall into the category of climate change, and in other cases, participants had perhaps one or two publications in the area of climate change, but had not published on the issue anytime in the past decade. It is impossible to know the motivation behind these inaccurate responses, but likely reasons include inadvertent answer selection that did not reflect their intended answers, misinterpretation of the question, or an attempt to mislead the researchers.
The online survey administration and data tracking site that was used for this survey, QuestionPro, repeatedly calculated standard error at 0.01 for each question data set. The error calculated using the specific example of false publication history claims is 0.76.
THE CONSENSUS ON THE CONSENSUS: AN OPINION SURVEY OF EARTH SCIENTISTS ON GLOBAL CLIMATE CHANGE, page 16.
- Who’s on first?
This journalistic-integrity question is directed at writers who misinterpret a narrow margin with high error as an indication that the leader is really ahead; but if the errors contain the margin very well, it is not acceptable to report that "candidate A is ahead".
In the present case, there is no indication that the researchers attempted to extrapolate the sample to the population, instead evaluating the results on the basis of whittled-down respondent populations. Again, instead of interpreting survey results as a sample of the true population, the survey was interpreted as containing the binary opinion (on Question 2) of a large group which that would be selected for special respondents.
- What other kinds of factors can skew poll results?
This survey had been built on the learnings from several previous surveys and is not likely to have been skewed by wording or method. Biases like the Bradley Effect that affect human polling behavior are notoriously difficult to remove from poll responses.
- What questions were asked?
We have been primarily talking about Question 2, "Do you think human activity is a significant contributing factor in changing mean global temperatures?"
- In what order were the questions asked?
The first question was "When compared with pre-1800’s levels, do you think that mean global temperatures have generally risen, fallen, or remained relatively constant?"
- What about "push polls?"
The important question #2 does not seem to be a push-poll, following the neutrally-worded question #1.
- What other polls have been done on this topic? Do they say the same thing? If they are different, why are they different?
The green pie-chart infographic refers to several other studies since the time of the Kendall-Zimmerman study. The latest study was Carlton 2015, in which 630 of 698 scientists in different fields answered "yes" to question 4 of the anonymous survey:
Q4 Do you think human activity is a significant contributing factor in changing mean global temperatures?
This represents 90% of the survey respondents, but it represents 97% of the survey respondents who answered Question 3, "When compared with pre-1800's levels, do you think that mean global temperatures have generally risen, fallen, or remained relatively constant?" with "Risen".
- What about exit polls?
This is not relevant for scientific-consensus polling.
- What else needs to be included in the report of the poll?
It may be relevant that in April 2008, George W. Bush was in his final year of his second term and Al Gore's "An Inconvenient Truth" had been out for two years.
- So I've asked all the questions. The answers sound good. Should we report the results?
I think that the 97% figure should only be reported with all of these qualifiers:
Of 77 scientists whose single specialty is climate science,
who have published over 50% of their peer-reviewed papers for the last five years on climate issues,
75 (97%) answered "yes" to Question 2 on an anonymous online survey: "Do you think human activity is a significant contributing factor in changing mean global temperatures?"