Comparative Environmental Policy/ Executive Orders

Question

For a paper, I am currently trying to compare Executive Orders by Trump within the first year of his presidency and Bush's first year of his first presidency concerning environmental issues.

My problem is, I don't know, how to define / identify environmental issues and categorise them as such objectively and scientifically within an executive order, without it specifically including the terms "environment", or "energy". Could anyone help me with an appropriate method of identifying and defining Executive Orders concerning the environment?

Would a content analysis of the documents be the best approach for categorisation?

Answer 1

If it's just Bush and Trump and only their first year that you're interested in, the best option might be to classify them manually. According to federalregister.gov, Trump issued 55 and Bush issued 54 executive orders in their first year.

The title of each order gives a general idea of what it's about, so you could categorise them into three categories in less than half a minute per order (meaning this initial classification process takes less than an hour):

• Definitely relevant (i.e. related to energy or the environment)
• Definitely not relevant (i.e. related to something else)
• Not sure

Ideally, most fall within the first two categories and you can take a little bit more time and consider the first few lines of the actual order to classify those in the third category. In your paper you explain classifying them manually this way and you add an appendix stating which specific orders you did consider.

Answer 2

A common way to do manual qualitative analysis for publication is to a have at least two people categorize the material... after you establish some guidelines for what they should look for.

Report in the paper both the guidelines you set and the number of consensus and non-consensus categorizations. Another thing that is sometimes done (when there are only two reviewers/analysts) is to have a consensus discussion among them on the disagreed material; this may help categorize more material. When such a consensus discussion is part of the process, it's also reported in the paper, e.g. 20 docs were agreed independently, there was (initial) disagreement on 6, which was narrowed down to 3 after a consensus discussion.

Also note that manual categorization doesn't exclude the use of some computer-assist. Such an approach is called usually hybrid, of course (even in political science circles), but in legal circles the preferred term seems to be "technology-assisted review".

Finally consider more than a binary yes/no categorization, e.g. [have reviewers] give a score how relevant the order is for the environment. The more guidelines you'll have for this, the less subjective the score will be.

Perhaps have a look at Roitblat et al. for more concrete ideas; they deal with categorizing documents (by content relevant to) e-discovery.

Regarding executive orders specifically, I found one mention of a paper doing such a classfication

Fine and Warber (2012) construct their executive order dependent variable by content analyzing every available executive order from 1953 to 2008 and classifying it as either routine, symbolic or major. They define major executive orders as those, “either departing from the status quo of a specific policy that has already been implemented, or interpreting and implementing legislation that diverts from the original intent of Congress” (Warber 2006, 143)

• Fine, Jeffrey A., and Adam L. Warber. 2012. “Circumventing Adversity: Executive Orders and Divided Government.” Presidential Studies Quarterly 42 (2): 256-274.

Alas the actual classification used in that paper seems to have been done in earlier work, which is a book:

• Warber, Adam L. 2006. Executive Orders and the Modern Presidency: Legislating from the Oval Office. Boulder, CO: Lynne Rienner Publishers

So its methods may not have been as rigorous as in some peer-reviewed publications, I don't really know. Anyway, it looks there's not a lot of prior art in classifying executive orders, so whatever method you choose is likely to be okay as long as it's reasonably justifiable in terms of the general approach to document classification.