For a paper, I am currently trying to compare Executive Orders by Trump within the first year of his presidency and Bush's first year of his first presidency concerning environmental issues.

My problem is, I don't know, how to define / identify environmental issues and categorise them as such objectively and scientifically within an executive order, without it specifically including the terms "environment", or "energy". Could anyone help me with an appropriate method of identifying and defining Executive Orders concerning the environment?

Would a content analysis of the documents be the best approach for categorisation?

  • Why do you need to do it "without [...] specifically including the terms "environment", or "energy""? How do you think content analysis works?
    – Fizz
    Apr 15 '19 at 16:10
  • 1
    I think a question about analysis methods might be off-topic here, possibly a better fit for Cross Validated. A question about what kinds of issues count as environmental issues might be on-topic, but could possibly be too broad
    – divibisan
    Apr 15 '19 at 16:17
  • I removed the emphasis on the content analysis part which wasn't added by the author. I think it's a good question as it requires more knowledge of the information that's available (which is political) than knowledge of how to do something like topic modelling (which might be even better suited for the datascience stack).
    – JJJ
    Apr 15 '19 at 17:07
  • Here's a guide to content analysis mailman.columbia.edu/research/population-health-methods/… And a much more in-depth review paper on it pilotscholars.up.edu/cgi/…
    – Fizz
    Apr 15 '19 at 17:34
  • @Fizz because many executive orders that concern environmental issues do not contain either terms, as they are on topics such as agriculture or trade deals, I thought content analysis would help me categorise them as such. Thank you very much for the links! Apr 16 '19 at 15:19

If it's just Bush and Trump and only their first year that you're interested in, the best option might be to classify them manually. According to federalregister.gov, Trump issued 55 and Bush issued 54 executive orders in their first year.

The title of each order gives a general idea of what it's about, so you could categorise them into three categories in less than half a minute per order (meaning this initial classification process takes less than an hour):

  • Definitely relevant (i.e. related to energy or the environment)
  • Definitely not relevant (i.e. related to something else)
  • Not sure

Ideally, most fall within the first two categories and you can take a little bit more time and consider the first few lines of the actual order to classify those in the third category. In your paper you explain classifying them manually this way and you add an appendix stating which specific orders you did consider.

  • Thank you for your response JJJ, what you have mentioned - categorising the Executive orders in what would be considered environmental - I have done before posting. Only after taking this step though, I was wondering if my personal categorisation would suffice for a scientific paper, or if other methods of coding needed to be included. Apr 16 '19 at 15:12
  • @DavidGatzegooHeckenberg unless the orders are already categorised you are going to have to use some way which isn't going to be perfect all the time. If it's really important to you, I imagine this is a common problem and you might want to look how other publicans dealt with it and if they did the same you can use that to justify your approach. And also ask around in your research group or whoever you're working with.
    – JJJ
    Apr 16 '19 at 15:18
  • yes, thank you very much. Trump's Whitehouse homepage, obviously due to improved website design over the years actually categorised most of his executive orders. Bush Jr. has categorised issues he had tried to tackle on his archived Whitehouse website, but they do not include executive orders. Thus I will just present the reasoning for categorisation then. Apr 16 '19 at 15:28

A common way to do manual qualitative analysis for publication is to a have at least two people categorize the material... after you establish some guidelines for what they should look for.

Report in the paper both the guidelines you set and the number of consensus and non-consensus categorizations. Another thing that is sometimes done (when there are only two reviewers/analysts) is to have a consensus discussion among them on the disagreed material; this may help categorize more material. When such a consensus discussion is part of the process, it's also reported in the paper, e.g. 20 docs were agreed independently, there was (initial) disagreement on 6, which was narrowed down to 3 after a consensus discussion.

Also note that manual categorization doesn't exclude the use of some computer-assist. Such an approach is called usually hybrid, of course (even in political science circles), but in legal circles the preferred term seems to be "technology-assisted review".

Finally consider more than a binary yes/no categorization, e.g. [have reviewers] give a score how relevant the order is for the environment. The more guidelines you'll have for this, the less subjective the score will be.

Perhaps have a look at Roitblat et al. for more concrete ideas; they deal with categorizing documents (by content relevant to) e-discovery.

Regarding executive orders specifically, I found one mention of a paper doing such a classfication

Fine and Warber (2012) construct their executive order dependent variable by content analyzing every available executive order from 1953 to 2008 and classifying it as either routine, symbolic or major. They define major executive orders as those, “either departing from the status quo of a specific policy that has already been implemented, or interpreting and implementing legislation that diverts from the original intent of Congress” (Warber 2006, 143)

Alas the actual classification used in that paper seems to have been done in earlier work, which is a book:

  • Warber, Adam L. 2006. Executive Orders and the Modern Presidency: Legislating from the Oval Office. Boulder, CO: Lynne Rienner Publishers

So its methods may not have been as rigorous as in some peer-reviewed publications, I don't really know. Anyway, it looks there's not a lot of prior art in classifying executive orders, so whatever method you choose is likely to be okay as long as it's reasonably justifiable in terms of the general approach to document classification.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .