A common way to do manual qualitative analysis for publication is to a have at least two people categorize the material... after you establish some guidelines for what they should look for.
Report in the paper both the guidelines you set and the number of consensus and non-consensus categorizations. Another thing that is sometimes done (when there are only two reviewers/analysts) is to have a consensus discussion among them on the disagreed material; this may help categorize more material. When such a consensus discussion is part of the process, it's also reported in the paper, e.g. 20 docs were agreed independently, there was (initial) disagreement on 6, which was narrowed down to 3 after a consensus discussion.
Also note that manual categorization doesn't exclude the use of some computer-assist. Such an approach is called usually hybrid, of course (even in political science circles), but in legal circles the preferred term seems to be "technology-assisted review".
Finally consider more than a binary yes/no categorization, e.g. [have reviewers] give a score how relevant the order is for the environment. The more guidelines you'll have for this, the less subjective the score will be.
Perhaps have a look at Roitblat et al. for more concrete ideas; they deal with categorizing documents (by content relevant to) e-discovery.
Regarding executive orders specifically, I found one mention of a paper doing such a classfication
Fine and Warber (2012) construct their executive order dependent variable by content
analyzing every available executive order from 1953 to 2008 and classifying it as either routine,
symbolic or major. They define major executive orders as those, “either departing from the status
quo of a specific policy that has already been implemented, or interpreting and implementing
legislation that diverts from the original intent of Congress” (Warber 2006, 143)
Alas the actual classification used in that paper seems to have been done in earlier work, which is a book:
- Warber, Adam L. 2006. Executive Orders and the Modern Presidency: Legislating from the Oval Office. Boulder, CO: Lynne Rienner Publishers
So its methods may not have been as rigorous as in some peer-reviewed publications, I don't really know. Anyway, it looks there's not a lot of prior art in classifying executive orders, so whatever method you choose is likely to be okay as long as it's reasonably justifiable in terms of the general approach to document classification.