15

Let's take Colorado for example.

Considering the prior existence of penalties for marijuana use, it is reasonable to expect that people may have been less inclined to admit their marijuana usage prior to its legalization.

Therefore, even if the data reflects an increase in self-reported marijuana usage following its legalization, we will still be unable to determine whether this increase is genuine or simply attributable to individuals feeling more comfortable disclosing their marijuana consumption.

So my question is: using self-reporting questionaires, how are we supposed to know if legalization of marijuana leads to increases in marijuana usage?

6
  • 10
    You don't and good studies will either have that in their fine print or a method of how they hope to account for that. Also more studies and asking about that. "How long have you been consuming?", "Did you ever lie about your consumption in a survey?". It won't be perfect but it can give you a hint about the numbers.
    – haxor789
    Jun 21, 2023 at 12:13
  • 3
    One useful concept is to recognize that part of the goal is to quantify as best you can the uncertainty. Basically no methodology is 100% certain, but that are methods that one can use to estimate the magnitude of the effects if there is one.
    – ohwilleke
    Jun 21, 2023 at 18:35
  • The other ways of counting are all problematic as well. Arrest data is affected by legalization. It becomes more present of mind and looked for by cops/investigators. Employees get less furtive and more caught at the same usage rate. There is no way to really know...
    – dandavis
    Jun 21, 2023 at 21:30
  • 6
    See if local stores selling late night munchies are reporting an increase in sales
    – dm63
    Jun 21, 2023 at 22:14
  • 2
    agreed with @ohwilleke. Probably impossible to know the real prevalence, but an approximation based on a sound methodology may be good enough to decide on policies to conduct. The definition of a "sound methodology" may (and should) be debated. But if meta-analyses and various well-thought-out methods converge to a given estimate, it may be reasonable to trust them to a certain extent, for example if you have to make a choice relative to public health policies.
    – J-J-J
    Jun 22, 2023 at 9:55

1 Answer 1

14

This is a good question, and you should look at the methodology of the specific survey you have in mind (which hopefully should be detailed in their report) to find out how they address that and justify the validity of the study.

For example, this interesting study of cannabis use prevalence in Canada from 1985 to 2015, examines nine national surveys and admits that:

Nevertheless, results of this study should be interpreted in light of several limitations. Changes over time in respondents’ willingness to admit drug use, in their definition of what constitutes drug use, and in the perceived or real risk of legal consequences could neither be controlled nor detected, but could affect trends

In the methodology of the 2021 US National Survey on Drug Use and Health, they address this issue by explaining:

Survey questions about topics such as substance use are considered to be sensitive because respondents may think the questions are intrusive (“none of your business”), pose risks for negative social or legal consequences if their answers were to become known, or require them to provide socially undesirable answers (Tourangeau & Yan, 2007).

[...]

NSDUH utilizes widely accepted methodological practices for increasing the accuracy of self-reports, such as encouraging privacy through self-administration of questions about sensitive topics—including audio computer-assisted self-interviewing (ACASI) for in-person data collection—and providing assurances that individual responses will remain confidential. Comparisons using these methods within NSDUH data (collected in person) have shown they reduce reporting bias (Gfroerer et al., 2002).

In addition, in academia, they may use specific methods to address this kind of bias (e.g. social-desirability bias), including randomized response and list experiments. These methods allow plausible deniability for the respondent, hence limiting bias.

The principle of these methods is to make it impossible to know if a specific respondent uses drug or not; but if you aggregate all the responses, you can still estimate general prevalence after conducting some additional calculations. As for the details of each method, you should read the previous links. Both methods can have some issues or limitations, but may be better than asking directly a sensitive question. I mention specifically academia as using these methods, because I'm not sure that any national survey institute use these methods to estimate drug use prevalence (but happy to be corrected here).

Not all drug use surveys conducted in academia will use these "plausible deniability" methods, but here are some examples from academic papers using randomized response for estimating prevalence of cannabis/marijuana use or other drugs:

In addition, besides surveys, there are indirect methods for estimating prevalence, in particular when it's about drug used by a small proportion of the population or by a hard-to-reach population (e.g. heroin users). These methods include "multiplier-benchmark", "capture-recapture" (see this UN document from 2003 for an account of these two methods), or other methods. They are all subject to criticism, like any method.

3
  • 3
    Difference in differences methods often by helpful in screening out a lot of potential confounding factors. en.wikipedia.org/wiki/Difference_in_differences Quantifying potential effect size from similar situations in the past (e.g. liquor use trends before and after prohibition was repealed) can also cabin the scale of what you might reasonable estimate the effect to be.
    – ohwilleke
    Jun 21, 2023 at 18:38
  • 2
    Another useful indirect way to measure drug use is monitoring wastewater for drug metabolites (see emcdda.europa.eu/publications/html/pods/waste-water-analysis_en) - if only willingness to report drug use (for yourself, your patient, ...) changes, the wastewater shouldn't show much change. Jun 22, 2023 at 14:23
  • 1
    @MartinModrák Great idea, although only useful for community level aggregate usage as opposed to usage by a particular demographic like teens.
    – ohwilleke
    Jun 22, 2023 at 17:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .