2

Are there any instances where a government used "test" laws to determine the effect of a policy before enacting that policy on the whole population. For example, before rolling out a plan to the whole country, give some money to a region who wants to test it, or make an A/B test with two regions? What were the justifications used for this kind of testing?

  • 1
    How do you propose to equalize other conditions in two regions to do proper A/B tests? – user4012 Aug 16 '18 at 13:06
  • 3
    "Does it make sense" is an opinion-based question. Depending on the law, the regions you want to test it on and your personal threshold for "making sense", you can come to completely different conclusions whether it "makes sense" or not. What you could ask instead is if it is constitutional to do this in your favorite constituency or if there are any such experiments in your favorite political area of interest. For example, there were indeed regional experiments to test what happens if everyone receives basic income or if everyone must own a firearm in their home. – Philipp Aug 16 '18 at 13:09
  • @Philipp "does make sense" is sure not the proper wording as I refer to increasing the human development index – J. Doe Aug 16 '18 at 13:11
  • 3
    @J.Doe Depending on how you wish "law" to be interpreted, it's arguable that this already happens. Across multiple jurisdictions, regulations and procedures are frequently trialled in smaller geographical regions before being introduced nationally. Of course, primary legislation is a different matter. – origimbo Aug 16 '18 at 13:19
  • 2
    It's not precisely geographic, but Finland had a randomised trial of Universal Basic income as a welfare scheme. In the end they chose not to adopt it. theguardian.com/world/2018/apr/23/… – origimbo Aug 16 '18 at 13:25
3

These are called policy pilots. The UK government has a survey of them circa 2003 (pretty dated), covering the US and UK, but most are pretty obscure, e.g.

Employment Retention and Advancement (ERA) Scheme: Aim: A trial of the effectiveness of new services to improve job retention and advancement prospects for low-wage workers

or

Drug Treatment and Testing Orders (DTTOs) – [Scottish Executive] Aim: Pilot to inform decisions on whether to introduce DTTOs in Scotland and to provide evidence on the logistical, financial and crime-reduction implications of the policy.

Also the survey notes that

Greenberg and Shroder’s Digest of Social Experiments (1997) describes over 140 US policy trials of one kind or another. [...] Some of these trials were designed to measure impact, some process and some both, but they were all aimed to assess as accurately as possible a particular option (or set of options) against the counterfactual.

As was noted in the comments, a better known one is the one from Finland on basic income.

Also the UK survey notes this interesting distinction between US and UK

For whatever reason, most policy trials which would routinely employ randomised trials of individuals in the USA tend to be conducted by somewhat less rigorous methods in Britain. This is partly a function of different political systems. Many policies in the USA are implemented and evaluated within one state in advance of, and with no commitment to, a national roll-out. Whether or not backed by federal funds, these are genuinely pilot schemes which will be abandoned if they prove ineffective. Britain’s more centralised structure makes this sort of experimentation and innovation more tricky. As noted, many more policies here are based on manifesto commitments or other well-amplified prior announcements, which means that there is stronger party commitment to their success. So a great deal of political capital is thus invested in ‘proving’ the success of the policy in Britain – circumstances that do not amount to optimal experimental conditions.

Basically, more decentralization seems to be more conductive to performing such policy pilots... only because one region may decide by itself (since it can) to implement a change, thus effectively acting as a pilot for the rest of the country; but such pilots aren't too well controlled in terms of alternatives, confounders etc. Probably "natural experiment" is a better term for region-based pilots, but that's just my opinion.

  • 1
    Quasi-experiment might be a useful term for region-based pilot. A natural experiment is something else: It's essentially an observational study where researchers discover some quirky variable that assigned subjects to a treatment after the fact, without them realizing it necessarily. – Relaxed Aug 17 '18 at 7:43
3

In the United States, there is a notion that states are "testbeds" for new policies. In this sense, each state tests policies before the federal government enacts (or doesn't enact) them for the entire national population.

This is commonly taught in introductory courses in American government in discussions about federalism, but it was more formally articulated by Supreme Court Justice Brandeis in 1932.

As an example, marijuana-usage laws vary between states: enter image description here [Source - Governing Magazine]

In some sense, states are trying out different policies which may be adopted later at the federal level.

1

It's interesting to see the A/B test terminology show up in many places from policy to cooking where it is in fact just the way the web/software development community calls good old scientific experiments. The basic method exists since at least the 16th century and it has been greatly refined and routinely used in applied settings during the last century, for example in agronomy, medicine or psychotherapy. Other fields haven't waited for web-based A/B tests to empirically evaluate interventions, whether you call that “clinical trials”, “experiments” or something else. So it does make sense and has been done for decades: Many countries regularly roll out policies in a limited “pilot” or “trial”, often in one state, province, city, office, road, train station, etc.

In policy and economics, one difficulty is that it's virtually impossible to have a purely experimental approach, where a large number of test units are randomly assigned a treatment. At most you can introduce a tentative policy to a country or a couple of regions and compare with other countries, but that's only one datum and it's difficult to tease out the effect of your intervention (the policy change) from a myriad other factors (a related idea is that of a “quasi-experiment”). You cannot properly analyze this at the individual level either (like you would visits on a website) because the inhabitants of a specific area have a lot in common that might distort the results.

Another thorny issue is the definition and measurement of the outcome. The human development index is a composite index trying to summarize several, presumably independent, variables (although in this particular case, this particular point is disputed). As such, the weight you give to these variables or factors is open for debate. Translating a complex concept like development in a measurable quantity (“operationalizing”) is not trivial, practically and theoretically. And once a measure is recognized as something desirable or used to evaluate performance, people will tend change their behavior to target the measure itself or game the system (this is known among others as Goodhart's law or Campbell's law).

Rolling out a policy also involves significant costs. You cannot simply try any intervention that comes to mind and get quick results by deploying some new version on a server for a few hours without anyone noticing. You need to flesh out the policy, create a legal basis for the trial, get buy-in from various stakeholders, train civil servants, etc. To justify all this, you also need a strong case that the policy could work and after so much investment in the trial, it's difficult to get an unbiased evaluation. By the time the trial is in place, there will be many people with an emotional, political or financial stake in its success.

Because of these technical difficulties, the evaluation of the effect is always fraught and involves a lot of modelling and you can hardly hope to “solve” political disagreements that way.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .