How does Skopos Labs (cited by govtrack.us) predict the probability of legislation passing? What is their track record & is it the first of its kind?

Question

This answer to Progress moving US oil and gas pipeline cybersecurity regulation and oversight from the TSA to the DOE? (HR 370) begins:

H.R.370 died in the last Congress. The new bill for the 117th Congress is H.R.3078 - Pipeline and LNG Facility Cybersecurity Preparedness Act and was reported June 10, 2021; but has not been otherwise acted on.

govtrack reports,

Prognosis - 2% chance of being enacted according to Skopos Labs

The fate of any given US bill and the probability that it will pass, fail or flounder is of extreme political interest to all parties, including legislators, lobbyists and to donors to each as well as to the press and the public.

Predictions thereof may (or may not) be useful to bolster or throttle funds, time and effort by all parties involved. Therefore these predictions can become an integral part of the political process and if it is subject to bias, influence or simply prone to error (statistical or otherwise) those are also introduced into the political process.

In the case of the legislation cited above, GovTrack.us says:

Prognosis: 2% chance of being enacted according to Skopos Labs (details)

and clicking "details" opens a text box that says:

Prognosis Details

This bill has a 2% chance of being enacted.

Factors considered:

The bill's primary sponsor is a Republican. The bill is assigned to the House Energy and Commerce committee.

(Factors are based on correlations which may not indicate causation.)

Predictions are by Skopos Labs.

Question(s):

1. How does Skopos Labs (cited by govtrack.us) predict the probability of legislation passing?
2. What is their track record? Can anything be said about the reliability of the method?
3. is it the first source of legislation success prediction of its kind?

These are so closely related, and sources for answers will overlap to such an extent that I think that in this case it's better to keep them together.

---

Basic information about the two sites (note: I have no affiliation to either):

GovTrack.us

Tracking the United States Congress

GovTrack.us began in 2004 as a project to use technology to make the U.S. Congress more open and accessible. Today we’re the leading non-governmental source of legislative information and statistics.

and

skoposlabs.com

The Skopos team created a machine learning and natural language processing platform.

Brooklyn Investment Group is our wholly-owned SEC Registered Investment Advisor subsidiary.

​Skopos is backed by a global investment bank, Thomson Reuters, and venture capital funds. We're headquartered in Brooklyn, New York.

Answer 1

How does Skopos Labs (cited by govtrack.us) predict the probability of legislation passing?

Skopos' approach is a machine-learning one, and while the algorithm itself isn't publicly available, the ideas behind it are described in full detail in John J. Nay's 2016 paper Predicting and Understanding Law-Making with Machine Learning. Using a corpus of bills from the 113th Congress (2013-15), the algorithm combines a "text model" based solely on semantic analysis of each sentence of a bill with a "context model", based on factors such as sponsorship by a committee chair, whether the bill passed a previous Congress but ran out of legislative time, and so on.

The semantic analysis in particular is apparently able to identify the key provisions in a bill which are most likely to affect its probability of being signed into law. An example described in the paper is of the terms 'anthropogenic' and 'sequestration', relating to climate change. Bills containing these terms generally fail to pass the Senate. On the other hand, bills relating to commemorations were found to be far more likely to be enacted.

What is their track record? Can anything be said about the reliability of the method?

Their track record seems to be fairly good - a 2018 report on the company's partnership with Wolters Kluwer to implement legislative predictions into their Federal Development Knowledge Center gave accuracy rates of 99% when predicting whether a bill would pass its first chamber, and 98% accuracy on predictions of whether a bill would be signed into law. Of course, because the algorithm produces a percentage likelihood of being enacted rather than a simple 'yes/no' determination, this is quite hard to measure.

Is it the first source of legislation success prediction of its kind?

No - from 2013 to 2016 GovTrack used a statistical analysis method based on a combination of context factors and textual analysis of a bill's title which was also fairly successful, and commercial tools such as FiscalNote described here by the Washington Post also use machine learning models based on similar approaches.

In 2011, Gerrish & Blei published Predicting Legislative Roll Calls from Text, in which they developed several models to predict voting patterns, while a 2012 paper by Yano, Smith, and Wilkerson focused on a textual-analysis-only approach to predict whether a bill would survive a congressional committee. A little later, in 2016, Kraft, Jain & Rush published An Embedding Model for Predicting Roll-Call Votes which used predictive models to "analyze which features in a bill text are most predictive of political support."