7

Stratfor Global Intelligence is a reasonably high profile private company which uses publicly available information and geopolitical methodology to analyze the situation in the world and provides the analysis to the subscribers.

Alongside analysis of current events they also publish predictions, both less-formal (as bits in their analysis pieces) as well as more formal (forecasts, both world and local).


Question

Is there any resource which tracks Stratfor's past predictions (formal and/or informal) for accuracy?

(the resource needs not be dedicated to Stratfor exclusively, but must include them).


Note

As random examples I recall, they very accurately predicted no-outcome Green Movement in Iran, the Muslim Brotherhood emerging as clear winners in Egypt's events; but dropped the balls in some of their predictions around Israel's 2006 incursion into Lebanon.

1
  • 1
    Fact-check groups usually look at government statements, not research groups. Such groups are likely to be rare, but should include newspapers. Good question.
    – Lizz
    Apr 14 '13 at 5:10
2

I actually found a source on this, but it's pretty obscure, appearing (in English) in a Romanian journal "STRATFOR VS. REALITY (1995-2025). DILEMMAS IN GLOBAL FORECASTING"

The abstract of the paper is kinda annoying as it's more of a teaser than actual abstract. The paper also lacks any real statistics, but has this conclusion, which contains some valid methodological criticism, in particular with respect to the open-endedness of the time frame of [some of] the forecasts and to the [complete] lack of forecast of some significant developments:

So what can we truly conclude after an exhaustive evaluation of Stratfor grand decade reports? Similarly to the forecasts we interpreted, our analysis is tributary to the glass hypothesis: what is more important in the end – the empty half, or one that is full? Are we therefore more surprised by what they predicted or by their mistakes? Naturally, critics stress on the sometimes resounding failures of Stratfor predictions – this is not done necessarily in bad faith, but must be juxtaposed to Stratfors’ claim to deliver forecasts. On the whole, the most conspicuous mistakes in their ten-year forecast are not necessarily perturbing – the world of international relations is extremely dynamic, not functioning as an engineered, relatively closed, systemic and clearly-marked causality mechanism, but inhabited by actors and stakeholders portraying different typologies of agency. As such, any prognosis could be valid in the context of a certain hypothesis. The substantial failure of Stratfor forecasting lies not in how events historically developed, but in that major events have not been anticipated at all – and exactly those are the ones reformulating international relations and the international system itself (international terrorism at the beginning of 2000, the failure of democracy in areas such as the Middle East or Northern Africa in the past twenty years, the Islamic State phenomena, etc.).

[...]

Stratfor forecasts are close-fisted, even criptic, when it comes to their methodology. Also, historicism seems to be an essential incredient for Stratfor „mathematics”, fact only made more obvious by a certain oversimplifications employed. No one expects Stratfor experts to guess the future, of course, and we accept that such products have a comercial, mediatic and strategic value. Most times though, Stratfor correctly intuits linear historical paths (such as that of the US), is obstinate in continously repeating the same scenarios until they finally take place (the repositioning of Russia as enemy of the West) and regularly postpones dezasters portraied as minor signals (such as the fall of the European Union and China) – for Stratfor analysits it doesn’t appear to be fundamentally paramount if their predictions will happen now or in the next 100 years.

If you have the patience, you could go over the paper and construct a summary statistic form the (fairly numerous) examples they analyze, but I'm not sure the sampling is unbiased.

FWIW, the also quote at one point

Although Stratfor defines itself as a ”private global intelligence company” employing several former CIA analysts, its reputation among a number of policy and international relations experts is poor, some going to the length of saying that ”Stratfor is just The Economist a week later and several hundred times more expensive” (Max Fisher, 2012).

A much better methodology is in "Measuring the Forecast Accuracy of Intelligence Products". They first point out the methodological issues with such intelligence forecasts: lack of actual probabilities in statements, but the paper works around this by assigning some based on the language used (as sort of sentiment analysis). With that in had they do offer the comparison below, but only for a small sample of forecasts:

https://i.stack.imgur.com/C5XKO.png

(Image currently broken due to SE/imgur bug; see figure 2 in paper)

As an example of a comparative analysis, Figure 2 shows the difference between the NIEs, which originally were produced as classified products and the open source products. Visually they appear similar, and the description of the forecasts as somewhat conservative, reasonably calibrated yet still a little overconfident applies equally to both.

We urge the reader not to draw strong substantive conclusions from this comparison. Only ten documents were examined, with only two of them NIEs. Instead, readers should simply recognize that this type of accuracy measurement and comparative accuracy analysis are straightforward to perform.

Somebody could in theory apply this to a large body of Stratfor's forecasts, but it's probably a lot of work... and actually the same authors have done that themselves a couple of years later in "Using Inferred Probabilities to Measure the Accuracy of Imprecise Forecasts" for 187 events, mostly using Stratfor's forecasts. They are actually pretty coy not to draw any conclusions about Stratfor, but you probably can from the data (see table 2 in the paper, again it won't show up here for now.)

table 2

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .