Institute For The Future

Simulation And the Danger of Confusing it with Prediction

The BBC had a fascinating article the other day running under the unfortunate headline of "Supercomputer Predicts Revolution," about an intriguing effort by Kalev Leetaru to use "tone and location" to forecast political revolutions and uprisings, and otherwise anticipate large-scale social disruptions.

 

As the BBC describes the study:

 

The study's information was taken from a range of sources including the US government-run Open Source Centre and BBC Monitoring, both of which monitor local media output around the world.

News outlets which published online versions were also analysed, as was the New York Times' archive, going back to 1945.

In total, Mr Leetaru gathered more than 100 million articles.

Reports were analysed for two main types of information: mood - whether the article represented good news or bad news, and location - where events were happening and the location of other participants in the story.

Mood detection, or "automated sentiment mining" searched for words such as "terrible", "horrific" or "nice".

Location, or "geocoding" took mentions of specific places, such as "Cairo" and converted them in to coordinates that could be plotted on a map.

Analysis of story elements was used to create an interconnected web of 100 trillion relationships.

 

And Leetaru's results were impressive; his model showed consistent shifts in sentiment in Egypt, Kuwait and other countries just before uprisings, and also showed stability in neighboring countries like Saudi Arabia that remained politically stable. More impressively, Leetaru claims that:

 

While far from a definitive lock on Bin Laden’s location, global news content would have suggested Northern Pakistan in a 200 km. radius around Islamabad and Peshawar as his most likely location, and that he was nearly twice as likely to be making his residence in Pakistan as Afghanistan.

 

Again - it's impressive stuff - which is, at some level, what gives me a bit of pause, particularly in light of some other mentions I've seen of emerging efforts to use data and other creative tools to understand the future. The police department of Santa Cruz, California is using recent crime data to forecast likely targets for the next day's crimes; Fast Company reports on a new effort to build a $200 million city capable of housing 350,000 but home to no one to test out new technologies and techniques to manage urban life; perhaps most intriguingly, Technology Review had a long feature by a writer who had his blood cells reverse engineered into heart cells to test potential effects of different drugs--a concept which he speculates may "become a routine part of medical care."

 

We are, in other words, on the verge of much more robust and impressive efforts to simulate future possibilities. Which is, by itself, a great thing. We need more and better efforts to think about the future.

 

What concerns me, though, is the possibility that we won't recognize the limits of these simulations. For example, I noted the BBC headline - "Supercomputer predicts revolutions", and the Boston Globe, which wrote about efforts to forecast crime locations, wasn't much better with its headline of "Introducing: Predictive Policing." While more technical efforts, like Kaley Leetaru's academic paper describing his efforts to anticipate uprisings, acknowledge the limits of relying on the past to anticipate the future, and otherwise point out uncertainties, more public efforts hype the most impressive findings.

 

During the financial crisis of 2008, NPR reporter Adam Davidson described the faith in housing market bonds as "the triumph of data over common sense," as a means to describe financial analysts who essentially placed undue amounts of faith in financial models that turned out to rely on bad assumptions. Part of the problem was that, over time, the analysts stopped thinking about their assumptions and focused on the data.

 

We're on the verge of having a lot more data about what the future could look like. Let's hope that rather than letting that data triumph over common sense, we'll be willing to acknowledge the limits of that data, and instead of thinking of simulations as firm depictions of future states, will instead understand them to be useful tools for gauging and anticipating possibilities.