Institute For The Future

From #20forecasts: Risks of an algorithm arms race

An algorithm arms race risks a Sorcerer's Apprentice scenario

As prediction algorithms are deployed to detect geopolitical and financial discontinuities, they will increasingly be designed to anticipate and outdo one another. Once these bots are empowered to act upon and adjust our increasingly critical connected systems—such as global markets, smart cities, and social networks—competing programs could create chaos. Financial trading algorithms, for example, have already been implicated as the cause of market "flash crashes".

SO WHAT? Chaos caused by competing algorithms could thwart our ultimate goal of fine-tuning systems to avoid potential problems.

Strong Signal: Machines Rule Everything Around Me

At first blush, it may seem absurd to think of the variety of ways that algorithms run the world around us. After all, we don’t live in the Matrix, and reality is vastly too complex for a mere computer program to be able to determine the quality of life in a city or predict when we’ll next buy a roll of toilet paper. Right?

Yet a quick peak under the hood offers a different narrative. The industrial use of algorithms is increasing at an alarming rate. Used in everything from recommending movies, suggesting ads, developing delivery routes, trading commodities, algorithms are now an integral and ever-growing part of our everyday lives.

But what happens when systems become dependent on algorithms to help manage our critical infrastructures such as markets and energy grids? Finding ways to avoid the unintended behaviors of errant AI may be our only defense against potential calamity.

Sub-forecasts:

Smart Cities Go Haywire

The growth of smart cities is increasing the world over. Public administrators and policy makers everywhere from Dubai to Portland, are pursuing efforts to make their cities more connected and intelligent, leveraging advances in communication technology and the Internet of Things to connect everything from bridges to buildings to vehicles to each other. With the smart buildings industry set to expand to $22 B by 2017, it’s clear this trend is going to continue to grow. But with the proliferation of smart cities, how secure are the systems managing them?

Attacks on Supervisory Control and Data Acquisition (SCADA) systems within cities have been occurring for years. SCADA systems – which control critical infrastructures and will likely provide the connective tissue for smart cities – have been a focus of major scrutiny for their vulnerability to cyber attack. As the capabilities of hackers and potentially rogue algorithms grow, imagining scenarios where everything from traffic lights to water sanitation become damaged by cyber attacks does not seem infeasible. Indeed, the possibilities for smart cities to go haywire seem very real.

HFT Algos Go Rogue

Between 2005 and 2009 the amount of trading activity conducted by high-frequency trading (HFT) algorithms grew as much as 164% on US equity markets. Today, as much as 95% of all trading activity is originated by HFT machines, making up millions and millions of trades per day. While the negative consequences have been relatively minor, so it seems, several examples of flash crashes and micro flash crashes have occurred in recent years. In 2012, Knight Capital famously lost over $440 M in less than 30 minutes due to a glitch in one of its HFT algorithms. Some HFT algorithms have even been shown to launch Denial of Service attacks on stocks – effectively shutting down all trading activity on the affected company or fund.

In addition, as Michael Lewis points out in his book, Flash Boys: A Wall Street Revolt, HFT causes distortions to markets in a variety of ways. Predictive algorithms can sense when a human investor is about to make a trade, jump in moments before and capture the minor price inflation by buying the stock and selling it back to the investor right as he or she buys.

Such fluctuations can not only rig the market, but over time cause instabilities in unforeseen ways. As such technologies advance, keeping in mind their proper utilization, and developing a sound understanding of their costs and benefits, should be a priority for financial institutions and regulators alike.

Summary

With the combined growth in the use of algorithms to automate dissemination of information across critical infrastructures along with the increasing interconnectedness of our system, the risk of a scenario where our most important systems are vulnerable to attack or failure becomes very real. Securing such systems should be a top priority for policy makers and business leaders alike. Avoiding the worst of what may come from an algorithm arms race will likely mean the cooperation of humans to determine which systems are most vulnerable, and working collaboratively to address major concerns before they become a casualty of our own pursuit of progress.

---

This post is the 4th in a series exploring IFTF's new interactive map from our Technology Horizons program, 20 Combinatorial Forecasts—featured in FastCompany Co.EXIST's "When Technologies Combine, Amazing Innovation Happens." Every few weeks we'll take a deep dive into another of the map's twenty forecasts on the future of technology at the intersections of different fields.

All posts in the series »

Curious about the Technology Horizons program?

• Follow the forecasts at @iftf and #20forecasts
• Find out more about the program
• Check out previous years' Technology Horizons research
• Contact Sean Ness ([email protected])