Institute For The Future

Don't Fear The Black Box

Interpreting the future of AI

 

Written by Matthew Hutson for volume three of Future Now,
IFTF's print magazine powered by our Future 50 Partnership.

In 2013, hackers installed malware onto servers belonging to the retailer Target, eventually stealing the credit and debit card information of 40 million customers. It’s not as though Target wasn’t warned—its FireEye security software generated alerts soon after the initial breach. But the alerts were disregarded. Why? As one security expert told the security news site Dark Reading, “In two words: ‘actionable intelligence’ … The operator/analyst should be able to understand the risk.” The risk was apparently not understood.

Target likely receives many alerts, with little detail about each. FireEye doesn’t discuss client relationships, but their website does advertise “machine-learning techniques” for malware detection. Machine-learning is a set of tools artificial intelligence uses to learn from experience. But how it organizes its accumulated knowledge is not always interpretable to humans, so you can’t always tell why the artificial intelligence (AI) is making certain decisions. It’s possible that Target employees weren’t given enough justification by the software to make the alerts actionable. Perhaps, if it had been able to explain itself better, millions of people would not have been put at risk. Perhaps we’d all be better off if the gadgets around us could tell us what they’re thinking. But do they all need to?

One case in which it would be nice is when they’re responsible for putting people in jail. “A lot of new technologies bypass traditional forms of accountability,” says Madeleine Elish, an anthropologist at the Data & Society Research Institute, “including sometimes legal accountability.” AI is increasingly used to make judgments about people: criminal risk, credit worthiness, and academic promise. Judges, bankers, and admissions officers might make decisions using algorithms they don’t fully understand, and such algorithms could contain hidden bias. A widely cited ProPublica article reported last year that the correctional software COMPAS overestimated the criminal risk of blacks and underestimated the risk of whites. Even if factoring in race didn’t harm accuracy, we’d probably prefer to avoid doing so. Sometimes self-teaching algorithms use factors that we, for reasons of performance or justice, don’t want them to use, pushing researchers to find methods of peering inside their electronic minds.

Transparent algorithms make sense in many other arenas, too. In May, DARPA, the military’s advanced research arm, began a 4-year, $75 million-dollar program on explainable artificial intelligence, or XAI. They’ve contracted with eight academic and four corporate labs to help them make the AI analyzing their intelligence or piloting their autonomous robots more transparent—and to provide an open toolbox of techniques for other developers. AI, they feel, should be more predictable, correctable, and satisfying to work with.

Programmers have a few ways of making machine-learning systems more interpretable. The most occult machine-learning systems use what’s called “deep learning”—multi-layered networks of simple computational units inspired by the architecture of the brain. These deep neural nets break input into many small pieces, send them throughout the network, and pop out a response. What happens in between is essentially an impenetrable black box. So some researchers emphasize or delete certain features of inputs—faces in photographs, say—to see how sensitive the network is to those features, and thus, what role they play in its “reasoning.” Other approaches model the world with more interpretable substitutes for deep neural nets, like decision trees, where you can follow its “if this then that” logic. And others use interpretable algorithms (like decision trees) to model not the world, but the deep neural net, adding an interpretive layer that allows them to keep using the net, but then have a simpler approximation that translates what it’s doing (“if it sees this then it will conclude that”).

Which one approach to use depends on the application and the audience. Programmers might accept an explanation in the form of code. A driver might want his car to talk to him in natural language. Other people might prefer bar charts, or concrete examples of what the system might do in different situations. And is the user trying to debug the system, or learn something new about the structure of the world?

Darpa’s XAI program is addressing these questions by combining explainable models with explanation interfaces based on research in human-computer interaction. One contractor is exploring the psychology of effective explanations and will share recommendations with the others.

For now, enabling users to correct mistakes based on explanations provides extra credit in a nascent field that is still grappling with the basics of machine-learning. What people are calling “explainable AI,” says David Danks, a philosopher at Carnegie Mellon who studies psychology and machine-learning, “is very rarely grounded in what our best cognitive science tells us about the nature of satisfactory explanations.” According to Subbarao Kambhampati, the president of the Association for the Advancement of Artificial Intelligence, “What is an explanation is not dependent on how you arrived at your answer. Explanations are about the other person’s mind.” So ideally, an AI system aimed at explaining itself would model not just its own internal processing, but the user’s as well. Great communicators know their audience.

Communication, of course, is a bedrock of trust, and without trust, AI systems won’t be used. David Gunning, who manages DARPA’s XAI program, describes the program’s origin: an intelligence analyst was telling him about a problem she had. “She’s getting recommendations from these big data analytic routines, but she has to put her name on them, and she doesn’t understand their rationale well enough to be comfortable with that. So that kind of resonated.”

Researchers at Wharton have documented what they call “algorithm aversion”: even when people know that a decision algorithm is more accurate than they are, they’re often unwilling to rely on it if it’s not perfect. They hold it to a higher standard than they do people. That’s in part because they don’t understand how algorithms work and they think people are better at improving. At the most recent O’Reilly Artificial Intelligence Conference, Mark Hammond, the founder of the AI company Bonsai, relayed the tale of a firm that decided not to use a machine-learning system that was 20 percent better than their current method because they didn’t want an artificial ceiling—they didn’t understand the machine-learning system well enough to get under the hood and improve it.

Such aversion is a bit unfair to computers, because humans are also black boxes. We don’t always know how we make up our minds, or we think we know but we have no idea. The ProPublica article was alarming, but even more alarming is the journal article reporting that judges granted parole at rates ranging from 65 percent just after a food break to about 0 percent just before one. If you asked them to explain their judicial decisions, few, if any, would have pointed to tummy rumbles.

It’s possible that an explanation doesn’t even need to explain anything for it to increase user trust. In a classic psychology experiment, people asking to cut in line at a photocopier improved their chances just as much by adding an empty justification, “Because I have to make copies,” as they did by giving a more informative excuse, “Because I’m in a rush.” There might be cases where a machine could get by with saying, “Do X, because it’s the best decision.”

Making AI explainable doesn’t just help reduce unwanted bias and increase trust. It can also improve task performance. Freddy LeCue, a principal scientist in “large-scale reasoning systems” at the large-scale management consultancy Accenture, has worked with airlines to predict flight delays based on their schedules. Computers make predictions after observing historical patterns and finding complex relationships between aspects of flight schedules and delays. If airlines could trace those relationships, he says, they could know which aspects of their schedules to adjust in order to reduce delays. “They can take corrective actions.”

Understanding your AI also gives you a sense of its strengths and weaknesses so you can further improve it—and so you’re not caught off guard by any surprise outcomes. “Sometimes a machine can solve incredibly complicated problems,” Gunning says, “but just makes an incredibly stupid mistake because it has no common sense.”

So explanations help us in at least two ways. First, they enhance our relationship with the algorithm. Imagine you’re deciding whether to follow through on an AI’s recommendation—to, say, investigate a malware alert or to undergo an operation. In some of those cases, you might just need a plausible-sounding reason behind the recommendation so you know the AI isn’t doing something stupid—it increases trust. In others, knowing a model’s strengths and weaknesses might lead you to not apply it in certain types of scenarios to begin with. In still others, you might alter the algorithm, for example, to make it less biased. The second way explanations help is by revealing how to change a predicted outcome. Knowing the key input factors that led to the AI’s prediction (like the elements of a flight schedule it associates with delays) can focus efforts on changing those factors.

Whether explanations help to understand the algorithm or the world it models, there’s often a tradeoff between explainability and algorithm performance. “In a lot of applications, you only care about the performance,” Gunning says. “The explanation is not so important. So, you know, have at it if you’re finding cat videos on Facebook.” Other times, you might choose to sacrifice accuracy for the ability to, say, tell someone why you’re denying him bail, or why she should undergo a risky operation. In those cases, even “just because” won’t cut it. And there are also tradeoffs between profits and rights. As some corporations fight to keep their secret silicon sauce under wraps, some lawmakers are fighting to break those seals, as in the European Union’s recent legislation requiring that people be granted “meaningful information about the logic” behind any automated decision made about them.

So what’s in our future? Gunning sets the stage: “I’m envisioning one of these bar scenes in Star Wars. Not with people but all kinds of little autonomous systems, a huge range of these things, with some little things crawling around cleaning the floor, and you don’t expect much of an explanation out of them, but there’s your, hopefully, really intelligent personal assistant that’s managing your calendar and helping you to set your priorities, and you want that system to give you good explanations and understand the explanations you give back.” It will be a world filled not just with black boxes or transparent boxes, but with a diverse population of boxes, in all shades of translucency.