DARPA making AI to explain why

The field of AI has made great strides in the last several years, thanks to developments in machine learning algorithms and deep learning systems based on artificial neural networks (ANNs). Researchers have found that vast sets of example data are the way to train up such systems to produce the desired results, whether that is picking out a face from a photograph or recognizing speech input.

But the resultant systems often turn out to operate as an inscrutable “black box” and even their developers find themselves unable to explain why it arrived at a particular decision. That may soon prove unacceptable in areas where an AI’s decisions could have an impact on people’s lives, such as employment, mortgage lending, or self-driving vehicles.

The value of so-called explainable AI was called into question recently by Google research director Peter Norvig, who noted that humans are not very good at explaining their decision-making either, and claimed that the performance of an AI system could be gauged simply by observing its outputs over time.

Explainable AI can help uncover problems with a system. A machine learning tool developed at Vanderbilt University in Tennessee to identify cases of colon cancer from patients’ electronic records. Although it performed well at first, the developer eventually discovered it was picking up on the fact that patients with confirmed cases were sent to a particular clinic rather than clues from their health records.

Arxiv – Attentive Explanations: Justifying Decisions and Pointing to the Evidence

Researchers have been following three broad strategies.

1. is deep explanation, whereby modified deep learning techniques are developed that are capable of producing an explainable model. This could be achieved by forcing the neural network to associate nodes at one level within the network with semantic attributes that humans can understand, for example.

2. use a different machine learning technique, such as a decision tree, that will produce an interpretable model.

3. model induction, and is basically a version of the black box testing method. An external system runs millions of simulations, feeding the machine learning model with a variety of different inputs and seeing if it can infer a model that explains what the system’s decision logic is.

Explainable AI then has to convey or present its decision-making process to the human operator through some form of user interface.