Dramatic success in machine learning has led to an explosion of new AI capabilities. Continued advances promise to produce autonomous systems that perceive, learn, decide, and act on their own. These systems offer tremendous benefits, but their effectiveness will be limited by the machine’s inability to explain its decisions and actions to human users. This issue is especially important for the Department of Defense (DoD), which is facing challenges that demand the development of more intelligent, autonomous, and symbiotic systems. Explainable AI will be essential if users are to understand, appropriately trust, and effectively manage this incoming generation of artificially intelligent partners.
The problem of explainability is, to some extent, the result of AI’s success.
In the early days of AI, the predominant reasoning methods were logical and symbolic. These early systems reasoned by performing some form of logical inference on (somewhat) human readable symbols. Early systems could generate a trace of their inference steps, which then became the basis for explanation. As a result, there was significant work on how to make these systems explainable.
AI Tell it to me like I am a human six year old
DARPA is interested in creating technology to make this new generation of AI systems explainable. Because the most critical and most opaque components are based on machine learning, XAI (eXplainable Artificial Intelligence) is focusing on the development of explainable machine learning techniques. By creating new machine learning methods to produce more explainable models and combining them with explanation techniques, XAI aims to help users understand, appropriately trust, and effectively manage the emerging generation of AI systems.
The target of XAI is an end user who depends on decisions, recommendations, or actions produced by an AI system, and therefore needs to understand the rationale for the system’s decisions. For example, an intelligence analyst who receives recommendations from a big data analytics algorithm needs to understand why the algorithm has recommended certain activity for further investigation. Similarly, a test operator of a newly developed autonomous system will need to understand why the system makes its decisions so that he/she can decide how to use it in future missions.
This concept presents three related research and development challenges:
(1) how to produce more explainable models;
(2) how to design the explanation interface; and
(3) how to understand the psychological requirements for effective explanations.
The XAI program would like to improve explainability while maintaining a high level of learning performance for a range of machine learning techniques. There is an inherent tension between machine learning performance (predictive accuracy) and explainability; often the highest performing methods (e.g., deep learning) are the least explainable, and the most explainable (e.g., decision trees) are less accurate. The program plans to fund a variety of machine learning techniques to provide future developers with a range of design options covering the performance versus explainability trade space.
Here are three possible strategies to illustrate the range, but not the entire space, of approaches that may be pursued:
* Deep Explanation: Develop modified or hybrid deep learning techniques that learn more explainable features, explainable representations, or explanation generation facilities. There are a number of design choices that affect deep learning: the selection of training data, initial conditions, architectural layers, loss functions, regularization, optimization techniques, and training sequences.
These design choices might be engineered to produce more explainable representations. Recent work has made several small steps towards explainability. Researchers have used deconvolutional networks to visualize the layers of convolutional networks.
There are a variety of techniques for associating semantic concepts with nodes of a deep network, such as those used to identify items in a visual scene and recount multimedia events. Moreover, it might be possible to extend the approaches used to generate image captions to train a second deep network to generate explanations without explicitly identifying the sematic features of the original network. Because deep learning is so prominent, it is important for the XAI program to pursue several approaches to make these systems more explainable.
* Interpretable Models: Develop alternative machine learning techniques that learn more structured, interpretable, or causal models. There is a wide variety of techniques that may be pursued: learning simpler, more compact models, such as Bayesian Rule Lists; learning richer, more conceptual, generative models, such as Bayesian Program Learning, learning models of causal relationships using stochastic grammars to learn more interpretable structure
* Model Induction: Develop techniques that would experiment with any given machine learning model—as a black box—to infer an approximate, explainable model. Demonstrate an example of such a model-agnostic explanation system that infers explanations by observing and analyzing the input-output behavior of the original black-box model. Moreover, it may be possible to develop richer modelagnostic
techniques by using abduction, reasoning, and story generation to “rationalize” plausible explanations of the system’s reasoning. Because these techniques could apply to almost any machine learning application, it is important for the XAI program to also pursue promising techniques for model induction.
DARPA anticipates the design and development of effective interfaces by integrating state-ofthe-art HCI (Human Computer Interface) techniques with new strategies for presenting understandable explanations.
Kulesza, Burnett, Wong, and Stumpf (2015) provide an example of the development and evaluation of a basic explanation interface. The work followed a complete development strategy that included identifying principles of explainability, developing a prototype interface from those principles, and evaluating the effectiveness of the explanations provided by the prototype. It was basic in the sense that the system explained a very simple machine learning model, a naïve Bayesian text classifier. The XAI program is seeking developments that go beyond this work by explaining more complex learning systems, by devising additional explanation strategies, and by applying more sophisticated HCI techniques where appropriate.
The design of an effective explanation interface should consider a range of HCI and cognitive science techniques.
Psychology of Explanation
In addition to the application of psychology to the design of the explanation interface, the program is interested in work focused primarily on psychological theories of explanation.