Why Did The Model Fail? Empowering Non-Experts to Explore Computer Vision Models through Interactive Dialogue

Abstract

Modern computer vision (CV) models often generate predictions without explanations of how or why they arrived at those outcomes. While significant efforts have been made toward explainable AI (XAI) to help people understand model behavior and outputs, many available tools are either geared towards technical experts or not applicable to CV models. In this work, we address this gap by developing an accessible and easy-to-use tool that empowers people without technical expertise to explore CV models and understand their behavior. To make the tool intuitive and user-friendly, we choose dialogue as the mode of interaction. Concretely, we build on an existing tool named TalkToModel that was developed for exploring models trained on tabular datasets and create TalkToVisionModel, an interactive model exploration tool that answers user questions about a CV model. We instantiate this tool to explain a ResNet18-based scene classification model trained on the Places365 dataset. We design and conduct a user study to evaluate the tool's usability and effectiveness in helping people understand model behavior, which we measure by their ability to simulate model predictions. We recruit 10 participants, 5 with a technical background and 5 without. We find that non-technical and technical users enjoy using TalkToVisionModel, find it helpful for exploring a CV model, and would use a similar dialogue-based tool in the future. We also observe that seeing examples of the model's other predictions is more helpful than seeing concept-based explanations in identifying model mistakes. Additionally, we find that technical participants desire more fine-grained control, while non-technical participants enjoy the exploratory interface, and that a dialogue-based system may be better suited for the latter group. This work demonstrates the potential of dialogue-based interactive model explanation tools in the computer vision domain.