How to Answer a Question? Rethinking Open-Domain Question Answering with Multi-Type Questions

Abstract

Previous research on open-domain question answering (QA) mainly focuses on questions with short answers. However, information-seeking QA often requires various formats of answers depending on the nature of the questions, e.g., why/how questions typically require a long answer. In this paper, we present MoQA1, a benchmark for open-domain QA that requires building one system that can provide short, medium, long and yes/no answers to different questions accordingly. We build MoQA based on Natural Questions [21] with more fine-grained annotations provided through crowdsourcing. We adapt state-of-the-art models to this task, and reveal unique findings in multi-type open-domain QA: (1) For retriever-reader models, the difficulty of retrieving passages for different question types is relatively balanced, and training one retriever on all types achieves the overall best performance. However, it is challenging to train one reader model to output answers of different formats, or to train a question classifier to distinguish between types; (2) An end-to-end closed-book QA model trained on multiple types struggles with the task across the board; (3) Few-shot large language models such as GPT-3, Codex, and InstructGPT lag behind trained baselines and fails to adapt to the different question types. Our benchmarkand analysis call for more effort into building versatile open-domain QA models in the future.