Trust but Verify: Confronting the unique challenges of LLMs in intelligence analysis

Abstract

Organizations across the U.S. intelligence community are racing to integrate LLMs in what may become one of the technology’s most consequential applications. This paper argues that intelligence is not just a high-stakes version of typical LLM use. First, at the implementation level, we define a set of unique challenges and assumptions by assessing where core LLM tendencies diverge from intelligence needs. To address this misalignment, we propose a set of system design principles that mitigate their effects. Then, this paper evaluates these challenges and principles in a practical intelligence workflow by building our own system. We introduce a platform capable of autonomously monitoring open-source intelligence on X/Twitter to visualize and analyze geopolitical events in real time. The results highlight the impact of system design on LLM outcomes, while demonstrating weaknesses with adversarial content and analysis production. Our work seeks to encourage the intelligence community to build more trustworthy interfaces, systems, and policies as LLM integration continues.