Where VLMs Look, and What They Miss: Comparing Visual Attention in Reasoning and Non-Reasoning Vision-Language Models

Abstract

Vision-Language Models (VLMs) process visual inputs in addition to text, making them powerful tools for real-world scene interpretation. A growing line of work fine-tunes them to produce explicit reasoning chains before answering, and several VLM families now release paired non-reasoning (Instruct) and reasoning (Thinking) model variants on shared weights. While Thinking variants generally achieve higher accuracy, prior work has shown that visual attention decreases as reasoning length grows, which may lead to model hallucinations. Existing analyses, however, do not compare the attention patterns of paired, same-weight reasoning variants. This thesis uses a comparative study design to examine the Instruct and Thinking variants of the Qwen3-VL model family at 8B and 30B scales, evaluated on CLEVR-CoGenT and DriVQA datasets. The study investigates di!erences in visual grounding across these conditions. We track both attention magnitude and object localization, and develop methods for matching generated tokens to their relevant scene regions. We find that Thinking models exhibit greater visual attention but less spatial precision than Instruct models, and that this visual attention isn’t evenly allocated— it is heavily concentrated in the reasoning phase. In the context of incorrect responses and hallucinations, we find evidence for a “seeing but not believing” phenomenon, where models attend to the correct region of the image yet still produce incorrect or hallucinated outputs. Specifically, attention magnitude consistently separates correct from incorrect answers, but spatial precision does not. Additionally, specific hallucinated tokens often carry equal or higher attention magnitude than their local context. Together, these findings identify two distinct failure modes— “not looking” versus “seeing but not believing”— and argue that understanding reasoning VLMs requires characterizing both the magnitude and the spatial structure of their visual attention.