Controlling Inference-Time Compute for SLO-Aware Early Exit in Reasoning Model Serving

Abstract

The emergence of reasoning capabilities in large language models has transformed their use in real world production environments. Reasoning models achieve higher accuracy on logical tasks at the cost of generating long chain-of-thought sequences that can span tens of thousands of tokens. At the same time, however, applications integrating language models are governed by strict latency deadlines. This creates a fundamental tension between serving reasoning models and the long response times associated with them. In this thesis, we resolve this tension by creating a serving system that can dynamically adjust model reasoning to meet latency deadlines. Our core insight is that reasoning depth can be a controllable resource: requests close to their deadlines can use less reasoning effort to meet their SLOs, while requests with loose deadlines can use full reasoning chains. We introduce ACER (Adaptive Control for Efficient Reasoning), a serving system that adapts reasoning depth as a function of each request’s remaining latency budget. To find the right control mechanism for reasoning, we first benchmark efficient reasoning strategies, characterizing their accuracy–latency trade-offs. Leveraging these insights, ACER tunes thresholds for early exit to maximize accuracy given a latency constraint. Our evaluation across four model–dataset configurations show that ACER outperforms fixed-threshold baselines, reaching peaks of +10.5pp accuracy at tight SLOs and reducing tail latency by as much as 38% at matched accuracy. Finally, in a two-tier SLO serving experiment, ACER improves goodput by 21% and reduces SLO violations from 28% to under 1%.