Energy use of AI inference, efficiency pathways, and test-time scaling
Summary
Study estimates frontier-scale AI inference energy use at 0.31 Wh/query, contradicting public estimates by 4–20×; reasoning queries increase energy use by 13×.
Key quotes
Frontier-scale inference uses 0.31 Wh/query, 4–20× below public estimates.
Reasoning queries (∼5,000 output tokens) raise energy use ∼13× vs. standard queries.
Model, serving, and hardware gains could reduce per-query energy by 8–20×.
Even a modest share of long reasoning queries (e.g., 10% of daily requests) can more than double total energy consumption.
This paper analyzes the energy consumption of AI inference, particularly large language models (LLMs), under large-scale deployment conditions. It introduces a bottom-up framework to estimate energy use per query, highlighting discrepancies between optimized production conditions and non-production estimates. The findings emphasize the necessity of efficiency improvements to manage escalating energy demand as AI inference scales.