BETA RELEASE
Skip to main content

Energy Use of AI Inference: Efficiency Pathways and Test-Time Compute

Primary paper Microsoft Research
Published: 2025-09-01 Accessed: 2026-10-10
Original source ↗ Archived copy ↗

Summary

A paper estimating per-query energy usage of AI inference systems and pathways for improving efficiency.

Key quotes

"median energy per query of 0.34 Wh for frontier-scale models (>200 billion parameters)"
"test-time compute scaling could increase median energy 13-fold to 4.32 Wh per query"

This Microsoft Research paper introduces a bottom-up approach to estimate the energy consumption of AI inference systems, particularly for large language models deployed at scale. The authors argue that many existing estimates overstate energy use by failing to account for efficiency gains in real-world deployments. The study highlights that targeted efficiency interventions at model, platform, and hardware levels could reduce energy consumption by 8–20×, offering significant savings for large-scale AI systems.