Inference now drives two-thirds of AI workloads, changing where chips get deployed

What happened
Inference workloads now make up about two-thirds of all AI processing, a shift from the previous dominance of training workloads. This change is leading to more AI chips being deployed in smaller, modular data centers rather than in large training clusters. Positron AI, a company valued at $5 billion, is producing inference servers that use standard memory instead of high-bandwidth memory. Oracle and other providers have started deploying Positron's servers to support this new demand.
Key details
- Inference workloads have risen from 15% in 2023 to about 66% of AI processing now.
- Oracle is deploying around 340 Positron Atlas servers in its cloud this quarter.
- Positron's servers use commodity memory and draw about 4,500 watts each.
- A full rack of Positron Titan servers fits within a 40-kilowatt power envelope.
Why it matters
Cloud providers and data center operators now need to optimize for inference rather than training, affecting how and where they invest in infrastructure. Companies deploying AI can use smaller, quickly built modular data centers, which require less power and cooling, to run inference workloads. This reduces deployment time and allows providers to monetize new AI capacity faster.