Inference now drives two-thirds of AI workloads, changing where chips get deployed

SiliconANGLE
Read full story· 4 min
Inference workloads now constitute about two-thirds of AI processing, shifting chip deployment from large training clusters to smaller, modular data centers optimized for power and cooling. Positron AI, valued at $5 billion, designs inference servers using commodity memory to meet this demand, with deployments underway at Oracle and other providers.

What happened

Inference workloads now make up about two-thirds of all AI processing, a shift from the previous dominance of training workloads. This change is leading to more AI chips being deployed in smaller, modular data centers rather than in large training clusters. Positron AI, a company valued at $5 billion, is producing inference servers that use standard memory instead of high-bandwidth memory. Oracle and other providers have started deploying Positron's servers to support this new demand.

Key details

  • Inference workloads have risen from 15% in 2023 to about 66% of AI processing now.
  • Oracle is deploying around 340 Positron Atlas servers in its cloud this quarter.
  • Positron's servers use commodity memory and draw about 4,500 watts each.
  • A full rack of Positron Titan servers fits within a 40-kilowatt power envelope.

Why it matters

Cloud providers and data center operators now need to optimize for inference rather than training, affecting how and where they invest in infrastructure. Companies deploying AI can use smaller, quickly built modular data centers, which require less power and cooling, to run inference workloads. This reduces deployment time and allows providers to monetize new AI capacity faster.


More in Chips & Compute