MiTAC Proposes Two-Server AI Inference Architecture at SC26

Unite.AI
Read full story· 4 min
MiTAC Computing introduced a two-server AI inference architecture featuring 2 TB of accelerator memory for on-premises use with large models like the 2.8-trillion-parameter Kimi K3. The design separates compute-intensive prefill and memory-intensive decode stages, demonstrated with liquid cooling and high-density GPU racks at SC26. The system aims to optimize AI inference workloads by balancing accelerator memory and network performance.

What happened

MiTAC Computing announced a two-server AI inference architecture at SC26 on November 17, 2026, developed in partnership with Skymizer. The system is designed for on-premises use with very large AI models, such as the 2.8-trillion-parameter Kimi K3. The architecture separates the compute-intensive prefill stage and the memory-intensive decode stage onto different servers, with demonstrations of the cooling and storage systems shown at the event. Some components are live, while others are still in development or proposed.

Key details

  • The decode server uses eight Skymizer HTX301-4S accelerator cards, each with 256 GB of memory, totaling 2 TB.
  • The prefill server is based on the 8U G8825Z5 platform with AMD EPYC 9575F or 9755 CPUs and Instinct MI350X GPUs.
  • A 52U liquid-cooled rack demonstration integrates 12 G4826Z5 servers and 96 AMD Instinct MI355X GPUs.
  • The storage system combines DDN Infinia with seven GC68C-B8056 1U servers for continuous data delivery.
  • Canonical Ubuntu, Rafay, and MiTAC’s MiCoreView software provide the operating system, orchestration, and monitoring.

Why it matters

Enterprises running large AI models on their own hardware can now provision compute and memory resources separately for different stages of inference, potentially reducing bottlenecks and improving efficiency. Data center operators must consider not just GPU count but also memory capacity, network speed, cooling, and storage throughput to deploy these systems effectively. The new architecture gives organizations more control over how they scale and manage AI workloads on-premises, with specific hardware and software options to address performance and energy demands.


More in Industry & Manufacturing