MiTAC Proposes Two-Server AI Inference Architecture at SC26

What happened
MiTAC Computing announced a two-server AI inference architecture at SC26 on November 17, 2026, developed in partnership with Skymizer. The system is designed for on-premises use with very large AI models, such as the 2.8-trillion-parameter Kimi K3. The architecture separates the compute-intensive prefill stage and the memory-intensive decode stage onto different servers, with demonstrations of the cooling and storage systems shown at the event. Some components are live, while others are still in development or proposed.
Key details
- The decode server uses eight Skymizer HTX301-4S accelerator cards, each with 256 GB of memory, totaling 2 TB.
- The prefill server is based on the 8U G8825Z5 platform with AMD EPYC 9575F or 9755 CPUs and Instinct MI350X GPUs.
- A 52U liquid-cooled rack demonstration integrates 12 G4826Z5 servers and 96 AMD Instinct MI355X GPUs.
- The storage system combines DDN Infinia with seven GC68C-B8056 1U servers for continuous data delivery.
- Canonical Ubuntu, Rafay, and MiTAC’s MiCoreView software provide the operating system, orchestration, and monitoring.
Why it matters
Enterprises running large AI models on their own hardware can now provision compute and memory resources separately for different stages of inference, potentially reducing bottlenecks and improving efficiency. Data center operators must consider not just GPU count but also memory capacity, network speed, cooling, and storage throughput to deploy these systems effectively. The new architecture gives organizations more control over how they scale and manage AI workloads on-premises, with specific hardware and software options to address performance and energy demands.
