NVIDIA AI Releases Nemotron-Labs-Sigma, a Continuous Diffusion Language Model with 14 checkpoints at 3B/8B

MarkTechPost
Read full story· 5 min
NVIDIA released Nemotron-Labs-Sigma, a continuous diffusion language model family with 14 checkpoints at 3B and 8B parameters, using a smooth latent space for text denoising. Sigma models achieve competitive performance on NVIDIA's 6-task suite, running efficiently on a single NVIDIA GPU.

What happened

NVIDIA has launched Nemotron-Labs-Sigma, a new family of continuous diffusion language models, making 14 open checkpoints available at 3 billion and 8 billion parameter scales. The models use a smooth 16-dimensional latent space for text denoising, differing from previous models that work with discrete tokens. Sigma models are designed to run efficiently on a single NVIDIA GPU and are available for commercial use under the NVIDIA Nemotron Open Model License. The release includes base, instruct, distilled, and speculative variants, all hosted on Hugging Face.

Key details

  • 14 checkpoints released: 3B and 8B parameter sizes, with base, instruct, distilled, and speculative versions.
  • Models denoise text in a 16-dimensional embedding space, not on discrete tokens.
  • Sigma-8B base model averages 71.16 on NVIDIA’s 6-task benchmark suite.
  • All models require a CUDA-capable NVIDIA GPU in bfloat16; no CPU support or quantized variants are provided.
  • The fastest speculative 8B model decodes 3.46 times faster than autoregressive decoding on a GB200 GPU.

Why it matters

Developers and organizations using NVIDIA GPUs can now access a new type of language model that offers efficient text generation in a continuous latent space. The models provide competitive performance on standard benchmarks and can be deployed commercially, but require specific NVIDIA hardware and do not support CPUs. This release gives users more options for high-speed, high-quality language modeling, especially for those with existing NVIDIA infrastructure.


More in LLM & Text Generation