ZAYA1-8B: How Zyphra's Tiny MoE Model Achieves Giant Performance on AMD Hardware
Introduction: A Small Model That Defies Expectations
Zyphra AI has unveiled ZAYA1-8B, a compact language model that challenges the notion that bigger is always better. With only 760 million active parameters out of a total of 8.4 billion, this Mixture of Experts (MoE) design delivers performance that rivals frontier reasoning systems many times its size. Trained entirely on AMD hardware, ZAYA1-8B is now available under the Apache 2.0 license on Hugging Face and as a serverless endpoint on Zyphra Cloud.
What Is a Mixture of Experts Model and Why Active Parameters Matter
The key to ZAYA1-8B's efficiency lies in its MoE architecture. Unlike standard dense models where every parameter activates for each input, an MoE model selectively engages only a subset of specialized 'experts' per forward pass. Here, ZAYA1-8B's 8.4 billion total parameters are distributed across multiple experts, but just 760 million are active at any one time. This approach dramatically reduces inference compute and memory bandwidth while preserving the representational capacity of a far larger model.
This design makes ZAYA1-8B ideal for on-device deployment, efficient test-time compute scenarios, and low-latency serving—all while matching or exceeding the benchmark scores of dense models that are orders of magnitude bigger.
Benchmark Results: Punching Above Its Weight
Despite its modest active parameter count, ZAYA1-8B achieves scores competitive with first-generation frontier reasoning models like DeepSeek-R1-0528, Gemini-2.5-Pro, and Claude 4.5 Sonnet on challenging mathematical reasoning tasks. Using a novel test-time compute methodology called Markovian RSA, it surpasses Claude 4.5 Sonnet and GPT-5-High on the HMMT'25 benchmark (89.6 vs. 88.3) and closes in on frontier open-weight models like DeepSeek-V3.2 on mathematics benchmarks.
These results underscore a central theme: ZYPHRA's focus on maximizing intelligence per parameter and per FLOP yields outsized returns in math and coding domains.
Architecture: MoE++ and Three Key Innovations
ZAYA1-8B is built on Zyphra's MoE++ architecture, which introduces three specific improvements over standard MoE designs. Together, they form the foundation of the model's intelligence efficiency.
Compressed Convolutional Attention (CCA)
Zyphra developed Compressed Convolutional Attention, a sequence mixing mechanism that operates in a compressed latent space. It achieves 8× KV-cache compression versus standard attention. The KV-cache stores intermediate attention states during inference—an 8× reduction directly lowers memory requirements and enables longer effective contexts within the same hardware envelope.
ZAYA1 MLP-based Router with PID-Controller Bias Balancing
Standard MoE routers typically use linear projections to assign tokens to experts. Zyphra replaces this with an MLP-based router enhanced by PID-controller-style bias balancing. This improves routing stability and actively prevents load imbalance across experts—a common failure mode in MoE training.
Learned Residual Scaling
ZAYA1-8B employs learned residual scaling to control residual-norm growth through deep layers, ensuring stable training and better convergence. This technique helps maintain representational power without exploding gradients.
Training on AMD Hardware
An important differentiator is that ZAYA1-8B was trained end-to-end on AMD hardware. This demonstrates the growing maturity of AMD's ecosystem for deep learning and offers a viable alternative to NVIDIA-centric pipelines. The model's success on this platform opens doors for more diverse hardware choices in the AI industry.
Availability and Deployment
ZAYA1-8B is released under the permissive Apache 2.0 license, making it free for commercial and research use. You can download it from Hugging Face or access it via a serverless endpoint on Zyphra Cloud. For developers, the model's small active parameter count means it can be run on consumer-grade hardware or edge devices, enabling private, low-latency AI applications.
To explore further details, visit the official Zyphra announcement.
Key Advantages at a Glance
- Active parameters: 760M → efficient inference
- Total parameters: 8.4B → high capacity
- Benchmark wins: surpasses larger models on math tasks
- Open license: Apache 2.0
- Novel techniques: CCA, MLP router, residual scaling
ZAYA1-8B represents a step forward in efficient AI, proving that smart architecture and targeted training can deliver frontier-level performance from a fraction of the parameters.