You can get a 262K context window, GPQA Diamond of 84.3, AIME 2026 of 89.2, LiveCodeBench of 80.0 and MMMU-Pro of 76.9 from Gemma 4 at just 31B parameters.
For comparison, Kimi K3 gets 93.5 on GPQA Diamond and 81.6 on MMMU-Pro, but it is a 2.8T parameter MoE model with about 104B parameters active per token and a 1M context window.
Kimi K3 is like a large full-frame camera system with a lot of sensor area and glass. Gemma 4 is more like a much smaller camera that gets surprisingly close in image quality because the optics, sensor processing and software are very well optimized.
A lot of that efficiency comes from a few architectural choices.
1. Local and global attention. Most layers use cheaper local attention, with periodic global attention layers for access to the full context. This reduces the cost of maintaining long context without giving up global retrieval.
2. KV-cache optimization. Global attention layers reuse keys as values in parts of the family and use partial RoPE, reducing the memory needed for the global KV cache. Smaller models also share KV representations across layers.
3. Sparse capacity where it helps. Gemma 4 includes a 26B MoE model that activates only about 3.8B parameters during inference, while the E2B and E4B models use per-layer embeddings to get more stored capacity without increasing effective compute proportionally.
4. Inference optimization. Gemma 4 also supports multi-token prediction drafters, continuing the move toward designing model architecture and serving architecture together.
Kimi K3 is clearly the more capable model overall and operates at a very different scale. Gemma 4 shows how far a much smaller model can be pushed with careful architectural choices.