The most efficient training systems are designed around the specific model architecture, where parallelism strategy, kernels, networking, precision and memory hierarchy are all part of the model performance story. This is a move away from generic training infrastructure patterns.
Kimi K3 and DeepSeek V4 Pro are good examples. Kimi K3 scales a very large MoE model by focusing heavily on expert parallelism, predictable communication patterns, optimized kernels and keeping communication off the critical path. At this scale, expert placement, routing balance, memory movement become first-class parts of the training system.
DeepSeek V4 Pro takes a similar systems-first approach, but pushes harder on model and infrastructure co-design across MoE execution, long-context attention, low precision and fused kernels that overlap compute, communication and memory access.