When Training Pipeline Issues Look Like Data Deficiency
During training pipeline issues can manifest themselves as data deficiency from the outside. Here are few things you can do
Measuring Actual Data Exposure During Training
During training a data mixture config will say something like code (data domain) is 10 percent of the run. That number is a plan and it is not what the model actually experiences during training.
Monitoring Parallelism in Large Training Runs
When a large training run is slow, the usual dashboard shows GPU utilization, memory used and NVLink traffic per GPU. None of these tells a researcher which parallelism dimension is costing time.
AgentFactory: Joint Model and Workflow Optimization
Continuing my thread on agentic systems optimization I found AgentFactory concept interesting. It is a recent paper to optimize the model and the agent workflow together, rather than treating the underlying model as fixed and only searching over prompts or workflow structure.
SFT and On-Policy RL: Preserving Successful Strategies
Suppose a student knows three valid ways to solve a problem. They can use algebra, draw a diagram, or reason through the numbers directly.
Heterogeneous Parallelism for Multimodal Training
A multimodal model has two main parts: encoders that turn images, video, or audio into embeddings, and the LLM backbone, which is the language model that takes those embeddings together with the text tokens and does most of the computation. Memory profile a. The encoder for images, video, or audio is usually small, but long samples create a lot of activation memory.
MoE Parallel Folding in Megatron and JAX
It was interesting to compare MoE Parallel Folding in Megatron with JAX/TPU architecture. MoE Parallel Folding lets attention layers and MoE layers use different parallelism settings in the same model.
NVIDIA cuObject: Object Storage to GPU Memory
NVIDIA cuObject lets applications read and write S3 compatible object storage directly into GPU memory over RDMA. The concept has strong efficiency promise.
RL Post-Training for Agents: Infrastructure and Weight Updates
RL post-training for agents (2 of 2) 1. Rollouts and training a. Training and Inference infra can be colocated and same GPUs alternate between generating and training, and weights are resharded between the training layout and the inference layout. b. Colocated GPUs wait on the longest trajectory.
RL Post-Training for Agents: From Traces to Rewards
RL post-training for agents (1 of 2) What it takes to post-train a model to operate reliably as an agent. This is a 2 post series.
Avoiding Representation Collapse in JEPA
JEPA models can suffer from representation or embedding collapse. Meaning many different patches of video or images can map to the exactly same embedding space.
Teacher Forcing and Rollout Loss in V-JEPA
One interesting part of V-JEPA 2-AC is how Meta combines teacher forcing with rollout loss. The model learns: visual state + robot state + action → next visual state
Training and Serving NVIDIA Cosmos 3
Continuing on topic of Cosmos 3.0 I have also been comparing its training infra with some other world models but as a baseline this is the training setup. As I posted earlier the model is a 16B (Nano) or 64B (Super) Mixture-of-Transformers with a Reasoner tower and a Generator tower in every layer.
Scaling RL Infrastructure: Lessons from Nemotron
The Nemotron 3 Ultra technical report has a scaling RL infra section which has very useful and practical lessons learnt. The basic setup 1. Prompts are sent to vLLM for rollout generation
Connecting GPU and TPU Metrics to Model Performance
I think there is an opportunity to make low-level GPU and TPU metrics much more useful to model researchers. The profiling systems are already quite good.
Serving with vLLM, TorchTPU and Ray
One of the interesting talks for me at Ray Summit was vLLM on TPU with TorchTPU and Ray. Layering should be obvious
Architectural Efficiency in Gemma 4
You can get a 262K context window, GPQA Diamond of 84.3, AIME 2026 of 89.2, LiveCodeBench of 80.0 and MMMU-Pro of 76.9 from Gemma 4 at just 31B parameters. For comparison, Kimi K3 gets 93.5 on GPQA Diamond and 81.6 on MMMU-Pro, but it is a 2.8T parameter MoE model with about 104B parameters active per token and a 1M context window.
Kubernetes Primitives for AI Workloads
Over the last year, a lot of the Kubernetes AI work can look like Kubernetes itself learned to understand GPUs, distributed training and model state. In my observation, the model is more modular.
Efficient Attention and Multi-Token Decoding
Lot of discussion on GLM 5.3-Flash built on Chinese domestic chips but my observation has been more focused on continuing trends in model architectures. Full attention at every layer and one-token-at-a-time decoding are both becoming too expensive.
Integrated Infrastructure for RL Post-Training
The opportunity with RL-enabled post-training is to build a much more integrated platform across training, inference, data, environments and agents. The post-training loop now includes rollout workers, environments, reward models and verifiers, policy optimization, and continuous evaluation.