Avoiding Representation Collapse in JEPA
JEPA models can suffer from representation or embedding collapse. Meaning many different patches of video or images can map to the exactly same embedding space.
Teacher Forcing and Rollout Loss in V-JEPA
One interesting part of V-JEPA 2-AC is how Meta combines teacher forcing with rollout loss. The model learns: visual state + robot state + action → next visual state
JEPA and Generative World Models
Continuing with series of posts on world models - Both Cosmos 3 and PhiZero are generative world models predict compressed video latents that can be decoded into frames. Those latents need to preserve visual detail for reconstruction.
PhiZero’s Physical Language for World Models
Continuing the series of posts on world models - PhiZero from CASIA is a research model but it introduces a novel idea of physical language. The problem it starts from is how world models represent change.
Training and Serving NVIDIA Cosmos 3
Continuing on topic of Cosmos 3.0 I have also been comparing its training infra with some other world models but as a baseline this is the training setup. As I posted earlier the model is a 16B (Nano) or 64B (Super) Mixture-of-Transformers with a Reasoner tower and a Generator tower in every layer.
Data Curation for NVIDIA Cosmos 3
The level of data strategy details in the NVIDIA Cosmos 3 technical report is amazing. Here is a quick summary of process and quantity
Inside NVIDIA Cosmos 3’s Two-Tower Architecture
NVIDIA Cosmos 3 is combination of autoregressive reasoning and continuous world generation inside one Mixture-of-Transformers architecture. Every transformer layer holds two full sets of weights.
How Agent Gateways Manage Tools and Access
I have been looking at how agent gateways are evolving, and recent work from Solo, Gravitee and Envoy shows where they are going. The first generation looked like an API gateway with new protocols: Agent → Gateway → Models / MCP servers / Other agents
Scaling RL Infrastructure: Lessons from Nemotron
The Nemotron 3 Ultra technical report has a scaling RL infra section which has very useful and practical lessons learnt. The basic setup 1. Prompts are sent to vLLM for rollout generation
Shared Transformers for Vision and Language
I wrote recently about Gemma 4 in the context of embeddings. I then looked more closely at its vision architecture in Gemma 4 12B, and that led me to recently released NeoMME.
DataFusion Query Optimization Through Less Work
I’ve been down with the flu, so keeping this one very short. DataFusion 55 was released in late August, and it is a good example of how a query engine gets faster by doing less work.
OmniTable and Large-Scale LLM Data Curation
I read the OmniTable paper from Ant Group, which won the Best Industry Paper award at VLDB this year. The core architecture is built around a simple idea that I think is useful for large-scale LLM data curation: Logical Unification, Physical Separation.
Connecting GPU and TPU Metrics to Model Performance
I think there is an opportunity to make low-level GPU and TPU metrics much more useful to model researchers. The profiling systems are already quite good.
Serving with vLLM, TorchTPU and Ray
One of the interesting talks for me at Ray Summit was vLLM on TPU with TorchTPU and Ray. Layering should be obvious
Architectural Efficiency in Gemma 4
You can get a 262K context window, GPQA Diamond of 84.3, AIME 2026 of 89.2, LiveCodeBench of 80.0 and MMMU-Pro of 76.9 from Gemma 4 at just 31B parameters. For comparison, Kimi K3 gets 93.5 on GPQA Diamond and 81.6 on MMMU-Pro, but it is a 2.8T parameter MoE model with about 104B parameters active per token and a 1M context window.
Converging Patterns in Language Model Architecture
Reading the Qwen3.8-Flash-Next architecture blog felt a bit like déjà vu. Many of the ideas look familiar from Kimi K3, DeepSeek V4 and GLM-5.3-Flash, even if the implementations are different.
Kubernetes Primitives for AI Workloads
Over the last year, a lot of the Kubernetes AI work can look like Kubernetes itself learned to understand GPUs, distributed training and model state. In my observation, the model is more modular.
Connecting Lakehouses and Real-Time Analytics
I see a clear trend across open-source analytical stacks. They are trying to close the gap between the lakehouse and real-time analytical systems.
Storage and Query Architecture for Agent Traces
Over the next few years, I think agent traces are going to become one of the fastest-growing data types in the AI stack. As more applications become agentic, every interaction can generate a tree of model calls, tool invocations, retrievals, sandbox executions, verifier results and intermediate state.
Efficient Attention and Multi-Token Decoding
Lot of discussion on GLM 5.3-Flash built on Chinese domestic chips but my observation has been more focused on continuing trends in model architectures. Full attention at every layer and one-token-at-a-time decoding are both becoming too expensive.