I wrote recently about Gemma 4 in the context of embeddings. I then looked more closely at its vision architecture in Gemma 4 12B, and that led me to recently released NeoMME.
Both are interesting because they question a very common multimodal design:
Image → vision encoder (ViT/SigLIP) → adapter → language model
In this setup, a separate vision Transformer first converts image patches into rich visual representations. The language model sees those representations later.
Gemma 4 12B takes a different approach:
Image → patches → adapter (linear projection) → shared Gemma Transformer
The image is split into patches. Each patch is basically a long list of RGB pixel values.
The adapter here is a single learned linear layer, with normalization on either side. It takes those pixel values and converts them into a fixed-size vector that the Gemma Transformer knows how to process. It is not doing the visual understanding.
The main Transformer now has to learn much more of that work itself e.g. shapes, objects, spatial relationships and their connection to language.
There is an important attention detail 🙂 . For image patches, Gemma allows bidirectional attention. A patch on the left side of an object can look at a patch on the right side, regardless of their order in the sequence.
But Gemma also generates text. Once it starts generating an answer, text remains causal.
NeoMME
NeoMME takes the same basic idea in a different direction:
Image → 32×32 patches → adapter (small MLP) → shared Transformer
Text goes into the same Transformer through factorized token embedding, a compact lookup table projected up to the model's hidden size, not a standard full-size embedding.
There is again no separate vision encoder.
The adapter here is a small MLP, a slightly richer version of the same idea. It converts raw pixel values into the model's hidden representation. Unlike Gemma's linear adapter, this one has a hidden layer and a nonlinearity, so it does a bit more shaping before the shared Transformer takes over.
But NeoMME is not trying to generate text. Its main job is to produce strong multimodal representations for things like retrieval and embeddings.
Because the entire input is already known, all attention can be bidirectional.
Meaning, every image patch and text token can use information from both directions.
This makes NeoMME conceptually closer to BERT, while Gemma is closer to a multimodal GPT.
The common idea is more interesting than the individual implementations
Move modality-specific intelligence out of a large separate vision encoder and into the shared Transformer.
The inputs become relatively simple:
pixels → lightweight adapter
text → token embedding
I think its an interesting architectural direction to watch.