Google DeepMind's TIPSv2: AI That Truly Understands Images, Not Just Glances

AI image understanding currently has a core limitation.

When asked "What is in this picture?" it can provide a detailed response. However, asking "Where is the panda's left hind leg?" leads to vague answers. This is not a flaw in any specific model but a persistent issue across the entire visual-language large model domain: strong global understanding but weak local localization.

Google DeepMind introduced TIPSv2 in their latest paper, specifically designed to address this challenging problem.

The research team observed a counterintuitive finding: in fine-grained segmentation tasks, smaller student models frequently outperform larger teacher models. This happens because distillation removes the masking mechanism, compelling the model to learn every detail of the entire image, creating a form of "full-area supervision." Motivated by this insight, TIPSv2 introduced three key enhancements.

First, iBOT++. Traditional pre-training only computes loss for masked regions, leaving visible areas in a neglected state that causes local semantics to drift. iBOT++ requires the model to provide precise supervision over all visible areas, effectively upgrading the task from a "puzzle game" to "carefully reading the entire text." This single improvement boosted zero-shot segmentation performance by 14.1 percentage points.

Second, Head-only EMA. Traditional self-supervised training requires keeping two nearly identical large models in memory, which is highly resource-intensive. TIPSv2 discovered that the image-text contrastive loss alone is enough to stabilize the backbone network, so EMA only needs to be applied to the final projection head, eliminating the need to duplicate the backbone. This reduces the training parameter count by about 42%, making it faster with almost no performance drop.

Third, multi-granularity text pairing. During training, short web descriptions, medium-detail descriptions, and long descriptions generated by Gemini are randomly mixed and fed into the model, alternating between easy and hard tasks. This prevents the model from coasting on simple tasks while ensuring no details are overlooked.

The final results are compelling. TIPSv2 underwent frozen evaluation across nine tasks and 20 authoritative datasets. Zero-shot semantic segmentation achieved a new industry benchmark, while image-text retrieval and classification outperformed comparison models with 56% more parameters. Pure visual tasks also placed among the top performers.

The code and model weights for TIPSv2 are fully open-sourced. For teams working in medical imaging, autonomous driving, industrial inspection, and other domains that demand high-precision image understanding, this solution is well worth a close look.

Paper: https://www.alphaxiv.org/abs/2604.12012