Tencent Unveils OpenSearch-VL: Open-Source Multimodal Deep Search Agent

As multimodal large language models (MLLMs) advance rapidly, the key challenge in AI competition is shifting from simply "passively understanding images" to actively seeking evidence and reasoning. Yet the lack of high-quality training data, automated trajectory synthesis pipelines, and detailed training recipes has made it difficult for the open-source community to replicate top-tier multimodal search agents.

To overcome this bottleneck, a research team from Tencent Hunyuan, together with UCLA and The Chinese University of Hong Kong, has launched OpenSearch-VL. This fully open-source blueprint provides a path to building a state-of-the-art deep search agent through reinforcement learning (RL).

Innovative Data Pipeline Overcomes Search Shortcuts

The team identified high-quality training data as the primary bottleneck for model advancement. To train models for multi-step reasoning—beyond simple one-click image recognition—they designed a meticulous data consolidation pipeline.

This pipeline samples paths from Wikipedia's hyperlink graph, converting intricate entity relationships into multi-hop QA pairs. To prevent shortcut learning, researchers applied fuzzy entity rewriting to conceal direct answers and integrated visual localization anchored to source code. This approach compels the model to first detect visual signals, then iteratively retrieve information using external tools, preventing skill degradation during retrieval. As a result, the team constructed the SearchVL-SFT database (36,000 instruction fine-tuning trajectories) and the SearchVL-RL database (8,000 reinforcement learning trajectories).

Versatile Toolset Beyond Simple Search

OpenSearch-VL goes beyond basic text search. In real-world use, user-provided images are often blurry, distorted, or low-resolution, rendering conventional search tools ineffective.

To tackle this, the project incorporates a rich set of tools: web search, reverse image search, OCR, image cropping, sharpening, super-resolution, and perspective correction. The agent actively perceives and corrects imperfect visuals—much like a human—before querying external knowledge, ensuring reliable subsequent searches.

Fault-Aware Algorithm Enables Learning from Mistakes

In long-horizon tasks, tool calls can create cascading failures. A single timeout or error may derail the entire task. Traditional RL often discards these failed trajectories, wasting training resources.

OpenSearch-VL introduces Multi-round Fault-Aware GRPO, a training algorithm that identifies critical failure points in tool calls. It uses masking to filter out invalid post-failure information and applies one-sided advantage clipping to preserve useful logic from earlier steps. This way, even when the final outcome is failure, the model still learns effective search paths and exploration strategies from the preceding stages.

Experimental Results Rival Commercial Proprietary Models

Evaluation results demonstrate strong performance across seven mainstream multimodal deep search benchmarks, with an average improvement exceeding 10 percentage points. On certain specific tasks, OpenSearch-VL already matches the top closed-source commercial models.

The team plans to fully open-source all training data, code, and model weights for OpenSearch-VL, offering global developers a reproducible and improvable foundation that advances multimodal agent research into deeper territory.

Paper: https://arxiv.org/pdf/2605.05185