VLM2-Bench

A Closer Look at How Well VLMs Implicitly Link Explicit Matching Visual Cues

Jianshu Zhang*, Dongyu Yao*, Renjie Pi, Paul Pu Liang, Yi R. (May) Fung
▶ HKUST ▶ CMU ▶ MIT
*Equal Contribution
arXiv Code HF Dataset
In daily life, we often accomplish tasks purely through Linking Explicit Matching Visual Cues without relying on extensive prior knowledge. For example, we can recognize the same person across different photos without seeing them before, or match an online product to one seen in-store without knowing its brand. But what about increasingly powerful Vision-Language Models—do they possess this vision-centric, knowledge-independent ability?

Abstract

Visually linking matching cues is a crucial ability in daily life, such as identifying the same person in multiple photos based on their cues, even without knowing who they are. Despite the extensive knowledge that vision-language models (VLMs) possess, it remains largely unexplored whether they are capable of performing this fundamental task. To address this, we introduce VLM2-Bench, a benchmark designed to assess whether VLMs can Visually Link Matching cues, with 9 subtasks and over 3,000 test cases. Comprehensive evaluation across eight open-source VLMs and GPT-4o, along with further analysis of various languageside and vision-side prompting methods, leads to a total of eight key findings. We identify critical challenges in models’ ability to link visual cues, highlighting a significant performance gap where even GPT-4o lags 34.80% behind humans. Based on these insights, we advocate for (i) enhancing core visual capabilities to improve adaptability and reduce reliance on prior knowledge, (ii) establishing clearer principles for integrating language-based reasoning in vision-centric tasks to prevent unnecessary biases, and (iii) shifting vision-text training paradigms toward fostering models’ ability to independently structure and infer relationships among visual cues.

VLM2-Bench

VLM2-Bench is designed to evaluate models' ability to visually link matching cues across multiple images and videos, which is structured around three types of visual cue connection: general cue, person-centric cue, and object-centric cue, encompassing a total of nine subtasks.

1. General Cue (GC): Assessing matching (Mat) and tracking (Trk) of visual elements.
2. Object-centric Cue (OC): Evaluating comparison (Cpr), counting (Cnt), and grouping (Grp) of objects.
3. Person-centric Cue (PC): Focusing on comparing (Cpr), counting (Cnt), grouping (Grp), and video identity describing (VID) of individuals.

The dataset comprises over 3,000 question-answer pairs generated via a semi-automated pipeline with human verification, covering various question formats such as True/False, multiple-choice, numerical, and open-ended queries.

VLM2-Bench Overview


Dataset Statistics

VLM2-Bench Statistics


Experiments & Findings

We present a comprehensive evaluation of 8 state-of-the-art vision-language models on VLM2-Bench, including open-source models ranging from 7B to 26B parameters and the commercial model GPT-4o. Our benchmark reveals significant challenges in current models: even the best-performing model, GPT-4o, lags 34.80% behind human-level performance. Moreover, our analysis uncovers distinct performance patterns across different cue types.

VLM2-Bench Evaluation Results

Finding I: Simple tasks for humans pose significant challenges for VLMs. (Table 1)

Error Type Analysis

Finding II: Relatively consistent error patterns in Mat and Trk of GC. (Table 2)

Finding III: Models perform better in linking person-centric cues than object-centric cues. (Table 1)

Various prompting method's impact on performance in VLM2-Bench

Language-Side Prompting Methods
  • CoT-normal: Generic step-by-step reasoning prompt ("Let's think step by step").
  • CoT-special: Task-tailored for GC tasks, enforcing a structured four-step process: understand, perceive, compare, and conclude.
Vision-Side Prompting Methods
  • VP-grid: Overlays a dot matrix with 3D coordinates on images.
  • VP-zoom-o: Crops images to focus on object regions.
  • VP-zoom-p: Isolates facial regions to minimize distractions.
Error Type Analysis
Error Type Analysis

Finding IV: Reasoning in language aids models in logically linking visual cues. (Figure 5 and Figure 15)

Error Type Analysis
Error Type Analysis

Finding V: Effectiveness of visual prompting depends on models' ability to interpret both prompting cues and the visual content. (Figure 5a, Figure 16 and Figure 17)

Error Type Analysis

Finding VI: The open-ended nature of language may hinder object grouping. (Figure 5b and Figure 18)

Finding VII: Amplifying object cues benefits stronger models while having minimal impact on others. (Figure 5b)

Finding VIII: CoT and visual prompting fail to improve linking on highly abstract personcentric cues, leading to a performance drop. (Figure 5c)

Take aways

Improving core visual abilities not only enhances overall performance but also increases adaptability. A stronger visual foundation maximizes the effectiveness of visual prompting and reduces reliance on prior knowledge, enabling models to operate more independently in vision-centric tasks. → Strengthening Fundamental Visual Capabilities.
Integrating language into vision-centric tasks requires careful calibration. Future research should establish clearer principles on when language-based reasoning aids visual understanding and when it introduces unnecessary biases, ensuring models leverage language appropriately. → Balancing Language-Based Reasoning in Vision-Centric Tasks.
Current training paradigms focus heavily on emphasizing vision-language associations. However, as models expand their visual context window, their ability to reason purely within the visual domain becomes increasingly crucial. We should prioritize developing models that can structure, organize, and infer relationships among visual cues. → Evolving Vision-Text Training Paradigms.

BibTeX


@misc{zhang2025vlm2benchcloserlookvlms,
      title={VLM$^2$-Bench: A Closer Look at How Well VLMs Implicitly Link Explicit Matching Visual Cues}, 
      author={Jianshu Zhang and Dongyu Yao and Renjie Pi and Paul Pu Liang and Yi R. Fung},
      year={2025},
      eprint={2502.12084},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2502.12084}, 
}

Contact

Jianshu Zhang: jianshu.zhang777@gmail.com
Yi R. (May) Fung: yrfung@ust.hk

Acknowledgement

This website is adapted from LLaVA-VL, Nerfies, and VL-RewardBench, licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Usage and License

Our VLM2-Bench is available under the CC-BY 4.0 license for academic use with proper attribution. The images, videos, and annotations in this benchmark are intended solely for research purposes. These data were sourced from publicly available online platforms, and while efforts were made to use them responsibly, explicit permissions may not have been obtained for all content. Users are responsible for ensuring that their use of the data complies with applicable intellectual property laws and ethical guidelines. We encourage users to verify the sources and ensure compliance with any terms of service or licensing agreements.