Multi-Modal Large Language Models (MLLMs) recently demonstrated strong capabilities in both instruction comprehension and generation, positioning them as promising tools for human-computer interaction. However, the inherent ambiguity of language poses a challenge, potentially leading models astray in task implementation due to differing interpretations of the same text within varying contexts. In multi-modal settings, visual information serves as a natural aid in disambiguating such scenarios. In this paper, we introduce the first benchmark specifically designed to evaluate the performance of MLLMs in Ambiguous contexts (MMA). This benchmark employs a multiple-choice visual question-answering format and includes 261 questions with ambiguous meaning. Each question is linked to a pair of images that suggest divergent scenarios, thus leading to different answers given the same question. These questions are stratified into three categories of ambiguity: lexical, syntactic, and semantic, to facilitate a detailed examination of MLLM performance across varying levels of ambiguity. By evaluating 16 proprietary and open-sourced MLLMs, we find that: (1) When presented with two different contextual images and asked the same question, MLLMs achieved an accuracy rate of only 50.59% in answering both correctly, compared to human performance at 88.97%. This is because MLLMs often overlook scenario-specific information provided by images. (2) Among the three types of ambiguity, models perform best under lexical ambiguity and worst under syntactic ambiguity. (3) Open-sourced models generally perform significantly lower than limited-access MLLMs, with an average performance gap of 10.66%. GPT-4o emerges as the top model, achieving 70.00\% accuracy. These results indicate that current MLLMs struggle with multi-modal ambiguity.

In order to systematically explore the capability of MLLMs to perceive and resolve ambiguities of varying complexities, we categorize ambiguities into lexical, syntactic, and semantic types based on the linguistic characteristics. The benchmark tasks are structured as multiple-choice VQA scenarios, a format that simplifies the evaluation process, where the meaning of each question is ambiguous, and they are associated with multiple images that provide varying contexts, allowing the same question to elicit different correct responses based on the visual information provided. This design forces the MLLMs to adeptly integrate and interpret both textual and visual data to select the most accurate answer, reflecting the true potential and challenges of deploying such models in diverse, ambiguity-filled environments.

The illustration of benchmark samples, where each sample consists of pairs of images, each associated with the same question. The model needs to answer the question based on the visual information presented in each image.

Ambiguity Accuracy (Amb_A) This metric is calculated as the percentage of questions where the model correctly answer for both paired images. A high Amb_A indicates that the model does not simply latch onto one possible interpretation of the ambiguity. Instead, it effectively integrates visual information from images to arrive at the most appropriate answer for each scenario.

Lexical ambiguities. Lexical ambiguity mainly evaluates the ambiguity caused by polysemy in sentences. We considered the ambiguity caused by nouns, adjectives, and verbs. The verb category includes both the ambiguity of polysemy and the ambiguity of different emotions it may evoke.
Syntactic ambiguities. Syntactic ambiguities occur when sentence structures allow for multiple interpretations. There are three main types: (a) Attachment Ambiguity: This occurs when a modifying phrase, usually a prepositional phrase or clause, can logically attach to more than one part of the sentence. (b) Coordination Ambiguity: This happens when adjectives, adverbs, or other modifiers can ambiguously apply to one or more nouns in a series, creating uncertainty about whether the modifiers apply to all or just some elements. (c) Structural Ambiguity: This arises when verbs can be used in both transitive and intransitive forms, leading to different meanings.
Semantic ambiguities. Semantic ambiguities involve the broader meanings of text and their interaction with visual elements : (a) Idiomatic Ambiguity: This occurs with idiomatic expressions that can be interpreted both literally and metaphorically. (b) Pragmatic Ambiguity: This arises from interpreting a sentence in different contexts provided by visual cues, affecting how the listener or viewer understands the relevance and expected response. (c) Rhetorical Ambiguity: This involves the use of rhetorical devices like irony, sarcasm, or hyperbole, which can lead to multiple interpretations depending on the visual context.

We make a table to compare our benchmark with existing benchmarks.