CounterVQA: Evaluating and Improving Counterfactual Reasoning in Vision-Language Models for Video Understanding

Let $\mathcal{A}=\{a_{1},a_{2},\ldots,a_{n}\}$ denote a sequence of actions observed in the video, where $a_{i}$ causally precedes $a_{i+1}$. An adjacent counterfactual question intervenes on action $a_{i}$ with an alternative $a^{\prime}_{i}$ and asks about the resulting change to the immediate next action:

$$\text{Query: }a_{i}\rightarrow a^{\prime}_{i}\implies a_{i+1}\rightarrow\,?$$

(1)

The model must predict the modified outcome $a^{\prime}_{i+1}$ under the counterfactual condition $a_{i}\rightarrow a^{\prime}_{i}$. This requires both identifying the causal nature of the relationship $a_{i}\rightarrow a_{i+1}$ distinguishing it from mere temporal adjacency and reasoning about how the intervention propagates through this causal link while holding other factors constant.

In Figure 2 coffee-making video, We assume the original action $a_{i}$ is ”pour milk into the cup” leading to $a_{i+1}$ = ”add hot water into the coffee mug”, the counterfactual question asks: ”If pouring milk had not happened ($\neg a_{i}$), would the agent still add hot water into the coffee mug?” The model must recognize that these are adjacent but independent events. That is pouring milk does not causally affect adding hot water.

Given an action sequence $\mathcal{A}=\{a_{1},\ldots,a_{n}\}$ and a causal chain $a_{i}\rightarrow a_{i+1}\rightarrow\cdots\rightarrow a_{i+k}$ where $k\geq 2$, a long-chain counterfactual question intervenes on $a_{i}$ and queries about the effect on a non-adjacent action $a_{i+k}$:

$$\text{Query: }a_{i}\rightarrow a^{\prime}_{i}\implies a_{i+k}\rightarrow\,?$$

(2)

The model must reason through the intermediate causal links $a^{\prime}_{i}\rightarrow a^{\prime}_{i+1}\rightarrow\cdots\rightarrow a^{\prime}_{i+k}$ to predict the final outcome. This requires tracking how changes propagate through the causal graph over multiple hops.

In Figure 2 cooking preparation video, we assume $a_{i}$ = ”open drawer”, which leads through $a_{i+1}$ = ”take bowl out” to $a_{i+2}$ = ”put the cucumber into the bowl”, the counterfactual asks: ”If opening drawer had not happened ($\neg a_{i}$), would the agent have put the cucumber into the bowl ($a_{i+2}$)?” The model must trace the multi-step causal chain: opening drawer $\rightarrow$ taking bowl out $\rightarrow$ putting cucumber. Without the initial action, the entire chain collapses.

Let $\mathcal{A}_{\text{true}}=\{a_{1},\ldots,a_{n}\}$ denote the ground-truth action sequence and $\mathcal{A}_{\text{distractor}}=\{\tilde{a}_{1},\ldots,\tilde{a}_{m}\}$ denote a set of plausible but non-existent actions that are semantically coherent with the video context. A Level 3 question presents a counterfactual scenario involving $\tilde{a}_{j}\in\mathcal{A}_{\text{distractor}}$:

$$\text{Query: }\tilde{a}_{j}\rightarrow\tilde{a}^{\prime}_{j}\implies a_{k}\rightarrow\,?$$

(3)

The correct response is to recognize that $\tilde{a}_{j}$ never occurred, thus rendering the counterfactual premise invalid. This tests the model’s ability to distinguish $\mathcal{A}_{\text{true}}$ from $\mathcal{A}_{\text{distractor}}$ under challenging conditions where semantic plausibility conflicts with factual accuracy.

In the Figure 2 cooking egg video showing only an unopened oven with an egg inside, the question asks: ”If opening oven had not happened ($\neg\tilde{a}_{j}$, where $\tilde{a}_{j}$ is a non-existent event), would the agent finish cooking egg?” The model must recognize that the oven never actually opened in the video. Since the oven remained closed throughout, no cooking occurred.

For each video, we employ a multi-agent system to construct a causal graph G = (V, E), where nodes V represent key events and edges E capture causal dependencies. Specifically, we use 4 specialized agents. Observer Agent: We first generate a set of candidate causal relations and let the LLM assign each relation a confidence score, representing the model’s self-estimated likelihood that the relation is correct. We then rank all candidate relations by these confidence scores and retain only the top 20% for further evaluation by the Verifier Agent and the Critique Agent. Verifier Agent: Evaluate a single proposed causal link between two actions based on Commonsense Knowledge. Critic Agent: The goal is to find flaws in the Verifier’s causal analysis. Synthesizer Agent: Make a final decision for this causal relational decision based on confidence scores from Verifier and Critic Agents. After that, all accepted causal relations are aggregated into a causal graph for each video. Human evaluation indicates that roughly 90% of the extracted relations align with human judgments of causal correctness, as detailed in Appendix D.

Not all videos are equally suitable for counterfactual reasoning. To ensure our benchmark contains videos with rich causal structures across all three difficulty levels, we design three graph-based metrics that directly correspond to the capabilities tested at each level. These metrics quantify different aspects of causal complexity and enable systematic filtering of suitable videos. Adjacent Non-Causal Density (ANCD): To assess whether a video contains sufficient temporal correlations that must be distinguished from true causality, we compute the ratio of adjacent non-causal relationships to all adjacent relationships:

$$\text{ANCD}(G)=\frac{\#\text{ Adjacent Non-causal Relations}}{\#\text{ Adjacent Relations}}$$

(4)

A higher value indicates more challenging scenarios for Level 1, as models must carefully differentiate spurious temporal adjacency from genuine causal links. We retain videos with ANCD $\geq 0.2$ to ensure sufficient difficulty in identifying true causal dependencies. Causal Depth (CD): To evaluate a video’s suitability for long-chain counterfactual reasoning, we measure the maximum path length in the causal graph:

$$\text{Causal-Depth}(G)=\max_{p\in P(G)}\text{length}(p)$$

(5)

where $P(G)$ denotes the set of all simple paths in the causal graph $G$, i.e., paths that do not revisit any node. Videos with greater causal depth contain longer causal chains, enabling multi-hop reasoning questions. We select videos with Causal-Depth $\geq 3$ to ensure adequate complexity for Level 2 questions. Contextual CNDA Score: Level 3 questions require plausible but non-occurring actions as distractors. We compute the Contextual CNDA score to quantify semantic diversity within each video: videos with diverse actions provide richer semantic contexts where additional plausible alternatives can be naturally conceived. For each action node $v$ in the graph, we first calculate its outlier score based on semantic deviation from its task context $\tau(v)$:

$$\mathbf{c}_{\tau(v)}=\frac{1}{|C_{\tau(v)}|}\sum_{u\in C_{\tau(v)}}e(u)$$

(6)

where $C_{\tau(v)}$ is the set of actions sharing the same task context, and $e(u)$ is the embedding of action $u$. The outlier score measures how semantically atypical an action is:

$$\text{OutlierScore}(v)=1-\frac{e(v)\cdot\mathbf{c}_{\tau(v)}}{\|e(v)\|\cdot\|\mathbf{c}_{\tau(v)}\|}$$

(7)

We then average across all nodes to obtain the video-level CNDA score:

$$\text{Avg-CNDA-Score}(G)=\frac{1}{|V|}\sum_{v\in V}\text{OutlierScore}(v)$$

(8)

A higher CNDA score indicates that the video contains diverse actions, making it easier to construct plausible hallucinated alternatives. We retain videos with avg-CNDA-score $\geq 0.12$ to ensure sufficient semantic diversity for Level 3 distractor generation. These metrics filtering reduces the candidate set from 1,200 videos to 700+, ensuring that retained videos exhibit rich causal structures necessary for challenging counterfactual reasoning evaluation.

For each filtered video, we use Deepseek-V3 to generate counterfactual questions based on the causal graph. The generation prompt instructs the LLM to identify a pivotal node in the graph, hypothesize an alternative outcome at that node, and reason about downstream effects. To ensure question quality, we implement a paired verification process. Two independent text-only LLMs (Qwen 2.5-32B) attempt to answer each generated question using only the video captions. We only keep the challenging questions which can be answered correctly by LLMs with video annotations. This verification step removes approximately 80% of generated questions, resulting in a final set of 3,000+ high-quality QA pairs. To ensure the reliability of causal graph and questions, we are conducting a validation on the dataset, including inter-annotator agreement and relation-type statistics. The detailed annotation consistency results and question-type distributions are included in the Appendix C.