WACV 2027 · Algorithms Track
Learning to Adapt LVLM Attention for Reasoning Video Segmentation
Attention Transformation for Localization through Adaptive Segmentation
1York University 2Macquarie University 3CSIRO Data61
Reasoning Video Object Segmentation (RVOS) aims to localize and segment objects referred to by natural-language expressions throughout a video. Although large vision-language models (LVLMs) implicitly encode localization cues in their attention maps, these attentions are often noisy, diffuse, and scattered across layers, heads, and modalities, limiting their direct use for segmentation. Existing methods typically rely on handcrafted attention-selection heuristics or computationally expensive attention refinement. We propose ATLAS, an attention adaptation framework that learns to transform raw LVLM attentions into object-centric localization cues. ATLAS treats attentions from all layers, heads, and modalities as structured features, combining high-resolution frame-level attentions that capture spatial information with low-resolution video-level attentions that encode temporal context. A lightweight U-Net equipped with temporal attention and FiLM-based modulation fuses these complementary representations to generate coarse segmentation heatmaps, which are propagated across the video using SAM2. Extensive experiments on challenging referring and reasoning video segmentation benchmarks show that ATLAS outperforms existing frozen-backbone methods.
Reasoning expressions call for an LVLM, but getting a precise location out of a frozen LVLM is unreliable.
End-to-end fine-tuning works, but it optimizes several large foundation models together. That costs a lot of compute, needs large annotated datasets, and ties the adaptation to one specific LVLM and segmenter.
LVLMs are trained for text generation, not coordinate prediction. Querying Qwen2.5-VL 200 times at temperature 0.7 gives coordinate estimates scattered widely across the image.
Visual-token attentions do encode localization, but averaging over all layers and heads gives diffuse, noisy maps. Prior training-free methods hand-pick a few heads or refine attention with repeated inference.
The gap: much of the localization signal is spread across layers and heads and goes unused. In video, the model also has to work out when the referred object becomes relevant, not only where it is. ATLAS learns to adapt the full multimodal attention stack into object-centric localization cues, keeping the LVLM and SAM2 frozen.
Given a video 𝒱 = {It}Tt=1 and a reasoning expression 𝓔, ATLAS predicts binary masks 𝓜 = {Mt}Tt=1 in three stages. Only the attention adaptation network is trained. The LVLM and SAM2 stay frozen.
Uniformly sample Ts keyframes (16 by default). Wrap the expression in a fixed object extraction prompt and have the LVLM generate a single response token.
Query each keyframe separately with a high visual-token budget (1024). Stack the response token's attention over all heads of layers L0…L, with L0 the middle layer, to get A(f)k ∈ ℝC×Rf×Wf. This captures where the object is.
Run one joint pass over all keyframes with a small budget (64 tokens) to get A(v)k. This captures when the object is relevant. Resize it and concatenate it with the frame-level stack: Ak ∈ ℝ2C×Rf×Wf.
A two-stage U-Net encodes Ak into Zk. A two-layer Transformer runs over the GAP vectors z1…Ts, and the result produces γk, βk, which modulate the bottleneck: Z′k = Zk ⊙ (1+γk) + βk. The decoder outputs Pk.
Each map's maximum pk = argmax Pk becomes a positive point that prompts a frozen SAM2 on its keyframe.
Starting from the earliest seed, propagate its mask through the video. Drop any later seed whose mask has IoU > τ = 0.8 with the propagated mask. Repeat until no seeds are left, giving M distinct tracklets.
Pick the tracklet with the highest per-keyframe Pearson correlation to Pk, averaged over keyframes. If the maximum correlation is below 0.1, return an empty mask.
{Expression}
What is the main object (or objects) referred to in the given expression or question?
Focus on the primary subject or agent involved in the described action or behavior.
Respond with a single word (e.g., "cat", "person", "dog") that best describes the target object(s).
Treat the whole attention stack as a feature map and learn how to read it, instead of hand-selecting or repeatedly refining attention maps.
Attentions from every head of the middle-to-last layers are stacked as spatial feature channels. A supervised adapter learns to turn them into object-centric localization cues, with no handcrafted head selection and no costly refinement.
High-resolution frame-level attentions provide spatial detail. Low-resolution video-level attentions provide temporal context. A temporal Transformer and FiLM inject this context into each keyframe's spatial decoding.
Only the small adapter is trained (≈6 GPU hours, Ref-YouTube-VOS only). The LVLM and SAM2 stay frozen, and ATLAS achieves state-of-the-art results among frozen-backbone methods.
ATLAS predictions (green overlay) on ReasonVOS videos. The expression is printed at the top of each clip. Expressions are shown as rendered in the clips, where they are truncated.
ATLAS is trained only on Ref-YouTube-VOS and is evaluated across LVLM backbones. It outperforms frozen-backbone methods and is competitive with fully fine-tuned approaches.
| Method | LVLM | Ref-DAVIS | ReasonVOS | ReVOS (Overall) | ReVOS (Referring) | ReVOS (Reasoning) | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 𝒥&𝓕 | 𝒥 | 𝓕 | 𝒥&𝓕 | 𝒥 | 𝓕 | 𝒥&𝓕 | 𝒥 | 𝓕 | 𝒥&𝓕 | 𝒥 | 𝓕 | 𝒥&𝓕 | 𝒥 | 𝓕 | ||
| Approaches with tuned foundation models | ||||||||||||||||
| LISA | LLaVA-7B | 64.8 | 62.2 | 67.3 | 31.1 | 29.1 | 33.1 | 40.9 | 39.1 | 42.7 | 45.7 | 44.3 | 47.1 | 36.1 | 33.8 | 38.4 |
| VISA | ChatUniVi-7B | 69.4 | 66.3 | 72.5 | – | – | – | 46.9 | 44.9 | 49.0 | 50.9 | 49.2 | 52.6 | 43.0 | 40.6 | 45.4 |
| GLUS | LLaVA-7B | – | – | – | 49.9 | 47.5 | 52.4 | 54.9 | 52.4 | 57.3 | 58.3 | 56.0 | 60.7 | 51.4 | 48.8 | 53.9 |
| VideoLISA | LLaVA-Phi-3-V | 68.8 | 64.9 | 72.7 | 47.5 | 45.1 | 49.9 | – | – | – | – | – | – | – | – | – |
| VRS-HQ | ChatUniVi-7B | 76.0 | 72.6 | 79.4 | – | – | – | 59.1 | 56.6 | 61.6 | 62.1 | 59.8 | 64.5 | 56.1 | 53.5 | 58.7 |
| Veason-R1 | Qwen2.5-VL-7B | – | – | – | 59.9 | 56.0 | 63.8 | 61.3 | 58.2 | 64.4 | 63.6 | 60.7 | 66.5 | 59.0 | 55.8 | 62.2 |
| Approaches with frozen foundation models | ||||||||||||||||
| Loc-Head† | LLaVA-7B | 56.3 | 52.1 | 60.5 | 33.6 | 29.3 | 38.0 | 32.5 | 28.2 | 36.9 | 36.9 | 32.5 | 41.3 | 28.1 | 23.8 | 32.5 |
| DecAF† | LLaVA-OV-7B | 59.4 | 54.8 | 64.0 | 52.8 | 49.3 | 56.3 | 40.0 | 35.8 | 44.1 | 43.4 | 39.1 | 47.6 | 36.6 | 32.6 | 40.7 |
| ATLAS | LLaVA-OV-7B | 68.6 | 65.9 | 71.3 | 53.0 | 49.8 | 56.2 | 48.0 | 44.8 | 51.3 | 53.4 | 50.4 | 56.4 | 43.0 | 39.5 | 46.6 |
| Loc-Head† | InternVL3-8B | 66.3 | 62.4 | 70.2 | 44.3 | 41.0 | 47.5 | 43.7 | 39.9 | 47.5 | 46.7 | 42.9 | 50.6 | 40.7 | 36.9 | 44.5 |
| DecAF† | InternVL3-8B | 62.8 | 56.9 | 68.6 | 58.9 | 55.1 | 62.7 | 47.4 | 43.7 | 51.2 | 51.7 | 47.9 | 55.5 | 43.2 | 39.5 | 46.8 |
| ATLAS | InternVL3-8B | 74.4 | 71.5 | 77.4 | 57.2 | 54.2 | 60.2 | 53.5 | 50.8 | 56.2 | 60.0 | 57.4 | 62.6 | 47.4 | 44.5 | 50.3 |
| Loc-Head† | Qwen2-VL-7B | 61.9 | 58.0 | 65.8 | 34.0 | 31.8 | 36.2 | 44.0 | 40.8 | 47.2 | 52.7 | 49.1 | 56.2 | 35.4 | 32.6 | 38.2 |
| DecAF† | Qwen2-VL-7B | 64.1 | 59.4 | 68.9 | 52.5 | 49.0 | 56.0 | 45.3 | 41.6 | 49.0 | 52.7 | 48.9 | 56.4 | 37.9 | 34.3 | 41.5 |
| ATLAS | Qwen2-VL-7B | 75.5 | 72.1 | 78.9 | 64.4 | 61.8 | 67.0 | 53.8 | 50.8 | 56.9 | 59.8 | 56.9 | 62.6 | 48.3 | 44.9 | 51.7 |
| Loc-Head† | Qwen2.5-VL-7B | 64.6 | 60.2 | 68.9 | 41.1 | 37.9 | 44.3 | 47.0 | 43.3 | 50.7 | 53.1 | 49.3 | 56.9 | 40.8 | 37.2 | 44.4 |
| DecAF† | Qwen2.5-VL-7B | 75.2 | 70.9 | 79.5 | 63.9 | 60.5 | 67.2 | 54.2 | 50.1 | 58.2 | 58.7 | 54.8 | 62.6 | 49.7 | 45.4 | 53.9 |
| ATLAS | Qwen2.5-VL-7B | 80.1 | 76.8 | 83.4 | 67.4 | 64.7 | 70.2 | 56.2 | 53.5 | 59.0 | 61.5 | 58.9 | 64.1 | 51.0 | 48.1 | 53.9 |
| Method | LVLM | MeViS | Ref-YouTube-VOS† | ||||
|---|---|---|---|---|---|---|---|
| 𝒥&𝓕 | 𝒥 | 𝓕 | 𝒥&𝓕 | 𝒥 | 𝓕 | ||
| VISA | Chat-UniVi-7B | 44.5 | 41.8 | 47.1 | 61.5 | 59.8 | 63.2 |
| VideoLISA | LLaVA-Phi-3-V | 44.4 | 41.3 | 47.6 | 63.7 | 61.7 | 65.7 |
| GLUS | LLaVA-7B | 51.3 | 48.5 | 54.2 | 67.3 | 65.5 | 69.0 |
| VRS-HQ | Chat-UniVi-7B | 50.6 | 47.6 | 53.7 | 70.4 | 68.3 | 72.5 |
| Veason-R1 | Qwen2.5-VL-7B | 52.2 | 48.4 | 56.0 | – | – | – |
| Loc-Head | Qwen2.5-VL-7B | 39.4 | 35.5 | 43.6 | 51.0 | 46.8 | 55.2 |
| DecAF | Qwen2.5-VL-7B | 48.1 | 44.0 | 52.1 | 59.9 | 56.2 | 63.5 |
| ATLAS | Qwen2.5-VL-7B | 54.5 | 51.0 | 58.0 | 69.5 | 67.7 | 71.3 |
| ATLAS | Qwen2-VL-7B | 52.1 | 48.4 | 55.8 | 71.0 | 69.2 | 72.8 |
| ATLAS | LLaVA-OV-7B | 46.0 | 42.2 | 49.9 | 59.9 | 58.1 | 61.7 |
| ATLAS | InternVL3-8B | 51.4 | 47.9 | 54.8 | 66.0 | 64.2 | 67.8 |
| Dataset | Method | 𝒥 | 𝓕 | 𝒥&𝓕 |
|---|---|---|---|---|
| Ref-DAVIS | ATLAS | 47.2 | 74.0 | 60.6 |
| Raw Attn. | 2.0 | 18.2 | 10.1 | |
| ReasonVOS | ATLAS | 40.3 | 57.4 | 48.9 |
| Raw Attn. | 4.3 | 27.3 | 15.8 |
Adaptation raises keyframe localization 𝒥&𝓕 from 10.1 to 60.6 on Ref-DAVIS and from 15.8 to 48.9 on ReasonVOS, turning diffuse LVLM attentions into reliable cues for mask propagation.
| Method | vid / frm | Attn. | Adapt. | Seed | SAM2 | E2E | 𝒥&𝓕 |
|---|---|---|---|---|---|---|---|
| ATLAS (ours) | 64 / 1024 | 2.06 | 0.003 | 0.001 | 13.2 | 15.3 | 67.4 |
| ATLAS | 256 / 1024 | 2.46 | 0.003 | 0.001 | 13.1 | 15.6 | 67.5 |
| DecAF | 256 / 1024 | 8.08 | – | 0.030 | 33.2 | 41.3 | 63.9 |
At the matched 256-video-token setting, ATLAS is 2.7× faster end-to-end (15.6 s vs. 41.3 s) and more accurate (67.5 vs. 63.9).
All ablations use Qwen2.5-VL-7B.
| Method | Ref-DAVIS | ReasonVOS | ||||
|---|---|---|---|---|---|---|
| 𝒥 | 𝓕 | 𝒥&𝓕 | 𝒥 | 𝓕 | 𝒥&𝓕 | |
| Frame only | 75.0 | 80.1 | 77.5 | 57.8 | 63.0 | 60.4 |
| Frame + Video | 76.3 | 82.0 | 79.1 | 62.2 | 67.7 | 64.9 |
| Frame + FiLM | 75.5 | 81.3 | 78.4 | 58.8 | 64.1 | 61.5 |
| Frame + Video + FiLM | 76.8 | 83.4 | 80.1 | 64.7 | 70.2 | 67.4 |
The two components are complementary. On ReasonVOS, combining them adds +7.0 𝒥&𝓕, compared with +2.6 on Ref-DAVIS. This suggests reasoning expressions rely more on temporal cues.
| Adapter | Ref-DAVIS | ReasonVOS | ||||
|---|---|---|---|---|---|---|
| 𝒥 | 𝓕 | 𝒥&𝓕 | 𝒥 | 𝓕 | 𝒥&𝓕 | |
| Linear | 53.6 | 61.5 | 57.5 | 45.3 | 51.3 | 48.3 |
| MLP | 70.1 | 76.2 | 73.2 | 59.4 | 64.5 | 61.9 |
| Conv3 | 71.2 | 77.3 | 74.3 | 58.6 | 63.9 | 61.3 |
| U-Net | 76.3 | 82.0 | 79.1 | 62.2 | 67.7 | 64.9 |
| U-Net + FiLM | 76.8 | 83.4 | 80.1 | 64.7 | 70.2 | 67.4 |
A linear probe falls far short. The MLP and single-scale conv head come within 4–5 points of the U-Net, which suggests the U-Net's gains come mainly from multi-scale aggregation.
Per-frame Pearson correlation is the best practical criterion on both benchmarks. Its scale and offset invariance suits uncalibrated attention maps. The oracle gap shows there is room for better selection.
| # Keyframes | 4 | 8 | 12 | 16 | 20 | 24 |
|---|---|---|---|---|---|---|
| 𝒥 | 59.4 | 61.6 | 63.0 | 64.7 | 63.5 | 64.2 |
| 𝓕 | 64.5 | 66.8 | 68.9 | 70.2 | 70.1 | 70.4 |
| 𝒥&𝓕 | 62.0 | 64.2 | 65.9 | 67.4 | 66.8 | 67.3 |
𝒥&𝓕 rises from 62.0 with 4 keyframes to a peak of 67.4 at 16, then saturates. 16 keyframes is the default.
@inproceedings{dastmalchi2027atlas,
title = {ATLAS: Learning to Adapt LVLM Attention for Reasoning Video Segmentation},
author = {Dastmalchi, Hamidreza and Cheraghian, Ali and An, Aijun and Khamis, Abdelwahed and Petersson, Lars},
booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
year = {2027}
}