Qwen3-VL-Seg 全文精读

摘要 (Abstract)

原文

Open-world referring segmentation requires grounding unconstrained language expressions to precise pixel-level regions. Existing multimodal large language models (MLLMs) exhibit strong open-world visual grounding, but their outputs remain limited to sparse bounding-box coordinates and are insufficient for dense visual prediction. Recent MLLM-based segmentation methods either directly predict sparse contour coordinates, struggling to reconstruct continuous object boundaries, or rely on external segmentation foundation models such as the Segment Anything Model (SAM), introducing substantial architectural and deployment overhead. We present Qwen3-VL-Seg, a parameter-efficient framework that treats the MLLM-predicted box as a semantically grounded structural prior and decodes it into pixel-level referring segmentation. At its core, a lightweight box-guided mask decoder combines multi-scale spatial feature injection, spatial-semantic query construction, box-guided high-resolution pixel fusion, and iterative mask-aware query refinement, introducing only 17M parameters (about 0.4% of the base model). For scalable open-world training, we construct SA1B-ORS, an SA-1B-derived dataset with two subsets: SA1B-CoRS (category-oriented samples) and SA1B-DeRS (descriptive, instance-specific samples). For evaluation, we curate ORS-Bench, a manually screened benchmark with in-distribution and out-of-distribution subsets covering diverse referring expression types. Extensive experiments on referring expression segmentation, visual grounding, and ORS-Bench show that Qwen3-VL-Seg performs strongly across closed-set and open-world settings, with clear advantages on language-intensive instructions and strong out-of-distribution generalization. Evaluations on general multimodal benchmarks further show that the model broadly preserves general-purpose multimodal competence after segmentation-oriented adaptation.

翻译

开放世界指代分割（Open-world referring segmentation）需要把不受限的自然语言表达，对应到精确的像素级区域。现有的多模态大模型（MLLM）具备很强的开放世界视觉定位（grounding）能力，但其输出仍局限于稀疏的边界框坐标，不足以支撑密集视觉预测。近期基于 MLLM 的分割方法要么直接预测稀疏的轮廓坐标——难以重建连续的物体边界；要么依赖 SAM（Segment Anything Model）等外部分割基础模型——带来大量的架构与部署开销。我们提出 Qwen3-VL-Seg：一个参数高效的框架，把 MLLM 预测出的框当作”语义已对齐的结构先验”，再将其解码为像素级的指代分割。其核心是一个轻量的 box-guided 掩码解码器，融合了多尺度空间特征注入、空间-语义 query 构造、box 引导的高分辨率像素融合、以及迭代式掩码感知 query 精炼，仅新增 1700 万参数（约为基座模型的 0.4%）。为支持可扩展的开放世界训练，我们构建了 SA1B-ORS 数据集（源自 SA-1B），含两个子集：SA1B-CoRS（类别导向样本）与 SA1B-DeRS（描述性、实例特定样本）。为评测，我们整理了 ORS-Bench，一个经人工筛查、含分布内与分布外子集、覆盖多种指代表达类型的基准。在指代表达分割、视觉定位、以及 ORS-Bench 上的大量实验表明，Qwen3-VL-Seg 在闭集与开放世界设定下都表现强劲，在语言密集型指令上优势明显，且分布外泛化能力强。通用多模态基准上的评测进一步表明，经过面向分割的适配后，该模型仍广泛保留了通用多模态能力。

1　引言 (Introduction)

摘要之后，引言从 visual grounding 讲起，一步步交代研究动机，最终引出全文要回答的核心问题。

原文

The rapid advancement of Multimodal Large Language Models (MLLMs) has significantly expanded the frontier of visual understanding. Beyond image-level question answering and multimodal dialogue, many vision-language tasks require a model to precisely associate linguistic concepts with specific image regions. This capability, commonly referred to as visual grounding, forms a critical bridge between linguistic semantics and fine-grained visual evidence.

翻译

多模态大模型（MLLM）的快速发展，极大地拓展了视觉理解的边界。除了图像级问答和多模态对话之外，许多视觉-语言任务还要求模型把语言概念与图像中的特定区域精确关联起来。这种能力通常被称为”视觉定位”（visual grounding），它是连接语言语义与细粒度视觉证据的关键桥梁。

大模型不光要能”看图说话”，还得能做到”你说哪儿、它指哪儿”——把一句话里的词，准确落到图上的具体位置。这个”按话找位置”的本事就叫 visual grounding。

原文

Recent frontier MLLMs, such as GPT, Gemini, and Qwen-VL, have demonstrated substantial progress in visual grounding. Through large-scale instruction tuning, these models can localize arbitrary visual entities with natural language instructions, enabling them to move beyond closed-set vocabularies. However, their outputs are still typically limited to bounding boxes, which provide only coarse spatial localization. Such box-level grounding is insufficient for applications requiring precise boundary-aware spatial reasoning, including robotic manipulation, medical image analysis, and fine-grained image editing. A natural next step is therefore to move from box-level localization to pixel-level mask prediction, namely referring segmentation.

翻译

近期的前沿 MLLM（如 GPT、Gemini、Qwen-VL）在视觉定位上取得了长足进步。通过大规模指令微调，这些模型能根据自然语言指令定位任意视觉实体，从而突破了闭集词表的限制。然而，它们的输出通常仍局限于边界框，只能提供粗糙的空间定位。这种”框级”定位，对于需要精确边界感知的应用是不够的——比如机器人操作、医学影像分析、精细图像编辑。因此，自然而然的下一步，就是从”框级定位”迈向”像素级掩码预测”，也就是指代分割（referring segmentation）。

现在的大模型已经能”按话画框”了，而且不限词表（你说啥它都能找）。但只画个框太糙——机器人抓取、看医学片子、精修图这些活儿，需要贴着物体边缘的精确轮廓。所以下一步得从”画框”升级成”精确抠图”。

原文

Existing approaches to extending MLLMs from box-level grounding to pixel-level referring segmentation mainly follow two technical routes. The first couple of MLLMs with external segmentation models, most commonly SAM. For example, LISA, GSVA, and SAM4MLLM use MLLM representations to identify target regions and then invoke SAM or SAM-like decoders to generate masks. Although effective, these decoupled pipelines introduce nontrivial parameter overhead, additional architectural dependencies, and less streamlined deployment. Separately, SAM3 studies concept-prompted segmentation within a segmentation foundation model rather than an MLLM-plus-decoder formulation. The second route avoids external segmenters by equipping MLLMs with lightweight dense prediction heads or autoregressive mask representations. Examples include Text4Seg, UFO, and recent lightweight mask-decoder approaches such as MLLMSeg. However, text-based formulations often suffer from discretization artifacts or representation bottlenecks, while lightweight dense decoders still struggle to recover precise boundaries from low-resolution latent features alone.

翻译

现有把 MLLM 从”框级定位”扩展到”像素级指代分割”的方法，主要走两条技术路线。

第一条把 MLLM 与外部分割模型（最常见的是 SAM）耦合：例如 LISA、GSVA、SAM4MLLM 用 MLLM 的表示来识别目标区域，再调用 SAM 或类 SAM 解码器生成掩码。这类方法虽然有效，但这种解耦式流水线会带来不小的参数开销、额外的架构依赖，以及不够顺畅的部署。此外，SAM3 研究的是在分割基础模型内部做”概念提示”分割，而非”MLLM + 解码器”这种范式。

第二条路线避免外部分割器，给 MLLM 配上轻量的密集预测头或自回归掩码表示，代表方法有 Text4Seg、UFO，以及近期的轻量掩码解码器方法如 MLLMSeg。然而，基于文本的表述常受离散化伪影或表示瓶颈之苦；而轻量密集解码器仅靠低分辨率的隐层特征，仍难以恢复精确边界。

以前想让大模型精确抠图，无非两招。招一：外挂 SAM——大模型负责”指哪儿”，SAM 负责”抠图”，效果好但又大又难部署。招二：给大模型自己接个小尾巴直接吐掩码——轻便，但要么把图”像素化描点”损失精度，要么因为特征分辨率太低抠不出清晰边缘。两招都有硬伤。

原文

Open-world referring segmentation (ORS) remains under-explored in existing benchmarks and methods. In terms of benchmarks, dominant datasets remain constrained by closed-set vocabularies. Classic Referring Expression Segmentation (RES) benchmarks are built around a relatively limited object space, while Generalized RES (GRES) broadens the scope to include multi-target and no-target cases, and ReasonSeg advances the paradigm by incorporating reasoning-intensive instructions. Despite these progresses, existing benchmarks fail to fully leverage the open-vocabulary capabilities inherent in modern MLLMs to achieve open-world training and evaluation.

翻译

开放世界指代分割（ORS）在现有基准和方法中都仍未被充分探索。在基准方面，主流数据集仍受限于闭集词表：经典的指代表达分割（RES）基准围绕相对有限的物体空间构建；广义 RES（GRES）把范围扩展到多目标和无目标情形；ReasonSeg 则通过引入推理密集型指令推进了这一范式。尽管有这些进展，现有基准仍未能充分利用现代 MLLM 内在的开放词表能力，来实现开放世界的训练与评测。

现在的测试集词表都太”封闭”——就那么些物体类别。虽然后来有 GRES（允许多个/没有目标）、ReasonSeg（要动脑推理）等改进，但都没把大模型”啥都能认”的开放能力用起来。所以”开放世界”这块，数据和方法都还是空白。

原文

Methodologically, existing approaches either rely on heavyweight external segmentation components or still struggle to recover precise boundaries when using compact dense predictors. Moreover, most methods do not directly exploit the grounding priors already present in modern MLLMs. This raises a central question: can we transform the open-world grounding capability of MLLMs into pixel-level referring segmentation without relying on heavyweight external segmentation models?

翻译

在方法层面，现有方案要么依赖重型的外部分割组件，要么在使用紧凑密集预测器时仍难以恢复精确边界。更重要的是，大多数方法并没有直接利用现代 MLLM 中已经存在的定位先验。这就引出了一个核心问题：我们能否在不依赖重型外部分割模型的前提下，把 MLLM 的开放世界定位能力，转化为像素级的指代分割？

原文

To address these limitations, we introduce Qwen3-VL-Seg, a parameter-efficient framework that converts box-level grounding priors of a pretrained MLLM into pixel-level referring segmentation without relying on an external segmentation model. Our key insight is that the bounding box predicted by the MLLM is not merely a terminal output, but a semantically grounded and spatially informative anchor for mask decoding. Consequently, we leverage the box as a structural prior throughout the decoder. Specifically, we design a lightweight box-guided mask decoder with four components: (i) multi-scale spatial feature injection, which enriches intermediate visual features for dense prediction; (ii) spatial-semantic query construction, which fuses box geometry and language features into an object query; (iii) box-guided high-resolution pixel fusion, which converts the predicted box into a differentiable soft gate to inject fine-grained image textures while suppressing background clutter; and (iv) iterative mask-aware query refinement, which feeds first-pass mask evidence back into the query to progressively sharpen object boundaries. In this way, box-level grounding is systematically transformed into mask-level prediction while introducing only 17M additional parameters, accounting for approximately 0.4% of the base MLLM.

翻译

为解决这些局限，我们提出 Qwen3-VL-Seg：一个参数高效的框架，能把预训练 MLLM 的框级定位先验转化为像素级指代分割，且不依赖外部分割模型。我们的关键洞察是：MLLM 预测的边界框不只是一个终端输出，而是一个”语义已对齐、且富含空间信息”的掩码解码锚点。因此，我们把这个框当作结构先验贯穿整个解码器。具体地，我们设计了一个轻量的 box-guided 掩码解码器，含四个组件：(i) 多尺度空间特征注入，为密集预测丰富中间层视觉特征；(ii) 空间-语义 query 构造，把框的几何信息与语言特征融合成一个 object query；(iii) box 引导的高分辨率像素融合，把预测框变成一个可微的”软门”，注入细粒度图像纹理、同时抑制背景杂波；(iv) 迭代式掩码感知 query 精炼，把第一遍掩码的证据反馈回 query，逐步锐化物体边界。这样一来，框级定位就被系统性地转化为掩码级预测，而仅新增 1700 万参数，约占基座 MLLM 的 0.4%。

这就是本文的核心方案。想法很巧：大模型画的那个框，别扔，把它当”线索”用到底。围绕这个框做了四步小加工：

1. 把中间层的图像特征捞出来补细节；
2. 把框的位置和那句话揉成一个”查询向量”；
3. 用框做个”软探照灯”，只照框里、屏蔽框外杂物；
4. 抠一遍不够就拿结果再抠一遍，把边缘磨清楚。

全部加起来只多了 0.4% 的参数。

原文

Towards model training, we utilize existing public datasets including the RefCOCO series, LVIS, and COCO as supervised sources. To further scale open-world referring segmentation, we construct SA1B-ORS, an SA-1B-derived dataset built from a sampled pool of 2 million raw SA-1B images. SA1B-ORS is a composite dataset with two complementary subsets: SA1B-CoRS, which converts category-agnostic mask fragments into category-level referring supervision where one expression may refer to one or multiple instances of the same semantic class; and SA1B-DeRS, which provides instance-specific supervision with attributes, relations, or contextual cues so that a single target can be uniquely identified. In total, SA1B-ORS contains 1.05 million SA1B-CoRS samples and 1.94 million SA1B-DeRS samples, providing both broad category-level coverage and fine-grained descriptive supervision.

翻译

在模型训练方面，我们使用了现有公开数据集作为监督来源，包括 RefCOCO 系列、LVIS 和 COCO。为进一步扩大开放世界指代分割的规模，我们构建了 SA1B-ORS——一个源自 SA-1B 的数据集，从 200 万张 SA-1B 原始图像的采样池中构建。SA1B-ORS 是一个复合数据集，含两个互补子集：SA1B-CoRS，它把类别无关的掩码碎片转化为类别级的指代监督，此时一条表达可以指代同一语义类别的一个或多个实例；以及 SA1B-DeRS，它用属性、关系或上下文线索提供实例特定的监督，使得单个目标能被唯一确定。总计，SA1B-ORS 含 105 万条 SA1B-CoRS 样本和 194 万条 SA1B-DeRS 样本，兼具广泛的类别级覆盖与细粒度的描述性监督。

训练要吃数据。公开的 RefCOCO/LVIS/COCO 先用上，但不够，于是自己造了个大数据集 SA1B-ORS（从 200 万张图里挖）。它分两半：CoRS——按”类别”指（如”所有的人”，可能对应好几个）；DeRS——按”描述”指（如”右边穿条纹衫的那个人”，只对应一个）。加起来近 300 万条。

原文

To systematically evaluate model generalization, we introduce ORS-Bench, a benchmark suite containing an in-distribution subset (ORS-ID-Bench) and an out-of-distribution subset (ORS-OOD-Bench). ORS-ID-Bench includes 9,055 manually curated samples across four instruction formats, providing a reliable assessment aligned with the training distribution. ORS-OOD-Bench is designed to probe the limits of model capability and robustness under challenging scenarios by covering six distributional shifts: category, instance scale, instruction complexity, occlusion, lighting, and domain risk. Guided by training distribution analysis, we curate approximately 200 challenging samples per dimension from non-overlapping sources, spanning extreme spatial extents, indirect reasoning, adverse environments, and safety-critical domains such as autonomous driving and medical diagnosis. ORS-Bench thus offers a rigorous testbed for measuring the operational limits of referring segmentation models.

翻译

为系统评测模型的泛化能力，我们提出 ORS-Bench 基准套件，含一个分布内子集（ORS-ID-Bench）和一个分布外子集（ORS-OOD-Bench）。ORS-ID-Bench 含 9,055 条人工整理的样本，覆盖四种指令格式，提供与训练分布对齐的可靠评估。ORS-OOD-Bench 则用于探测模型在挑战场景下的能力上限与鲁棒性，覆盖六种分布偏移：类别、实例尺度、指令复杂度、遮挡、光照、以及领域风险。在训练分布分析的指导下，我们从不重叠的来源中，为每个维度整理约 200 个挑战样本，涵盖极端空间尺度、间接推理、恶劣环境，以及自动驾驶、医学诊断等安全攸关领域。因此，ORS-Bench 提供了一个严格的测试平台，用以衡量指代分割模型的能力边界。

光有数据还得有考卷。ORS-Bench 分两套：ID（分布内）——9055 题，跟训练风格一致，测常规水平；OOD（分布外）——专挑六种”没见过/很刁钻”的情况（罕见类别、超大超小、绕弯说话、遮挡、暗光、自动驾驶/医疗），每类约 200 题，专门考模型极限。

原文

Comprehensive experiments on referring expression segmentation and visual grounding on both public and our curated benchmarks show that Qwen3-VL-Seg performs strongly across both closed-set and open-world settings. The gains are especially pronounced on language-intensive open-world instructions, where precise alignment between semantic understanding and mask prediction is most critical. In addition, the model demonstrates strong out-of-distribution generalization on ORS-OOD-Bench, while additional general multimodal evaluation shows that it broadly preserves general-purpose multimodal competence after segmentation-oriented adaptation.

翻译

在公开基准和我们自建基准上，围绕指代表达分割与视觉定位开展的全面实验表明：Qwen3-VL-Seg 在闭集与开放世界两种设定下都表现强劲。在语言密集型的开放世界指令上增益尤为明显——这类场景最考验”语义理解”与”掩码预测”之间的精确对齐。此外，模型在 ORS-OOD-Bench 上展现出很强的分布外泛化；额外的通用多模态评测还表明，经过面向分割的适配后，它仍广泛保留了通用多模态能力。

实验结论提前剧透：常规和开放世界都强，尤其”要读懂长句子才能找对目标”的场景优势最大；刁钻的 OOD 题也扛得住；而且练了抠图之后，原来的通用本事（问答、推理、OCR 等）基本没退步。

原文

In summary, our main contributions are as follows: (1) We introduce Qwen3-VL-Seg, a parameter-efficient framework that converts the open-world grounding priors of pretrained MLLMs into pixel-level referring segmentation without relying on external foundation models such as SAM. (2) We propose a lightweight box-guided mask decoder that leverages MLLM-predicted boxes as structural priors and transforms them into precise masks through multi-scale spatial feature injection, spatial-semantic query construction, box-guided high-resolution pixel fusion, and iterative mask-aware query refinement. (3) We construct SA1B-ORS, an SA-1B-derived open-world referring segmentation dataset with two complementary subsets: SA1B-CoRS for category-oriented references and SA1B-DeRS for fine-grained descriptive references. (4) We introduce ORS-Bench, comprising ORS-ID-Bench and ORS-OOD-Bench, to evaluate open-world referring segmentation under both in-distribution formats and six distinct out-of-distribution shifts. (5) We conduct extensive experiments showing that Qwen3-VL-Seg achieves strong performance across both closed-set and open-world settings, with especially clear gains on language-intensive instructions, strong out-of-distribution generalization, and broad preservation of general-purpose multimodal competence.

翻译

总结起来，我们的主要贡献如下：

1. 提出 Qwen3-VL-Seg：一个参数高效的框架，把预训练 MLLM 的开放世界定位先验转化为像素级指代分割，且不依赖 SAM 等外部基础模型。

2. 提出一个轻量的 box-guided 掩码解码器，把 MLLM 预测的框当作结构先验，通过多尺度空间特征注入、空间-语义 query 构造、box 引导的高分辨率像素融合、迭代式掩码感知 query 精炼，将其转化为精确掩码。

3. 构建 SA1B-ORS 数据集（源自 SA-1B），含两个互补子集：面向类别指代的 SA1B-CoRS 与面向细粒度描述指代的 SA1B-DeRS。

4. 提出 ORS-Bench（含 ORS-ID-Bench 与 ORS-OOD-Bench），在分布内格式与六种分布外偏移下评测开放世界指代分割。

5. 开展大量实验，表明 Qwen3-VL-Seg 在闭集与开放世界设定下都表现强劲，在语言密集型指令上增益尤为明显，分布外泛化强，且广泛保留了通用多模态能力。

把功劳打包成五条：

1. 提了个新框架（不外挂）；
2. 核心是那个四步小解码器；
3. 造了训练数据 SA1B-ORS；
4. 造了考卷 ORS-Bench；
5. 做了充分实验证明它好用。

①② 是方法，③④ 是数据/评测，⑤ 是结果。

2　相关工作 (Related Work)

交代完动机与贡献，这一章把相关研究按任务、模型、数据三条线梳理一遍，顺带点出本文与它们的差异。

2.1　指代表达分割 (Referring Expression Segmentation)

原文

Referring Expression Segmentation (RES) studies pixel-level localization of objects specified by language expressions, with classic benchmarks largely built around the one-expression, one-instance setting. Generalized RES (GRES) extends this formulation to multi-target and no-target cases through the gRefCOCO benchmark, while LISA broadens the instruction space toward reasoning-intensive segmentation. These works enrich the expression space of language-guided segmentation, but the dominant benchmarks remain largely human-curated and closed-vocabulary, limiting scalability to open-world entity spaces.

翻译

指代表达分割（RES）研究的是：对语言表达所指定的物体做像素级定位；其经典基准大多围绕”一条表达、一个实例”的设定构建。广义 RES（GRES）通过 gRefCOCO 基准，把这一表述扩展到多目标和无目标的情形；LISA 则把指令空间拓宽到推理密集型的分割。这些工作丰富了语言引导分割的表达空间，但主流基准仍主要靠人工整理、且是闭集词表，限制了向开放世界实体空间的可扩展性。

RES 就是”按话抠图”这门任务的统称。老设定是一句话对一个物体；后来 GRES 允许对多个或没有，LISA 允许出推理题。进步归进步，但数据都靠人工标、词表还封闭，撑不起”开放世界”。

2.2　用于区域理解与分割的 MLLM (MLLMs for region understanding and segmentation)

原文

A parallel line of work equips MLLMs with explicit region grounding ability. Representative models such as Shikra and Ferret support open-vocabulary region understanding, while more recent systems such as Youtu-VL suggest that vision-centric supervision can be absorbed into standard MLLM training at scale. Building on these advances, recent methods extend MLLMs from region grounding to pixel-level prediction. SAM-based approaches, such as LISA, GSVA, and SAM4MLLM, couple MLLMs with external segmentation models to obtain high-quality masks, but inherit the corresponding parameter and deployment cost. Recent progress on promptable concept segmentation, exemplified by SAM3, further strengthens this external-model line by extending large segmentation foundation models to concept-prompted segmentation, detection, and tracking. These models provide strong open-world mask prediction capability, but remain architecturally distinct from MLLM-native referring segmentation and preserve the deployment cost of large external segmenters. SAM-free approaches instead rely on lightweight dense heads or unified autoregressive formulations, including PerceptionGPT, Text4Seg, UFO, and MLLMSeg. Our approach is also lightweight and SAM-free at inference, but uses a different inductive bias: the MLLM-predicted box is treated as a structural prior for query construction, pixel fusion, and iterative refinement.

翻译

另一条并行的研究线，是给 MLLM 装上显式的区域定位能力。代表模型如 Shikra、Ferret 支持开放词表的区域理解；更近的系统如 Youtu-VL 表明，以视觉为中心的监督可以被大规模地吸收进标准 MLLM 训练中。在这些进展的基础上，近期方法把 MLLM 从”区域定位”扩展到”像素级预测”。

基于 SAM 的方法（如 LISA、GSVA、SAM4MLLM）把 MLLM 与外部分割模型耦合以获得高质量掩码，但也继承了相应的参数与部署成本。近期以 SAM3 为代表的”可提示概念分割”进展，进一步强化了这条外部模型路线——把大型分割基础模型扩展到概念提示的分割、检测与跟踪。这些模型提供了强大的开放世界掩码预测能力，但在架构上仍与”MLLM 原生的指代分割”不同，并保留了大型外部分割器的部署成本。

免 SAM 的方法则依赖轻量密集头或统一的自回归表述，包括 PerceptionGPT、Text4Seg、UFO 和 MLLMSeg。我们的方法在推理时同样轻量且免 SAM，但采用了不同的归纳偏置：把 MLLM 预测的框当作结构先验，用于 query 构造、像素融合和迭代精炼。

这一段把”同行”分成两派。外挂派（LISA/SAM3 等）抠图质量高，但拖着 SAM 这个大包袱；自研派（Text4Seg/UFO/MLLMSeg 等）轻便但精度有限。本文也属于自研派、推理时不挂 SAM，但独特之处在于：别人不怎么用那个框，本文专门拿框当核心线索——这就是它和别人的根本区别。

2.3　开放世界指代分割的数据 (Data for open-world referring segmentation)

原文

Existing referring segmentation datasets, including RefCOCO series, gRefCOCO, and ReasonSeg, have been crucial to progress, but they are limited in vocabulary, scale, or construction cost. Our work focuses on scalable construction of open-world referring segmentation data from SA-1B. SA1B-ORS is organized into two complementary subsets: SA1B-CoRS converts fragmented category-agnostic masks into category-oriented referring samples through entity distillation, mask reconstruction, MLLM verification, and caption generation, while SA1B-DeRS provides descriptive instance-level instructions with grounding-based filtering. This dataset organization aligns training supervision with the open-world grounding regime of modern MLLMs.

翻译

现有的指代分割数据集（包括 RefCOCO 系列、gRefCOCO、ReasonSeg）对推动进展至关重要，但它们在词表、规模或构建成本上受限。我们的工作聚焦于从 SA-1B 出发、可扩展地构建开放世界指代分割数据。SA1B-ORS 被组织为两个互补子集：SA1B-CoRS 通过实体蒸馏、掩码重建、MLLM 校验和描述生成，把碎片化、类别无关的掩码转化为类别导向的指代样本；SA1B-DeRS 则用基于定位的过滤，提供描述性的实例级指令。这样的数据组织方式，使训练监督与现代 MLLM 的开放世界定位机制相对齐。

现成数据集不是词太少就是造起来太贵。本文的做法是”废物利用”——拿海量但没配文字的 SA-1B 掩码，用一套自动流水线（蒸馏→重建→大模型把关→配文字）把它们变成能训练的”图-话-掩码”三件套，从而低成本地做出海量开放世界数据。

3　方法 (Method)

进入全文最核心的技术章节：先给出整体框架，再逐一拆解 box-guided 解码器的四个组件。

3.1　概述 (Overview)

原文

Open-world referring segmentation aims to ground natural language expressions to specific image regions and generate corresponding pixel-level masks. Modern multimodal large language models (MLLMs) exhibit strong open-world grounding ability, but their outputs are typically limited to coarse localization, e.g., bounding boxes. This creates a mismatch between grounding and segmentation: grounding provides coarse spatial localization, whereas segmentation requires precise pixel-level delineation.

翻译

开放世界指代分割的目标是：把自然语言表达定位到图像的特定区域，并生成对应的像素级掩码。现代多模态大模型（MLLM）具备很强的开放世界定位能力，但其输出通常局限于粗糙的定位（如边界框）。这就在”定位”与”分割”之间造成了错配：定位给的是粗略的空间位置，而分割要求精确的像素级勾勒。

任务=按话抠图。矛盾在于：大模型只会给个粗框，可抠图要的是贴边缘的精细轮廓——这中间差着一大截，本章就是来补这个差距的。

原文

To bridge this gap, we propose a box-guided mask decoder architected upon a pretrained MLLM, as illustrated in Figure 1. Our key insight is to leverage the grounded bounding box as a structural prior for dense mask decoding, rather than treating it as a mere auxiliary prediction. Concretely, the decoder first enriches intermediate ViT features with lightweight spatial adapters and combines them with multimodal visual embeddings to form a dense memory representation. It then constructs object queries by jointly encoding language semantics and box geometry. To recover fine boundaries, we further introduce a box-guided high-resolution pixel fusion module, which softly gates shallow image features before fusing them with upsampled visual features. Finally, we perform iterative mask-aware query refinement by pooling target-aware pixel evidence from the first-pass mask and feeding it back to the query for a second-pass prediction.

翻译

为弥合这一鸿沟，我们提出一个构建在预训练 MLLM 之上的 box-guided 掩码解码器，如图 1 所示。关键洞察是：把定位得到的边界框当作密集掩码解码的结构先验，而不只是把它当成一个辅助输出。具体地，解码器首先用轻量的空间适配器丰富中间层 ViT 特征，并将其与多模态视觉嵌入结合，形成一个密集的 memory 表示；随后通过联合编码语言语义与框的几何信息来构造 object query；为恢复精细边界，我们进一步引入 box 引导的高分辨率像素融合模块，它先对浅层图像特征做”软门控”，再与上采样的视觉特征融合；最后，我们做迭代式掩码感知 query 精炼——从第一遍掩码中 pool 出目标感知的像素证据，反馈回 query，进行第二遍预测。

原文

Overall, the proposed decoder follows a coarse-to-fine design: the pretrained MLLM provides robust grounding priors, and the box-guided decoder progressively transforms these priors into fine-grained segmentation masks. Given an input image I and a referring expression, the pretrained MLLM provides four types of information: multi-scale visual features, multimodal visual embeddings T_mm, text-conditioned segmentation token features T_seg, and a grounded bounding box B_box. Our decoder predicts the final mask (Eq. 1).

翻译

总体而言，该解码器遵循”由粗到细”（coarse-to-fine）的设计：预训练 MLLM 提供稳健的定位先验，box-guided 解码器再把这些先验逐步转化为细粒度的分割掩码。给定输入图像 I 和一条指代表达，预训练 MLLM 提供四类信息：多尺度视觉特征 {F_vis^l}、多模态视觉嵌入 T_mm、文本条件的分割 token 特征 T_seg、以及一个定位得到的边界框 B_box。解码器据此预测最终掩码（式 1）：

$\hat{M} = \mathcal{D}\left(\{F_{vis}^{l}\}_{l=1}^{L},\ T_{mm},\ T_{seg},\ B_{box},\ I\right)$　(1)

解码器 D 的输入清单就四样东西——多层图像特征、融合了语言的视觉向量、代表”要分割”的那个 token、以及那个框，外加原图。喂进去，吐出最终掩码 M̂。下面四个小节就是拆解 D 里到底干了啥。

3.2　模型架构 (Model Architecture)

3.2.1　多尺度空间特征注入与 memory 构建

原文

High-level MLLM features are typically too coarse to support accurate boundary delineation. We therefore extract intermediate visual features and adapt them with a lightweight SpatialFeatureInjector. We inject local spatial bias through a depthwise branch, where s is a learnable scalar initialized to 1e-3. This near-zero initialization keeps the adapter close to an identity mapping at the start of fine-tuning, which stabilizes optimization. The adapted features and the projected top-layer feature are concatenated and fused by a lightweight convolutional fusion block. To construct the decoder memory, we project the multimodal visual embeddings and reshape them into a 2D feature map, and obtain the final memory feature, where P_mem denotes a learnable 2D positional encoding.

翻译

MLLM 的高层特征通常太粗，不足以支撑精确的边界勾勒。因此我们抽取中间层视觉特征，用一个轻量的 SpatialFeatureInjector 做适配（式 2–3）。我们通过一个 depthwise 分支注入局部空间偏置，其中缩放标量 s 初始化为 1e-3。这种”近零初始化”让适配器在微调开始时接近恒等映射，从而稳定优化。适配后的特征与投影后的顶层特征拼接，再由一个轻量卷积融合块融合（式 4）。为构建解码器 memory，我们把多模态视觉嵌入投影并 reshape 成 2D 特征图（式 5），得到最终 memory 特征（式 6），其中 P_mem 是可学习的 2D 位置编码。

$X_0^{(l)} = \mathrm{Conv}_{1\times 1}(F_{vis}^{l})$　(2)

$\tilde{F}_l = X_0^{(l)} + s\cdot \mathrm{GELU}\!\left(\mathrm{DWConv}\!\left(\mathrm{GroupNorm}(X_0^{(l)})\right)\right)$　(3)

$F_{fuse} = \phi_{fuse}\!\left(\tilde{F}_1 \oplus \cdots \oplus \tilde{F}_{L-1} \oplus \tilde{F}_L\right)$　(4)

$T'_{mm} = \mathrm{Reshape}(W_{mm}\,T_{mm})$　(5)

$F_{mem} = T'_{mm} + F_{fuse} + P_{mem}$　(6)

大模型最后一层的特征太”概括”，抠不出边。所以把中间几层也捞出来，用一个小适配器加工（一开始几乎不改，慢慢学，避免训练崩）。把这些特征拼一起、再拌上带语言信息的视觉向量，做成一张”记忆图”F_mem，供后面查询检索。

3.2.2　空间-语义 Query 构造

原文

The object query should capture both target semantics and instance-level spatial identity. We therefore use the grounded box as an explicit conditioning signal during query construction. We obtain the width w, height h from the box coordinates and encode its geometry with Fourier positional encoding γ(·), where the log-scale transform improves robustness to large variations in object scale. The initial object query is then defined, and a stack of transformer decoder layers attends to the global memory and produces decoded query features. As a result, the query is jointly conditioned on target semantics and instance-level spatial cues.

翻译

object query 应当同时刻画”目标语义”和”实例级的空间身份”。因此我们在构造 query 时，把定位得到的框作为一个显式的条件信号。我们从框坐标得到宽 w、高 h，用 Fourier 位置编码 γ(·) 编码其几何（式 8），其中 log 尺度变换提升了对物体尺度大幅变化的鲁棒性。据此定义初始 object query（式 9），再由一叠 transformer 解码层对全局 memory 做注意力，产出解码后的 query 特征（式 10）。这样，query 就同时以”目标语义”和”实例级空间线索”为条件。

$B_{box} = (x_1,\ y_1,\ x_2,\ y_2)$　(7)

$E_{box} = \gamma(x_1)\oplus\gamma(y_1)\oplus\gamma(0.2\log w+0.5)\oplus\gamma(0.2\log h+0.5)$　(8)

$Q_{seg}^{(0)} = \mathrm{LayerNorm}\!\left(\mathrm{MLP}_{box}(E_{box}) + W_{seg}\,T_{seg}\right)$　(9)

$Q_{seg}^{(1)} = \mathrm{Decoder}(Q_{seg}^{(0)},\ F_{mem})$　(10)

查询向量得既知道”要找什么”（语义），又知道”大概在哪、多大”（空间）。做法是把框的坐标/宽高编码一下，和代表”要分割”的 token 拼成初始查询，再让它去”记忆图”里注意力检索。相比只靠一句话去找，一开始就带了框这个空间提示，起点更准。

3.2.3　Box 引导的高分辨率像素融合

原文

To recover boundary details that are largely absent from high-level MLLM features, we extract shallow image features through a lightweight convolutional stem. Directly fusing such shallow features may introduce substantial background clutter. We therefore use the grounded box to construct a soft spatial gate. Given the box coordinates, we enlarge the box by 15% in width and height to tolerate localization errors. When multiple boxes are available, we take the spatial maximum over their gates. The fused visual feature is progressively upsampled by a two-stage PixelShuffle module. We then fuse the gated shallow features with the upsampled visual feature. This design injects high-frequency local details into the decoder while suppressing irrelevant responses outside the grounded region.

翻译

为恢复高层 MLLM 特征中大量缺失的边界细节，我们用一个轻量卷积 stem 抽取浅层图像特征（式 11）。但直接融合这些浅层特征可能引入大量背景杂波。因此我们用定位框构造一个软空间门（式 12）：给定框坐标，我们把框的宽高各放大 15% 以容忍定位误差；当有多个框时，对它们的门取空间最大值。融合后的视觉特征由两级 PixelShuffle 模块逐步上采样（式 13），再把”门控后的浅层特征”与”上采样视觉特征”融合（式 14）。这一设计在向解码器注入高频局部细节的同时，抑制了定位区域外的无关响应。

$F_{cnn} = \mathrm{Stem}(I)$　(11)

$M(x,y) = \sigma(\alpha(x-x_1'))\cdot\sigma(\alpha(x_2'-x))\cdot\sigma(\alpha(y-y_1'))\cdot\sigma(\alpha(y_2'-y))$　(12)

$F_{up} = \mathrm{Upsample}(F_{fuse})$　(13)

$F_{pixel} = F_{up} \oplus \left(M(x,y)\odot F_{cnn}\right)$　(14)

高清细节藏在浅层特征里，但浅层特征也把背景一起带进来了。于是用框做一个”软探照灯”M(x,y)——框内亮（≈1）、框外暗（≈0），还特意把框放大 15% 防止框画歪切掉目标。用它先照一遍浅层特征只留框内细节，再和上采样特征拼起来。这样既清晰又干净。

3.2.4　迭代式掩码感知 Query 精炼

原文

A single-round mask prediction is often insufficient for thin structures or cluttered scenes. We therefore introduce a lightweight refinement step that feeds mask-aware pixel evidence back into the query. In the first pass, the decoded query generates dynamic kernels to predict an initial mask. We then use the corresponding soft mask to pool a target-aware feature, where ε=1e-6 ensures numerical stability. The pooled feature is projected and added back to the decoded query. The refined query is then used for second-pass mask prediction, and finally the mask logits are upsampled to the target resolution. In parallel, an auxiliary IoU head predicts the mask confidence from the refined query. This refinement loop establishes an explicit interaction between query prediction and pixel evidence, allowing the decoder to correct coarse initial masks and sharpen object boundaries.

翻译

对于细长结构或杂乱场景，单轮掩码预测往往不够。因此我们引入一个轻量的精炼步骤，把”掩码感知的像素证据”反馈回 query。第一遍：解码后的 query 生成动态卷积核，预测初始掩码（式 15）；随后用对应的软掩码 pool 出一个目标感知特征（式 16），其中 ε=1e-6 保证数值稳定。把该 pool 特征投影后加回解码 query（式 17）。第二遍：用精炼后的 query 做掩码预测（式 18），最后把掩码 logits 上采样到目标分辨率（式 19）。与此并行，一个辅助 IoU 头从精炼后的 query 预测掩码置信度（式 20）。这个精炼回路在”query 预测”与”像素证据”之间建立了显式交互，让解码器能够纠正粗糙的初始掩码、锐化物体边界。

$M_{logit}^{(1)} = \Psi(Q_{seg}^{(1)},\ F_{pixel})$　(15)

$F_{tar} = \dfrac{\sum_{h,w}\sigma(M_{logit}^{(1)})\odot F_{pixel}}{\sum_{h,w}\sigma(M_{logit}^{(1)}) + \epsilon}$　(16)

$Q_{seg}^{(2)} = \mathrm{LayerNorm}\!\left(Q_{seg}^{(1)} + \phi_{ref}(F_{tar})\right)$　(17)

$M_{logit}^{(2)} = \Psi(Q_{seg}^{(2)},\ F_{pixel})$　(18)

$\hat{M} = \mathrm{Interp}(M_{logit}^{(2)})$　(19)

$\hat{s}_{IoU} = W_{IoU}\,Q_{seg}^{(2)}$　(20)

抠一遍常常不够（尤其细长的、乱的）。所以先抠一版，再把”抠到的区域里的像素信息”汇总回填给查询向量，让它”看过初稿再抠第二版”，边界就更利落。顺带还预测一个”这次抠得有多好”的置信分（IoU 头）。

4　数据集与基准构建 (Dataset and Benchmark Construction)

方法讲完，接下来看支撑训练与评测的两大自建资产：训练数据集 SA1B-ORS 和评测基准 ORS-Bench。

4.1　SA1B-ORS

4.1.1　概述

原文

To support open-world referring segmentation, we construct SA1B-ORS, a large-scale composite dataset derived from 2 million raw images sampled from SA-1B. SA1B-ORS contains two complementary subsets: SA1B-CoRS (Category-oriented Referring Segmentation) and SA1B-DeRS (Descriptive Referring Segmentation), yielding 1.05M and 1.94M samples, respectively. SA1B-CoRS provides scalable category-consistent supervision, where each expression refers to a single semantic category and the target may contain one or multiple entities of that category. SA1B-DeRS complements this setting with descriptive, instance-specific supervision for scenes in which a category name alone is insufficient to disambiguate the target. Together, the two subsets provide both broad category-level coverage and fine-grained instance-level supervision. The selected examples are shown in Figure 3.

翻译

为支持开放世界指代分割，我们构建了 SA1B-ORS——一个源自 SA-1B、从 200 万张原始图像采样而来的大规模复合数据集。SA1B-ORS 含两个互补子集：SA1B-CoRS（类别导向指代分割）和 SA1B-DeRS（描述性指代分割），分别产出 105 万和 194 万条样本。SA1B-CoRS 提供可扩展的、类别一致的监督：每条表达指向单个语义类别，目标可能包含该类别的一个或多个实体。SA1B-DeRS 则用描述性的、实例特定的监督来补充——用于”仅凭类别名不足以区分目标”的场景。二者合起来，既有广泛的类别级覆盖，又有细粒度的实例级监督。样例见图 3。

SA1B-ORS = 两个子集拼起来。CoRS（105 万）按类别指、可能一对多；DeRS（194 万）按描述指、精确到一个。一个管”广度”，一个管”精度”。

4.1.2　SA1B-CoRS（类别导向）

原文

SA-1B provides large-scale masks, but its annotations are fragmented, category-agnostic, and not paired with language. SA1B-CoRS converts this raw supervision into category-oriented referring segmentation data through five stages: instance distillation, coarse mask acquisition, fine mask merging, MLLM verification, and referring caption generation.

翻译

SA-1B 提供了大规模掩码，但其标注是碎片化、类别无关、且不带语言的。SA1B-CoRS 通过五个阶段，把这些原始监督转化为类别导向的指代分割数据：实例蒸馏、粗掩码获取、细掩码合并、MLLM 校验、指代描述生成。

SA-1B 的原料问题是”掩码很多，但都是零碎的、没标类别、更没配文字”。CoRS 用五步流水线把它加工成能训练的数据。下面逐步看。

原文

Instance Distillation. We first identify reliable referable entities in each image. Since many SA-1B fragments correspond to partial regions, amorphous structures, or non-instance concepts, we begin with open-vocabulary tags from RAM++ and apply hybrid semantic filtering. A label is retained if it belongs to a curated high-priority vocabulary or can be mapped, via WordNet hypernym paths, to referable semantic roots such as person, vehicle, animal, artifact, clothing, food, and structure. The remaining labels are then verified by an open-vocabulary grounding model. We apply hierarchical non-maximum suppression to preserve specific labels among semantically overlapping concepts. This yields a visually validated entity set O = {(l_i, b_i)} (Eq. 21), where l_i and b_i denote the entity label and grounded box.

翻译

① 实例蒸馏（Instance Distillation）：先在每张图里找出可靠、可指代的实体。由于许多 SA-1B 碎片对应的是局部区域、无定形结构或非实例概念，我们先用 RAM++ 的开放词表标签，再做混合语义过滤：一个标签若属于精选的高优先词表，或能经 WordNet 上位词路径映射到”可指代的语义根”（如 person、vehicle、animal、artifact、clothing、food、structure），则予以保留；场景词、颜色、事件、身体部位等弱可指代概念被剔除。剩余标签再由开放词表定位模型校验，并用层次化非极大值抑制（NMS）在语义重叠的概念中保留更具体的标签。最终得到一个经视觉校验的实体集合 O = {(l_i, b_i)}（式 21），其中 l_i、b_i 分别为实体标签和定位框。

$\mathcal{O} = \{(l_i,\ b_i)\},\quad i = 1\ldots N$　(21)

第一步是”筛靠谱的目标”。先给图打一堆标签，然后过滤掉那些不好指的（颜色、天空、事件之类），只留”人/车/动物/物件”这种能指的实体，再让定位模型确认位置。产出一批”标签+框”。

原文

Coarse Mask Acquisition. For each label-image pair, Qwen3-VL-Plus predicts a grounded box and SAM2 produces a coarse mask, thereby converting box-level grounding into a mask-level entity hypothesis. Fine Mask Merging. SA-1B often splits a single semantic entity into multiple partial masks. To recover an entity-level mask, we merge SA-1B fragments under the guidance of the coarse mask, computing IoF (Eq. 22) rather than IoU because the key question is whether a small SA-1B fragment belongs to the target entity. The merged result is further refined by hole filling, morphological cleaning, and connected-component filtering. If refinement causes a substantial area drop, we fall back to the coarse mask.

翻译

② 粗掩码获取（Coarse Mask Acquisition）：对每个”标签-图像”对，用 Qwen3-VL-Plus 预测定位框、SAM2 产出粗掩码，从而把框级定位转化为掩码级的实体假设。

③ 细掩码合并（Fine Mask Merging）：SA-1B 常把单个语义实体拆成多个局部掩码。为恢复实体级掩码，我们在粗掩码的引导下合并 SA-1B 碎片——计算的是 IoF（式 22）而非 IoU，因为关键问题是”某个小碎片是否属于目标实体”。合并结果再经填洞、形态学清理、连通域过滤精修；若精修导致面积大幅下降，则回退到粗掩码。

$\mathrm{IoF}(m_p,\ m_f^{k}) = \dfrac{|m_p \cap m_f^{k}|}{|m_f^{k}|}$　(22)

第二步先用”Qwen 出框 + SAM2 出掩码”搞一版粗轮廓。第三步把 SA-1B 里属于同一物体的碎片拼回去（用 IoF 判断”这碎片是不是这物体的一部分”），再做清理。拼坏了就退回粗版，保底。

原文

MLLM Verification. Although mask merging substantially improves the raw SA-1B fragments, the resulting masks may still contain category mismatch, insufficient coverage, contamination, fragmentation, or omission. We therefore introduce an MLLM-based verification stage. For each candidate entity, we construct a verification triplet consisting of the original image, a mask-rendered image in which the candidate region is highlighted in blue, and the corresponding candidate label. This triplet is fed into Qwen3-VL-Plus, which assesses whether the mask is sufficiently reliable to serve as supervision. The criterion is designed for training usability rather than perfect pixel-level completeness. Only verified candidates are retained. Referring Caption Generation. For each verified annotation, we use its validated mask, associated label, and grounded box to generate a referring expression. After the full pipeline, SA1B-CoRS contains 1.05M training samples.

翻译

④ MLLM 校验（MLLM Verification）：尽管掩码合并大幅改善了原始 SA-1B 碎片，结果掩码仍可能有类别不匹配、覆盖不足、污染、碎裂或遗漏。因此我们引入一个基于 MLLM 的校验阶段：为每个候选实体构造一个”校验三元组”——原图、把候选区域用蓝色高亮的掩码渲染图、以及对应的候选标签，一起喂给 Qwen3-VL-Plus，让它评估该掩码是否足够可靠、可作监督。判据面向”训练可用性”而非完美的像素级完整性（允许轻微边界误差，但会拒绝大面积遗漏、混入明显非目标、严重碎裂等）。只有通过校验的候选才保留。

⑤ 指代描述生成（Referring Caption Generation）：对每条通过校验的标注，用其掩码、标签和定位框生成一条指代表达。走完整条流水线后，SA1B-CoRS 含 105 万条训练样本。

第四步找大模型当”质检员”——把原图、蓝色高亮的抠图、标签摆给它看，问”这抠得能用吗？“不合格的踢掉。第五步给合格的配上一句指代语（如”图中的船”）。至此，CoRS 105 万条齐活。

4.1.3　SA1B-DeRS（描述性）

原文

To address complex scenarios where category names alone are insufficient for unique instance identification, we introduce SA1B-DeRS. The construction process consists of three sequential stages: instruction curation, cognitive verification and saliency selection. Instruction Curation. We feed the original image, a version with the target mask overlaid, and the instance coordinates into Qwen3-VL-Plus, prompting it to generate descriptive instructions from five key perspectives: category name, target attributes, target state, relative position, and contextual relations. Cognitive Verification. The original image and the generated instruction are fed back into Qwen3-VL-Plus, which localizes the target and outputs predicted box coordinates. We compute the IoU between predicted and ground-truth boxes; samples with IoU below 0.8 are discarded. Saliency Selection. Finally, we apply a saliency-based filter: instances with excessively small area ratios are discarded, retaining only salient objects.

翻译

为应对”仅凭类别名无法唯一确定实例”的复杂场景，我们提出 SA1B-DeRS。其构建含三个顺序阶段：指令生成、认知校验、显著性筛选。① 指令生成（Instruction Curation）：把原图、叠加了目标掩码的版本、以及实例坐标喂给 Qwen3-VL-Plus，提示它从五个关键视角生成描述性指令——类别名、目标属性、目标状态、相对位置、上下文关系。

② 认知校验（Cognitive Verification）：把原图和生成的描述再喂回 Qwen3-VL-Plus，让它定位目标并输出预测框坐标；计算预测框与真值框的 IoU，IoU 低于 0.8 的样本丢弃。

③ 显著性筛选（Saliency Selection）：最后做基于显著性的过滤——面积占比过小的实例（往往是碎片、琐碎或噪声）被丢弃，只保留显著物体。

DeRS 专造”精确描述”数据，三步：① 让大模型从属性/状态/位置/关系等角度写一句独一无二的描述；② 反向验证——拿这句描述再让它去找，找不准（IoU<0.8）就说明描述有歧义，扔掉；③ 太小/太碎的目标也扔掉。这样留下的都是”描述精准 + 目标显眼”的高质量样本。

4.1.4　SA1B-ORS 的分布

原文

We conduct a multi-dimensional statistical analysis. Our dataset significantly surpasses the RefCOCO series and LVIS in both category count and sample size. The top-20 most frequent categories (e.g., person, building, sky) show a long-tail distribution consistent with open-world scenarios. Notably, descriptive instructions account for 64.8% of the data; this prevalence is intentionally designed to enhance the model’s ability to interpret complex referring expressions. A word cloud highlights the frequent use of spatial prepositions (e.g., in, above, left) and attribute adjectives (e.g., blue, dark, far), providing discriminative descriptions.

翻译

我们做了多维统计分析（见图 6）。在类别数和样本量上，本数据集都显著超过 RefCOCO 系列和 LVIS。前 20 高频类别（如 person、building、sky）呈现出与开放世界场景一致的长尾分布。值得注意的是，描述性指令占数据的 64.8%——这一高占比是刻意设计的，用以增强模型解读复杂指代表达的能力。词云显示，描述性指令高频使用空间介词（如 in、above、left）和属性形容词（如 blue、dark、far），提供了区分性的描述。

统计结论：这数据集比经典的 RefCOCO/LVIS 更大更杂，类别呈长尾（少数类超多、多数类很少，跟真实世界一样）。特意让”描述性指令”占到 64.8%，就是要逼模型学会读懂复杂长句；这些句子里满是”在上面/左边""蓝色的/暗的”这类区分性词汇。

4.2　ORS-Bench

4.2.1　概述

原文

To comprehensively evaluate model performance on the open-world referring segmentation task, we construct a high-quality evaluation suite, ORS-Bench, which comprises ORS-ID-Bench and ORS-OOD-Bench. These two subsets are designed to assess the model’s generalization ability under in-distribution (ID) and out-of-distribution (OOD) settings, respectively.

翻译

为全面评测模型在开放世界指代分割任务上的表现，我们构建了一个高质量评测套件 ORS-Bench，含 ORS-ID-Bench 和 ORS-OOD-Bench 两部分。这两个子集分别用于评估模型在分布内（ID）与分布外（OOD）设定下的泛化能力。

考卷分两套——ID 考”常规题”，OOD 考”超纲刁钻题”。

4.2.2　ORS-ID-Bench：分布内基准

原文

We first develop an in-distribution benchmark derived from the identical data construction pipeline used during training. It comprises 9,055 high-quality samples distributed across four instruction types: single-instance category instructions (2,465), multiple-instance category instructions (1,823), phrasal instructions (2,946), and descriptive instructions (1,821). The category-based instructions are sourced from COCO and LVIS, complemented by our self-constructed SA1B-CoRS. Phrasal instructions are curated from the validation splits of RefCOCO, RefCOCO+, and RefCOCOg, while descriptive instructions originate from our custom-built DeRS. Each sample has undergone rigorous manual verification.

翻译

我们先构建一个分布内基准，它源自与训练相同的数据构建流水线，含 9,055 条高质量样本，分布在四种指令类型上：单实例类别指令（2,465）、多实例类别指令（1,823）、短语指令（2,946）、描述性指令（1,821）。类别类指令来自 COCO 和 LVIS，并由自建的 SA1B-CoRS 补充；短语指令整理自 RefCOCO、RefCOCO+、RefCOCOg 的验证集；描述性指令来自自建的 DeRS。每条样本都经过了严格的人工校验。

ID 卷 9055 题，四种题型（按单个类别指、按多个类别指、短语指、长描述指）都覆盖，且都人工核对过，保证跟训练风格一致、答案靠谱。

4.2.3　ORS-OOD-Bench：分布外基准

原文

To further explore the application scope, capability boundaries, and reliability of our model under risk-sensitive scenarios, we construct an out-of-distribution evaluation set, ORS-OOD-Bench (Figure 7). This benchmark comprises data types that are either entirely absent or extremely rare in the training set. We first sample data from the training set and employ an MLLM to annotate attributes across multiple dimensions, thereby characterizing the training data distribution. We focus on six key dimensions: target instance category, instance-to-image area ratio, descriptive instruction style, occlusion level, image lighting intensity, and risk-sensitivity. We curate 200 samples for each of the six OOD categories, which undergo rigorous manual verification.

翻译

为进一步探索模型的应用范围、能力边界，以及在风险敏感场景下的可靠性，我们构建了分布外评测集 ORS-OOD-Bench（见图 7）。该基准包含在训练集中”完全没有”或”极其罕见”的数据类型。我们先从训练集采样、并用 MLLM 在多个维度上标注属性，从而刻画训练数据的分布；聚焦六个关键维度：目标实例类别、实例占图面积比、描述性指令风格、遮挡程度、图像光照强度、以及风险敏感性。我们为六类 OOD 各整理 200 个样本，并经严格人工校验。

OOD 卷专挑训练时没见过或极少见的情况，从六个方向”找茬”。每类 200 题，人工核对。具体六类如下：

原文

• Category OOD: categories absent or with frequency lower than 1e-4. • Area OOD: instances with extremely small (area ratio < 0.002) or extremely large (> 0.7) spatial extents. • Instruction OOD: five challenging styles — indirect descriptions, negative/exclusionary constraints, interrogative queries, complex spatial/relational reasoning, and lengthy descriptions. • Occlusion OOD: targets with high occlusion ratios. • Lighting OOD: images under low-light or nighttime conditions. • Risk-Sensitive Scene: domains such as autonomous driving and medical diagnosis, virtually absent from training data.

翻译

• 类别 OOD：训练集中缺失、或频率低于 1e-4 的类别。

• 面积 OOD：空间尺度极小（面积比 < 0.002）或极大（> 0.7）的实例。

• 指令 OOD：五种挑战性风格——间接描述（禁止直接点名）、带否定/排除约束、疑问/交互式提问、复杂空间或物间关系推理、把目标埋进大段背景的长描述。

• 遮挡 OOD：高遮挡比例的目标。

• 光照 OOD：低光或夜间条件下拍摄的图像。

• 风险敏感场景：自动驾驶、医学诊断等训练数据中几乎没有的领域。

六个坑逐一说：

1. 没见过的类别；
2. 特别大或特别小的目标；
3. 故意绕着说、反着说、提问式、要推理、长篇大论的句子；
4. 被挡住大半的；
5. 暗光/夜景；
6. 自动驾驶、医疗这种高风险领域。

每一类都是专门用来”考崩”模型的。

5　实现 (Implementation)

模型和数据都齐了，这一章讲怎么把它们训到一起：两阶段训练策略、指令模板设计和关键超参数。

5.1　多阶段训练 (Multi-stage Training)

原文

We adopt a two-stage training strategy to achieve the simultaneous acquisition of robust multimodal understanding and fine-grained perception capabilities for dense prediction tasks. This progression ensures that the model’s reasoning logic and spatial awareness are optimized in a synergistic manner.

翻译

我们采用两阶段训练策略，以同时获得稳健的多模态理解能力和面向密集预测任务的细粒度感知能力。这种递进式安排，确保模型的推理逻辑与空间感知能以协同的方式被优化。

分两步练。既要保住”会理解、会推理”的老本事，又要新学”精确感知边界”的新技能，分阶段来才不会顾此失彼。

原文

Stage 1: Segmentation-Centric Adaptation. The primary objective of the first stage is to establish a strong referring segmentation capability by exploiting the grounding proficiency already present in the pretrained vision-language backbone. We adapt the language model through LoRA, together with the vision encoder and the mask decoder, under a unified instruction-following framework. Using a fixed input template, the model is trained on a mixture of public referring segmentation datasets and SA1B-ORS. During training, the model is supervised to generate structured outputs that contain bounding boxes and mask placeholders, while the mask decoder maps the corresponding mask tokens to final binary masks. This stage bridges coarse-grained grounding and fine-grained mask prediction, while the simultaneous fine-tuning of the vision encoder further improves mask-oriented visual representations.

翻译

阶段一：面向分割的适配（Segmentation-Centric Adaptation）。第一阶段的主要目标，是利用预训练视觉-语言主干中已有的定位能力，建立起强大的指代分割能力。我们通过 LoRA 适配语言模型，并连同视觉编码器和掩码解码器一起，在统一的指令跟随框架下训练。使用固定的输入模板，模型在”公开指代分割数据集 + SA1B-ORS”的混合数据上训练。训练时，模型被监督去生成包含边界框和掩码占位符的结构化输出，而掩码解码器把对应的掩码 token 映射为最终的二值掩码。这一阶段架起了”粗粒度定位”与”细粒度掩码预测”之间的桥梁；同时微调视觉编码器，进一步改善了面向掩码的视觉表示。

第一步专攻”抠图”。用 LoRA 轻量微调语言模型，同时也练视觉编码器和新加的掩码解码器。让模型学会输出”框 + 掩码占位符”这种结构化答案，解码器再把占位符变成真正的掩码。目的是把”会画框”顺势变成”会抠图”。

原文

Stage 2: Synergistic Enhancement of Perception and Understanding. The second stage aims to recover and strengthen the model’s general multimodal understanding and reasoning capabilities while maintaining referring segmentation performance. We first merge the LoRA weights from Stage 1 into the LLM backbone, and then perform full fine-tuning over the LLM backbone and the mask decoder, while keeping the vision encoder frozen. Training is conducted on a mixed dataset with a ratio of 3:1:2 for referring segmentation, general multimodal understanding, and multimodal reasoning, respectively. For the reasoning portion, we employ the original Qwen3-VL-Instruct model to generate STEM-focused distillation data via off-policy distillation.

翻译

阶段二：感知与理解的协同增强（Synergistic Enhancement of Perception and Understanding）。第二阶段旨在恢复并强化模型的通用多模态理解与推理能力，同时保持指代分割性能。我们先把阶段一的 LoRA 权重合并进 LLM 主干，然后对 LLM 主干和掩码解码器做全量微调，而保持视觉编码器冻结。训练在一个混合数据集上进行，指代分割 : 通用多模态理解 : 多模态推理的比例为 3:1:2。对于推理部分，我们用原始的 Qwen3-VL-Instruct 模型，通过离策略蒸馏（off-policy distillation）生成以 STEM 为主的蒸馏数据。

第一步练完，模型可能”偏科”（光顾抠图，通用能力退步）。第二步就来”补课找平衡”：把上一步的 LoRA 合并进主干，冻住视觉编码器、全量微调其余部分，喂”抠图:通用理解:推理 = 3:1:2”的混合数据。推理数据用原版 Qwen 蒸馏出来的 STEM 题。

5.2　指令范式与提示设计 (Instruction Paradigm and Prompt Design)

原文

To support open-vocabulary capabilities, our instruction-following paradigm strictly adheres to the Qwen3-VL context structure, utilizing the ChatML format. We define distinct task-specific templates: Category-based Segmentation: “Locate and segment every instance that belongs to the following categories <category>, report bbox coordinates and masks in JSON format.” Description-based Segmentation: “Locate and segment the object that matches the description <description> in the image, report bbox coordinates and masks in JSON format.” For each identified instance, the model predicts three key attributes: bbox_2d for normalized 2D bounding box coordinates, label for the category name or identifier, and mask for a specific placeholder (e.g., <mask_start>``<mask_token>``<mask_end>). This placeholder serves as the bridge to the mask decoder.

翻译

为支持开放词表能力，我们的指令跟随范式严格遵循 Qwen3-VL 的上下文结构，采用 ChatML 格式。我们定义了不同的任务专用模板：基于类别的分割：“Locate and segment every instance that belongs to the following categories <category>, report bbox coordinates and masks in JSON format.”；基于描述的分割：“Locate and segment the object that matches the description <description> in the image, report bbox coordinates and masks in JSON format.”。对每个识别出的实例，模型预测三个关键属性：bbox_2d（归一化的 2D 边界框坐标）、label（类别名或标识符）、mask（一个特定占位符，如 <mask_start><mask_token><mask_end>）。这个占位符充当了通往掩码解码器的桥梁。

给模型的话有固定格式（ChatML）。两种任务两个模板：按类别指、按描述指。它回答时吐 JSON，每个物体给三样：框坐标 bbox_2d、标签 label、还有一个掩码占位符——占位符本身不是图，而是个”信号”，交给解码器去生成真正的掩码。

原文

The following block illustrates the input and output template used during training and inference for a category-based instruction, following the Qwen3-VL protocol.

翻译

下面这段展示了在类别指令下、训练与推理所用的输入-输出模板（遵循 Qwen3-VL 协议）：

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
<|vision_start|><|image_pad|><|vision_end|>
Locate and segment every instance that belongs to the following categories
"dog", report bbox coordinates and masks in JSON format.<|im_end|>
<|im_start|>assistant
[
  {
    "bbox_2d": [210, 45, 890, 520],
    "label": "dog",
    "mask": "<mask_start><mask_token><mask_end>"
  }
]
<|im_end|>
<|endoftext|>

就是一段”对话剧本”——系统说你是助手，用户贴图并说”把狗都框出来并抠图、用 JSON 回”，助手就回一段 JSON，里面有框、标签、和掩码占位符。训练和推理都照这个格式走。

5.3　实现细节 (Implementation Details)

原文

We utilize Qwen3-VL-4B as our foundational backbone. The newly introduced mask decoder is lightweight, containing only 17M parameters (about 0.4% of the base MLLM). In the first stage, we employ LoRA with a rank of 32 for 10,000 iterations. During this phase, the original LLM weights are frozen, while the LoRA adapters, the vision encoder, and the mask decoder remain trainable. We employ an initial learning rate of 1e-4 with a cosine annealing schedule. The vision encoder is assigned a 0.01x differential learning rate. The model is optimized end-to-end using a hybrid objective that combines text generation loss with a segmentation loss (a weighted sum of per-pixel BCE loss and DICE loss). In the second stage, the LoRA weights are merged into the LLM backbone, followed by full fine-tuning for 5,000 iterations, jointly updating the LLM backbone and the mask decoder while the vision encoder remains frozen, using a conservative learning rate of 7e-7 with cosine decay.

翻译

我们以 Qwen3-VL-4B 作为基础主干。新引入的掩码解码器很轻量，仅含 1700 万参数（约为基座 MLLM 的 0.4%）。第一阶段：采用 rank=32 的 LoRA，训练 10,000 步；此阶段原始 LLM 权重被冻结，而 LoRA 适配器、视觉编码器、掩码解码器保持可训练。初始学习率为 1e-4，配余弦退火调度；视觉编码器被赋予 0.01× 的差分学习率（以在做分割适配的同时，保护预训练特征流形不被过度优化破坏）。模型以端到端方式优化，采用”文本生成损失 + 分割损失”的混合目标，其中分割损失是逐像素 BCE 损失与 DICE 损失的加权和。第二阶段：把 LoRA 权重合并进 LLM 主干，再做 5,000 步全量微调，联合更新 LLM 主干和掩码解码器、冻结视觉编码器，采用保守的学习率 7e-7 配余弦衰减。

关键配置一览——底座 Qwen3-VL-4B，新模块只占 0.4%。第一阶段：LoRA(rank32) 练 1 万步，视觉编码器用超小学习率（怕练坏），损失=文字损失+抠图损失(BCE+DICE)。第二阶段：合并后全量微调 5 千步，学习率压到很低(7e-7)做精修。想复现照着抄这些数就行。

6　实验 (Experiments)

到了验证环节：先交代设置，随后依次看闭集分割、定位、开放世界、OOD、通用能力和消融六组实验。

6.1　实验设置 (Experimental Setup)

原文

Datasets. We first train on the RefCOCO series for RES and REC tasks. We then further train the model for open-world referring segmentation using a mixed training set consisting of two public datasets, LVIS and COCO, together with our constructed SA1B-ORS. For evaluation, we use the validation/test splits of the RefCOCO series for RES and REC, and ORS-Bench for open-world referring segmentation. Evaluation Metrics. For referring segmentation tasks, we use mIoU (the mean of per-sample IoU), cIoU (the overall IoU computed across the entire dataset), and precision@t (the percentage of samples whose mask IoU exceeds a threshold t ∈ {0.5, 0.7, 0.9}). For REC, a prediction is considered correct if its IoU with the ground-truth box exceeds 0.5 (Prec@0.5). For multiple-instance benchmarks, we compute a pairwise IoU matrix and use the Hungarian algorithm to obtain the optimal one-to-one assignment.

翻译

数据集：我们先在 RefCOCO 系列上训练 RES 和 REC 任务；随后用”两个公开数据集 LVIS、COCO + 自建的 SA1B-ORS”组成的混合训练集，进一步训练开放世界指代分割。评测时，RES/REC 用 RefCOCO 系列的验证/测试集，开放世界指代分割用 ORS-Bench。评测指标：对指代分割任务，用 mIoU（各样本 IoU 的均值）、cIoU（在整个数据集上整体计算的 IoU）、以及 precision@t（掩码 IoU 超过阈值 t ∈ {0.5, 0.7, 0.9} 的样本占比）；对 REC，若预测框与真值框的 IoU 超过 0.5 即算正确（Prec@0.5）。对多实例基准，先算成对 IoU 矩阵，再用匈牙利算法求最优一对一匹配。

先学基础（RefCOCO），再加大数据练开放世界。打分用三把尺子：mIoU（平均像素重合）、cIoU（整体像素重合）、precision@t（有多少比例抠得足够准，0.5 一般、0.9 很严）。REC 看框对不对（IoU>0.5 算对）。多个目标时用匈牙利算法把预测和答案一一配对再打分。

6.2　指代表达分割 (Referring Expression Segmentation)

表 1｜RefCOCO(+/g) 上的指代表达分割结果（cIoU）（对比方法均为基于 MLLM 的方法，最佳加粗）

原文

As shown in Table 1, our model achieves the best results on 6 of the 8 evaluation splits, reaching 82.3/83.7/79.1 on RefCOCO, 76.2/80.2/70.8 on RefCOCO+, and 78.2/78.1 on RefCOCOg. Compared with SAM-based methods, Qwen3-VL-Seg consistently delivers higher mask quality without relying on an external segmentation model. For example, it surpasses LISA by 7.4 points on RefCOCO Val and 12.7 points on RefCOCO+ TestB. The comparison with SAM3 is also notable, suggesting that for referring expression segmentation, MLLM-based grounding and language understanding remain more decisive than the generic concept-segmentation capability of pure segmentation foundation models. Qwen3-VL-Seg also outperforms Unipixel (which also adopts a Qwen-family backbone) and remains more accurate and substantially more parameter-efficient than 8B models such as UFO and Text4Seg.

翻译

如表 1 所示，我们的模型在 8 个评测 split 中的 6 个取得最佳，在 RefCOCO 达到 82.3/83.7/79.1，RefCOCO+ 达到 76.2/80.2/70.8，RefCOCOg 达到 78.2/78.1。与基于 SAM 的方法相比，Qwen3-VL-Seg 在不依赖外部分割模型的情况下，持续给出更高的掩码质量——例如在 RefCOCO Val 上超过 LISA 7.4 分、在 RefCOCO+ TestB 上超过 12.7 分。与 SAM3 的对比也很值得注意：这表明对指代表达分割而言，基于 MLLM 的定位与语言理解，比纯分割基础模型那种通用的”概念分割”能力更具决定性。Qwen3-VL-Seg 还超过了同样用 Qwen 系列主干的 Unipixel，并且比 UFO、Text4Seg 等 8B 模型更准、且参数效率高得多。

8 项里赢 6 项。关键结论有两条：① 不挂 SAM 反而抠得更好（比外挂派的 LISA 高一大截）；② 打赢 SAM3 说明——想按话精确抠图，“懂语言、会定位”比”单纯会分割”更重要。而且它才 4B，却把一堆 8B 模型比下去了，性价比高。

6.3　指代表达理解 (Referring Expression Comprehension)

表 2｜RefCOCO(+/g) 上的指代表达理解结果（Prec@0.5）（分三组：视觉通才模型 / 通用 MLLM / 基于 MLLM 的 RES 方法）

原文

As summarized in Table 2, Qwen3-VL-Seg achieves state-of-the-art performance among all MLLM-based RES methods while remaining highly competitive against specialized Vision Generalist Models and large-scale MLLMs. Notably, Qwen3-VL-Seg shows a significant performance leap over its foundational backbone, Qwen3-VL, across all benchmarks. While the backbone provides the essential grounding capability, our results indicate that the referring segmentation task acts as a powerful dense supervisor that further sharpens spatial accuracy — such as the increase from 90.7% to 92.7% on RefCOCO Val and a substantial 6.6% jump on RefCOCO+ TestB.

翻译

如表 2 所示，Qwen3-VL-Seg 在所有基于 MLLM 的 RES 方法中达到 SOTA，同时与专门的视觉通才模型和大规模 MLLM 相比也极具竞争力。值得注意的是，Qwen3-VL-Seg 相比其基座 Qwen3-VL，在所有基准上都有显著跃升。虽然基座提供了必要的定位能力，但我们的结果表明：指代分割任务充当了一个强大的”密集监督者”，进一步锐化了空间精度——例如 RefCOCO Val 从 90.7% 提升到 92.7%，RefCOCO+ TestB 更是大涨 6.6%。

这张表看”框准不准”。结论很有意思：练了”精确抠图”之后，连”画框”都变准了（比自己的基座 Qwen3-VL 明显提升）。说明抠图这个更严格的训练，反过来把定位能力也带强了——一举两得。

6.4　开放世界指代分割 (Open-world Referring Segmentation)

表 3｜自建 ORS-ID-Bench 上的开放世界指代分割结果（cIoU 与 precision@0.5）（”-“表示该方法不支持多实例分割）

原文

Table 3 reports results on ORS-ID-Bench under four instruction formats. Qwen3-VL-Seg achieves the best performance in 7 out of 8 metrics. In category-oriented settings, SAM3 is the strongest competing baseline; Qwen3-VL-Seg outperforms SAM3 on all single-instance metrics and attains the best P@0.5 in the multiple-instance setting, while trailing slightly in multiple-instance cIoU by 1.4 points. The more decisive advantage of Qwen3-VL-Seg appears in language-intensive settings. On phrasal instructions, our model surpasses the best baseline by 19.0 cIoU points and 13.6 P@0.5 points. On descriptive instructions, it further improves over the strongest baseline by 15.5 cIoU points and 13.1 P@0.5 points. In contrast, both general-purpose MLLMs and segmentation-centric models exhibit a pronounced performance drop once the target can no longer be specified by a simple category name alone.

翻译

表 3 报告了 ORS-ID-Bench 在四种指令格式下的结果。Qwen3-VL-Seg 在 8 个指标中的 7 个取得最佳。在类别导向设定下，SAM3 是最强的竞争基线；Qwen3-VL-Seg 在所有单实例指标上超过 SAM3，并在多实例设定下取得最佳 P@0.5，仅在多实例 cIoU 上略微落后 1.4 分。Qwen3-VL-Seg 更具决定性的优势出现在语言密集型设定：在短语指令上，我们的模型超过最佳基线 19.0 个 cIoU 点、13.6 个 P@0.5 点；在描述性指令上，进一步超过最强基线 15.5 个 cIoU 点、13.1 个 P@0.5 点。相比之下，一旦目标无法仅用简单的类别名指定，通用 MLLM 和分割中心模型都会出现明显的性能下滑。

原文

Qualitative comparisons (Figures 8, 9, 10) compare three paradigms: general-purpose MLLMs (Gemini-2.5-pro, Gemini-3-flash, Seed-2.0 Pro), unified generalist models (UFO, Youtu-VL), and the segmentation foundation model SAM3. While general-purpose MLLMs demonstrate reliable semantic localization, their predicted masks often lack pixel-level resolution. Unified generalist and specialized segmentation frameworks produce higher-quality masks but may exhibit inconsistencies in instance coverage or semantic grounding. Qwen3-VL-Seg combines the grounding capability of a strong vision-language backbone with the boundary recovery capacity of the box-guided mask decoder, so its masks remain semantically aligned while preserving accurate object geometry. On phrasal and descriptive instructions, our model excels at precise spatial reasoning and fine-grained part-level grounding, resolving complex instructions such as “second bottle from the bottom” or the “elevated water cannon” of a fire truck.

翻译

定性对比（图 8、9、10）比较了三种范式：通用 MLLM（Gemini-2.5-pro、Gemini-3-flash、Seed-2.0 Pro）、统一通才模型（UFO、Youtu-VL）、以及分割基础模型 SAM3。通用 MLLM 语义定位可靠，但预测掩码常缺乏像素级分辨率；统一通才与专门分割框架掩码质量更高，但在实例覆盖或语义定位上可能不一致。Qwen3-VL-Seg 把强大视觉-语言主干的定位能力，与 box-guided 掩码解码器的边界恢复能力结合，因此掩码既在语义上对齐、又保持了准确的物体几何。在短语和描述性指令上，我们的模型擅长精确的空间推理与细粒度的部件级定位，能解决”从下往上第二个瓶子""消防车升起的水炮”这类复杂指令。

看图对比三派。通用大模型”找得对但抠得糊”；专业分割”抠得清但有时找错/漏”；本文两头兼顾——又找得对又抠得清，尤其能听懂”从下数第二个""消防车那根水炮”这种刁钻描述并精确抠出来。

6.5　分布外指代分割 (OOD Referring Segmentation)

原文

To evaluate the generalization capability of Qwen3-VL-Seg in challenging scenarios, we conduct a comprehensive assessment on ORS-OOD-Bench (Figure 11). Qwen3-VL-Seg achieves cIoU scores of 53.49%, 59.30%, 86.22%, 78.9%, 83.45%, and 8.64% across six categories of OOD scenarios (category, area, instruction, lighting, occlusion, and risk-sensitive), significantly outperforming other frontier MLLMs and unified generalist perception models. Despite outperforming current SOTA methods in OOD settings, all methods exhibit a notable performance drop compared to ORS-ID-Bench. This decline is particularly pronounced in risk-sensitive scenes, such as autonomous driving and medical diagnosis, where nearly all methods failed to achieve accurate identification and segmentation. This highlights that future referring segmentation models must further enhance their out-of-distribution generalization capabilities.

翻译

为评估 Qwen3-VL-Seg 在挑战场景下的泛化能力，我们在 ORS-OOD-Bench 上做了全面评测（见图 11）。在六类 OOD 场景（类别、面积、指令、光照、遮挡、风险敏感）上，Qwen3-VL-Seg 分别取得 cIoU 53.49%、59.30%、86.22%、78.9%、83.45%、8.64%，显著优于其他前沿 MLLM 和统一通才感知模型。尽管在 OOD 设定下超越了当前 SOTA，所有方法相比 ORS-ID-Bench 都出现明显下滑；这一下滑在风险敏感场景（如自动驾驶、医学诊断）尤为剧烈——几乎所有方法都无法实现准确的识别与分割。这凸显出：未来的指代分割模型必须进一步增强分布外泛化能力。

6.6　通用多模态评测 (General Multimodal Evaluation)

表 4｜Qwen3-VL-Seg 与现有 MLLM 在多种通用多模态基准上的对比（S-1=阶段一变体，S-2=最终模型；”–“表示无数据）

原文

Table 4 reveals a stage-wise trend. After the segmentation-centric adaptation of Stage 1, the model improves on perception-oriented tasks such as MMBench-EN and RefCOCO, but exhibits broader regressions on several reasoning- and OCR-intensive benchmarks, indicating that dense segmentation supervision shifts the model toward stronger spatial specialization. The second stage largely restores this balance. By reintroducing general multimodal understanding and reasoning data while keeping the vision encoder frozen, the final Qwen3-VL-Seg model recovers markedly over the stage-1 variant, remaining particularly strong on OCR- and grounding-related tasks. Overall, these results suggest that the proposed two-stage recipe yields a favorable rebalancing of model capabilities.

翻译

表 4 揭示了一个”阶段性”趋势。经过阶段一的面向分割适配后，模型在感知类任务（如 MMBench-EN、RefCOCO）上有所提升，但在若干推理和 OCR 密集型基准上出现较普遍的回退——表明密集分割监督把模型推向了更强的空间专门化。第二阶段则在很大程度上恢复了这一平衡：通过重新引入通用多模态理解与推理数据、同时冻结视觉编码器，最终的 Qwen3-VL-Seg 相比阶段一变体明显回升，并在 OCR 与定位相关任务上尤为强劲。总体而言，这些结果表明所提的两阶段配方，实现了模型能力的良性再平衡。

这张表回答”练抠图会不会把脑子练傻？“答案：阶段一确实有点偏科（感知强了、推理/OCR 退了）；但阶段二补课后基本找回平衡，最终版在 OCR、定位这些任务上甚至更强了。所以两阶段设计是值得的——鱼和熊掌基本都保住。

6.7　消融实验 (Ablation Studies)

表 5｜在 RefCOCO/+/g（VAL）上的架构消融实验

原文

Compared with the Qwen box+SAM baseline, both the full model and its ablated variants achieve substantial gains, confirming the effectiveness of the box-guided decoder. Notably, the full model improves the strict P@0.9 metric from 39.6 to 50.2 on RefCOCO. Freezing the vision encoder consistently degrades performance, particularly on P@0.9, indicating that adapting the visual backbone is important. Removing multi-scale visual features leads to consistent drops in cIoU and P@0.9. Removing the shallow image branch also hurts performance across all three datasets. The performance gap is most evident on P@0.9, which is the most sensitive metric to boundary quality, showing that multi-scale feature injection and the high-resolution image branch are particularly important for improving mask precision.

翻译

与 Qwen box+SAM 基线相比，完整模型及其各消融变体都取得了大幅增益，证实了 box-guided 解码器的有效性——尤其是完整模型把严格指标 P@0.9 在 RefCOCO 上从 39.6 提升到 50.2。冻结视觉编码器会持续损害性能（尤其在 P@0.9 上），说明适配视觉主干很重要；去掉多尺度视觉特征会使 cIoU 和 P@0.9 一致下降；去掉浅层图像分支同样在三个数据集上都损害性能。性能差距在 P@0.9（对边界质量最敏感的指标）上最为明显，表明多尺度特征注入和高分辨率图像分支，对提升掩码精度尤为关键。

拆零件验证每个模块有没有用。结论：① 换成”Qwen 框+SAM”的土办法明显更差，说明本文解码器确实强；② 视觉编码器不练会掉分（该练）；③ 多尺度特征、浅层图像分支去掉都掉分。而且掉得最狠的都是 P@0.9（最考验边缘精细度的指标）——正好印证这两个模块就是用来”抠清楚边”的。

7　结论 (Conclusion)

原文

We present Qwen3-VL-Seg, a lightweight framework for open-world referring segmentation that extends the pretrained MLLM from box-level grounding to pixel-level mask prediction. The key idea is to treat the MLLM-predicted box as a structural prior for mask decoding, enabling precise segmentation without relying on external segmentation foundation models. To support scalable training and evaluation, we construct the SA1B-ORS training dataset, with complementary category-oriented and descriptive subsets, and introduce ORS-Bench for in-distribution and out-of-distribution assessment of open-world referring segmentation. Experiments on referring expression segmentation, visual grounding, and open-world benchmarks indicate that Qwen3-VL-Seg performs strongly across both closed-set and open-world settings, with especially clear gains on language-intensive instructions and strong out-of-distribution generalization. Additional general multimodal evaluation shows that the model broadly preserves general-purpose multimodal competence after segmentation-oriented adaptation. These findings suggest that reusing MLLM grounding outputs as structural priors provides an effective path toward unified and efficient referring segmentation.

翻译

我们提出 Qwen3-VL-Seg——一个用于开放世界指代分割的轻量框架，它把预训练 MLLM 从”框级定位”扩展到”像素级掩码预测”。关键思想是：把 MLLM 预测的框当作掩码解码的结构先验，从而在不依赖外部分割基础模型的情况下实现精确分割。为支持可扩展的训练与评测，我们构建了 SA1B-ORS 训练数据集（含互补的类别导向子集与描述性子集），并提出 ORS-Bench，用于开放世界指代分割的分布内与分布外评估。在指代表达分割、视觉定位和开放世界基准上的实验表明，Qwen3-VL-Seg 在闭集与开放世界设定下都表现强劲，在语言密集型指令上增益尤为明显、分布外泛化强。额外的通用多模态评测显示，经过面向分割的适配后，模型仍广泛保留了通用多模态能力。这些发现表明：把 MLLM 的定位输出复用为结构先验，是通向统一且高效的指代分割的一条有效路径。

全文收尾一句话——“框不是终点，是抠图的路标”这个思路被证明行得通：不外挂大模型、只加 0.4% 参数，就能又准又轻地按话抠图，还不丢通用本事。这为”统一、高效的指代分割”指了一条明路。

📌 全文速记（一页看懂）

• 问题：大模型会”按话画框”，但画框太糙；要精确抠图，以前只能外挂笨重的 SAM。

• 方法：把大模型自己画的框当”结构先验”，用一个仅 17M（0.4%）的 box-guided 解码器四步走（多尺度特征注入 → 空间语义 query → box 软门像素融合 → 迭代精炼）变成精确掩码。

• 数据：自建 SA1B-ORS（CoRS 105 万 + DeRS 194 万）；考卷 ORS-Bench（ID 9055 题 + OOD 六维各 200 题）。

• 训练：两阶段（LoRA 分割适配 → 全量微调找平衡，数据配比 3:1:2）。

• 结果：RES/REC/开放世界多数指标 SOTA，语言密集场景断层领先；通用能力基本保住；唯一软肋是自动驾驶/医疗等风险敏感 OOD（仅 8.64 cIoU），是未来方向。

本文档为 Qwen3-VL-Seg 全文精读版，原文 / 翻译 / 大白话逐段对照，含论文全部 11 张图与 5 张表。

(注：内容由 AI 生成，请谨慎参考）

（注：部分内容可能由 AI 生成）