easoning Segmentation aims to generate pixel-level binary masks based on complex natural language queries . This task requires the model to interpret text instructions through logical reasoning. Unlike traditional semantic or instance segmentation tasks that rely on pre-defined categorical labels , reasoning segmentation addresses more complex, context-dependent queries or even instructions requiring reasoning to identify the target instance (e.g., “something that the person uses to fish”). This setting better reflects the real-world requirements for intelligent assistants and robotics, while simultaneously imposing significantly higher demands on the model: it must possess fine-grained visual perception, robust image-text alignment capabilities, and broad domain knowledge combined with common sense and logical reasoning.

style="width:95.0%" />

Given the comprehensive capabilities of Multi-modal Large Language Models (MLLMs) in reasoning and multi-modal perception, incorporating MLLMs has become the mainstream solution for this task.

Current approaches for reasoning segmentation are primarily dominated by two learning-based paradigms: Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL), yet both face inherent limitations regarding training costs and rigid inference patterns. The first paradigm, SFT-based methods , attempts to internalize the segmentation task by fine-tuning MLLMs to map text instructions directly to pixel masks. However, the extensive parameter updates required often induce catastrophic forgetting of general reasoning capabilities, leading to poor generalization on out-of-distribution instructions . The second paradigm, RL-based methods , treats the MLLM as a policy network optimized via reward signals. Despite introducing dynamic interaction, these models still execute a fixed policy during inference. This results in a brittle, linear reasoning trajectory: if the initial step falls into a local optimum (e.g., misidentifying the target), the agent lacks the intrinsic mechanism to perform dynamic backtracking or global search to recover .

The emergence of SAM 3  marks a pivotal turning point, as it endows a powerful interactive segmentation model with intrinsic semantic understanding capability for the first time. Unlike predecessor models that responded only to geometric prompts like points or boxes (which was the fundamental reason previous methods required expensive training for MLLMs to output coordinates ), SAM 3 can directly comprehend and locate simple noun phrases. Based on this technological leap, works represented by the SAM 3 Agent  have rapidly emerged, establishing a new paradigm of Training-Free Visual Agents. This paradigm completely discards parameter updates, utilizing a frozen SOTA MLLM as the brain for planning and SAM 3 as the hand with a semantic interface for execution. This design maximizes the preservation of the MLLM’s general reasoning abilities and world knowledge, endowing the system with strong zero-shot transfer capabilities.

However, despite the potential shown by the SAM 3 Agent, its inference mechanism remains trapped in a Heuristic Trial-and-Error strategy, suffering from two critical robustness defects. First, the refinement steps of the SAM 3 Agent rely heavily on the MLLM’s intuition to decide. This jumping logic based on dialogue history is often random and unstructured (e.g., the model may switch disorderly between color and location descriptions). Lacking a clear gradient or optimization direction, the agent easily oscillates around suboptimal solutions, failing to converge to the prompt that triggers SAM 3’s peak performance. Secondly, although the SAM 3 Agent includes a selection mechanism, it typically filters within a set of candidates generated by a single reasoning path. This means that if the initial direction of semantic understanding deviates, all subsequent interactions are confined within this erroneous semantic frame. The system lacks a mechanism to simultaneously maintain and compare distinct heterogeneous hypotheses, leading to an inability to escape local optima.

To address these challenges, we propose EVOL-SAM3, a novel training-free framework that reframes reasoning segmentation as an inference-time evolutionary search problem. Unlike the SAM 3 Agent, which relies on linear heuristic jumps, Evo-SAM3 maintains a dynamically evolving population of hypotheses. We design a comprehensive visual evolutionary algorithm: it conducts extensive directed exploration in the semantic space via semantic mutation operators and introduces a heterogeneous competition mechanism, forcing “text-based reasoning” and “box-based localization” to compete within the same ecological niche to resolve ambiguity. Most crucially, we leverage the MLLM’s superior discriminative capability to construct a Visual Arena replacing simple filtering with a tournament selection mechanism. This enables Evo-SAM3 to transform the MLLM’s discriminative power into a driving force for search without any parameter updates, ultimately evolving the optimal prompt that maximizes SAM 3’s potential.

Our main contributions are summarized as follows:

• We are the first to model the reasoning segmentation task as an inference-time evolutionary search problem. We demonstrate that maintaining a diverse population of hypotheses and using evolutionary algorithms for systematic search allows for a more robust approximation of the optimal solution.

• We introduce a specialized evolutionary mechanism featuring (1) semantic mutation for directed exploration, (2) heterogeneous competition to resolve ambiguity, and (3) a visual arena utilizing tournament selection for robust optimization.

• Experiments on benchmarks demonstrate that under the Zero-Shot setting, EVOL-SAM3 not only significantly outperforms existing training-free agents but also surpasses dedicated models trained via STF or RL on multiple key metrics.

Related Works

Learning-based Reasoning Segmentation. Current methodologies for reasoning segmentation can be broadly categorized into three paradigms: Supervised Fine-Tuning , Reinforcement Learning , and hybrid SFT+RL strategies. SFT-based methods  typically formulate the task as a sequence generation problem. By end-to-end fine-tuning Multi-modal Large Language Models on large-scale segmentation datasets, these models are trained to output specific mask tokens or polygon coordinates. However, this paradigm faces significant limitations. The aggressive parameter updates required for pixel-level alignment often induce catastrophic forgetting, degrading the MLLM’s inherent general reasoning capabilities. Furthermore, SFT models tend to overfit the training distribution, resulting in poor generalization when encountering complex, out-of-distribution instructions in open-world scenarios.

RL and Hybrid Strategies. To align the optimization objective with evaluation metrics, RL-based methods formulate segmentation as a policy decision process, optimizing the model via reward signals. However, existing RL methods often fail to achieve fully end-to-end gradient flow between the MLLM and SAM, or heavily rely on manually constructed trajectory data for initialization . This black-box optimization often yields suboptimal prompts that fail to trigger the executor’s peak performance. Recent works attempt to combine the strengths of both worlds through hybrid strategies. A representative method, LENS , jointly optimizes the Chain-of-Thought (CoT) and mask generation in an end-to-end manner by employing Group Relative Policy Optimization (GRPO) with rewards spanning sentence, box, and segment levels. Despite these architectural advancements, all learning-based paradigms—whether SFT, RL, or hybrid—share a critical bottleneck: excessive reliance on training resources. Developing such models requires substantial computational resources and massive annotated datasets, creating an extremely high barrier to entry. Moreover, once trained, the inference policy remains static, lacking the flexibility to adaptively self-correct without further parameter updates.

Training-Free Visual Agents. In response to the high training barriers, a Training-Free paradigm has emerged, represented by the SAM 3 Agent . This approach links a frozen MLLM as a planner with SAM 3 for execution, thereby preserving the MLLM’s inherent generalization capabilities. Nevertheless, current training-free agents rely heavily on heuristic trial-and-error. Their reasoning process is typically linear and unstructured, lacking a systematic optimization objective. If the agent’s initial semantic interpretation is flawed, it lacks the mechanism to globally explore alternative hypotheses, often leading to irreversible failures in complex scenes.To address these limitations, we propose EVOL-SAM3, which reformulates reasoning segmentation as an inference-time evolutionary search problem. Unlike linear approaches, our framework maintains a population of diverse hypotheses and leverages the discriminative power of MLLMs to perform global optimization, enabling robust alignment without any parameter updates.

style="width:100.0%" />

Method

Problem Formulation

We formally define the reasoning segmentation task as follows: given an input image $`I \in \mathbb{R}^{H \times W \times 3}`$ and a natural language query $`q`$, the objective is to generate a binary mask $`M \in \{0, 1\}^{H \times W}`$ that semantically aligns optimally with the target described by $`q`$. Traditional approaches, such as fully supervised fine-tuning or reinforcement learning, typically aim to learn a parameterized conditional distribution $`P_\theta(M | I, q)`$. However, this paradigm is inherently static during inference, relying heavily on the model’s memorization of the training distribution.

In this work, we reformulate this task as an inference-time latent variable optimization problem. We posit that for any arbitrarily complex query $`q`$, there exists an optimal latent prompt $`z^*`$ within a hybrid semantic-geometric space $`\mathcal{Z}`$. When this optimal prompt drives a powerful, deterministic segmentation executor, it yields the optimal solution. Based on this premise, we define our objective as searching for $`z^*`$ within the space of frozen foundation models:

\begin{equation}
    z^* = \mathop{\arg\max}_{z \in \mathcal{Z}} \mathcal{F}(M_z, q; \Psi_{\text{VLM}}) \quad \text{s.t.} \quad M_z = \Phi_{\text{SAM}}(I, z)
    \label{eq:optimization_objective}
\end{equation}

In this formulation, $`\Phi_{\text{SAM}}: \mathbb{R}^{H \times W \times 3} \times \mathcal{Z} \to \{0, 1\}^{H \times W}`$ denotes the frozen Executor (implemented via SAM 3 ), which deterministically maps the latent variable $`z`$ to a segmentation mask $`M_z`$. $`\mathcal{F}`$ represents the Fitness Function, parameterized by a frozen Vision-Language Model $`\Psi_{\text{VLM}}`$ (implemented via Qwen2.5-VL ), which quantifies the semantic alignment between $`M_z`$ and the original query $`q`$. The core advantage of this modeling lies in decoupling: it leverages the MLLM’s reasoning capabilities to navigate the complex semantic space $`\mathcal{Z}`$, while retaining SAM 3’s robust geometric priors for pixel-level generation. Crucially, this avoids the risk of catastrophic forgetting associated with fine-tuning foundation models.

Solving the optimization problem in Eq. [eq:optimization_objective] presents two intrinsic mathematical challenges: non-differentiability and reference-free evaluation. First, since the search space $`\mathcal{Z}`$ consists of discrete natural language tokens and geometric coordinates, and $`\Phi_{\text{SAM}}`$ operates as a black-box function, the gradient $`\nabla_z \mathcal{F}`$ is inaccessible, precluding gradient-based optimization methods. Second, the absence of ground truth masks during inference makes the precise calculation of $`\mathcal{F}`$ impossible.

To address these challenges, we propose the EVOL-SAM3 framework, which employs an Evolutionary Algorithm (EA) as a derivative-free solver. Specifically, we maintain a population of hypotheses $`\mathcal{P}_t = \{z_1^{(t)}, \dots, z_N^{(t)}\}`$ at generation $`t`$, and approximate $`z^*`$ by iteratively applying MLLM-driven Semantic Mutation and Selection Operators. To overcome the lack of ground truth, we exploit the property that MLLMs exhibit superior performance in discrimination tasks compared to generation tasks. We approximate the fitness landscape via a Visual Arena mechanism, where the scalar fitness $`\mathcal{F}(z)`$ is modeled as the win-rate of a candidate solution in pairwise tournaments judged by $`\Psi_{\text{VLM}}`$. In this way, we transform static pre-trained models into a dynamic system capable of deliberate search and optimization.

Framework Overview

As illustrated in Figure 2, EVOL-SAM3 reformulates the static reasoning segmentation task into a training-free, dynamic evolutionary search process. The framework leverages a frozen MLLM for high-level semantic planning and evaluation, while utilizing SAM 3 for low-level pixel execution. The pipeline initiates with Population Initialization (Phase 1), where the MLLM acts as a meta-generator to expand the user’s single query into a diverse population of prompts covering various descriptive perspectives (e.g., attributes, spatial relations). Subsequently, the system enters the core Evolutionary Reasoning Loop (Phase 2). In each generation, candidate prompts drive SAM 3 to generate segmentation masks. These masks are rigorously filtered by a Semantic Gate for consistency and then evaluated via pairwise competition in a Visual Arena. Following the principle of survival of the fittest, elite individuals undergo MLLM-driven Semantic Mutation to spawn the next generation, iteratively converging towards the optimal semantic description of the target.

To mitigate potential linguistic hallucinations in complex scenarios, we introduce a final Heterogeneous Arena (Phase 3). In this stage, the system extracts not only the best text-derived mask from the evolutionary search but also generates an auxiliary detection-based mask derived from the MLLM’s geometric intuition. These two candidates, originating from distinct reasoning paths (semantic reasoning vs. geometric localization), are submitted to a Heterogeneous Arena. Through a double-blind adaptive judgment mechanism, the MLLM selects the most robust final segmentation. This cascaded “Generate-Evolve-Arbitrate” mechanism enables Evo-SAM3 to achieve precise alignment and robust segmentation for complex visual queries without requiring any parameter updates.

Semantic Meta-Planning

The ultimate performance of evolutionary search is critically contingent on the quality and coverage of the initial population $`\mathcal{P}_0`$. In complex visual scenarios, the raw user query $`q`$ is often concise or semantically ambiguous (e.g., “the man” is ill-posed in a multi-person scene). Relying directly on such a single starting point makes the search process prone to getting trapped in local optima or failing to converge. To overcome this cold-start problem, we introduce a semantic meta-planning phase prior to the evolutionary loop. In this stage, the frozen MLLM functions as a vision-aware meta-generator. Unlike traditional blind textual augmentation or paraphrasing, this generator jointly analyzes the input image $`I`$ and query $`q`$ to deconstruct the abstract query into a set of visually grounded hypotheses. By identifying potential distractors within the image, it actively infers discriminative features capable of uniquely identifying the target, thereby extending a single query point into a distribution that covers the target’s potential semantic space.

Specifically, the meta-planning process constructs a heterogeneous population $`\mathcal{P}_0`$ through multi-dimensional semantic anchoring. The first dimension is attribute augmentation, where the MLLM automatically supplements visual details (e.g., color, texture) to distinguish the target from similar instances. The second is spatial explicitness, which converts implicit relative positional relationships into absolute spatial descriptions (e.g., refining “the person on the left” to “the leftmost person” or “the person in the bottom-left corner”). The third is referent specification, which replaces generic pronouns with specific nouns. The initial population generated through this strategy possesses high genetic diversity and eliminates referential ambiguity to a significant extent at the initialization stage. This provides a robust and high-quality search space for the subsequent evolutionary reasoning loop, significantly reducing the exploration cost of the algorithm.

Detailed Mechanisms of Evolutionary Reasoning Loop

To effectively optimize the latent prompt $`z`$ without ground truth supervision, we instantiate the evolutionary loop with three cascaded operators: Execution & Cascaded Verification, Visual Arena Selection, and Semantic Mutation. This loop is designed to align discrete semantic search with continuous pixel generation, iteratively converging towards the optimal solution.

Execution and Cascaded Verification Operator.

This step serves as the bridge between the discrete semantic space $`\mathcal{Z}`$ and the continuous pixel space, while simultaneously acting as a computational filter. For each latent prompt $`z_i`$ in the population $`\mathcal{P}_t`$, the frozen executor $`\Phi_{\text{SAM}}`$ deterministically maps it to a binary mask $`M_i`$ and an internal confidence score $`s_{conf}^{(i)} \in [0, 1]`$. This process is formulated as:

\begin{equation}
    (M_i, s_{conf}^{(i)}) = \Phi_{\text{SAM}}(I, z_i)
\end{equation}

To prune the search space of low-quality hypotheses, we employ a dual-layer gating function $`\mathcal{G}(z_i)`$. Initially, a Confidence Gate leverages the executor’s uncertainty to filter out hallucinations via an indicator function:

\begin{equation}
    g_{conf}(z_i) = \mathbb{I}(s_{conf}^{(i)} > \tau_{conf})
\end{equation}

Subsequently, candidates passing this preliminary filter are subjected to a Semantic Gate, where the VLM $`\Psi_{\text{VLM}}`$ acts as a verifier. By analyzing the joint distribution of the raw image $`I`$ and the mask overlay $`I \oplus M_i`$, it validates consistency with the query $`q`$, yielding a binary decision $`g_{sem}(z_i)`$. Only individuals satisfying both constraints are admitted into the valid population:

\begin{equation}
    \mathcal{P}_{valid} = \{z_i \in \mathcal{P}_t \mid g_{conf}(z_i) \cdot g_{sem}(z_i) = 1\}
\end{equation}

Visual Arena Selection Operator.

Approximating the fitness landscape $`\mathcal{F}(z)`$ in a reference-free setting is non-trivial. Exploiting the property that MLLMs are stronger discriminators than generators, we reformulate fitness estimation as a Pairwise Tournament. For pairs $`(z_a, z_b) \in \mathcal{P}_{valid}`$ with significant geometric discrepancy, we construct a visual triplet input. The evaluator $`\Psi_{\text{VLM}}`$ functions as a judge, estimating the preference probability $`P(z_a \succ z_b | q)`$ based on semantic alignment. This relative ranking mechanism circumvents the calibration bias inherent in absolute scoring. The Visual Arena Selection Operator approximates the fitness landscape $`\mathcal{F}(z)`$ in a reference-free setting. Exploiting the property that MLLMs are stronger discriminators than generators, we reformulate fitness estimation as a Pairwise Tournament. For pairs $`(z_a, z_b) \in \mathcal{P}_{valid}`$ with significant geometric discrepancy, we construct a visual triplet input. The evaluator $`\Psi_{\text{VLM}}`$ functions as a judge, estimating the preference probability $`P(z_a \succ z_b | q)`$ based on semantic alignment. This relative ranking mechanism circumvents the calibration bias inherent in absolute scoring. We dynamically update individual fitness scores using an asymmetric Elo-based update rule. To facilitate exploration, if the challenger $`z_b`$ defeats the incumbent $`z_a`$, we apply a boosted reward scaled by $`\beta > 1`$:

\begin{equation}
    \mathcal{F}(z_b) \leftarrow \mathcal{F}(z_b) + \beta \Delta, \quad \mathcal{F}(z_a) \leftarrow \mathcal{F}(z_a) - \beta \Delta
\end{equation}

Conversely, if the incumbent retains the advantage, a standard update step $`\Delta`$ is applied. This mechanism allows high-potential mutations to rapidly ascend in rank while maintaining population stability.

Semantic Mutation Operator.

To escape local optima and traverse the semantic manifold effectively, we introduce a Semantic Mutation operator $`\mathcal{M}`$ driven by the reasoning capabilities of the MLLM. Unlike stochastic character-level perturbations, this operator performs a logical reconstruction of the elite prompt $`z_{elite}`$ to generate a child $`z_{mut}`$ with higher “SAM-compatibility”:

\begin{equation}
    z_{mut} = \mathcal{M}(z_{elite}, q; \Psi_{\text{VLM}})
\end{equation}

Our implementation incorporates two strategies to constrain the solution space of $`\Phi_{\text{SAM}}`$: (1) Simplification, which removes non-discriminative redundancy to minimize executor attention drift; and (2) Ambiguity Avoidance, which transforms relative spatial descriptions into absolute superlatives (e.g., mutating “person on the right” into “rightmost person”). These mutations explicitly guide the search towards regions of the semantic space that yield deterministic and accurate segmentation.

Heterogeneous Arena

While the evolutionary reasoning loop (Algorithm [alg:evo_loop]) excels in semantic alignment, producing a high-quality candidate $`M_{text}`$, relying solely on textual prompts remains susceptible to linguistic hallucinations in scenes with complex textures. To establish a robust safeguard, we introduce a heterogeneous arena module. This component leverages the geometric intuition of the MLLM to construct an auxiliary spatial reasoning path, facilitating a final adjudication against $`M_{text}`$ via a double-blind mechanism.

Geometric Intuition and Spatial Filtering. Distinct from the iterative refinement of textual prompts, the geometric path focuses on capturing the global spatial localization of the target via a single forward pass. We prompt the frozen MLLM to perform a detection task, directly predicting a bounding box $`B_{pred}`$, formulated as $`B_{pred} = \Psi_{\text{VLM}}(I, q; \text{detect})`$. This box serves as a geometric prompt for SAM to generate a candidate mask $`M_{box} = \Phi_{\text{SAM}}(I, B_{pred})`$. To ensure the validity of this geometric prior, we enforce a spatial consistency filter. This filter calculates the intersection over union between the bounding box of the generated mask, denoted as $`\text{bbox}(M_{box})`$, and the MLLM-predicted box $`B_{pred}`$. The geometric mask is deemed a valid contender only if it exhibits high spatial coherence:

\begin{equation}
    \text{IoU}(\text{bbox}(M_{box}), B_{pred}) > \tau_{spat}
\end{equation}

This constraint effectively discards instances where the MLLM localizes a general region but the segmentation model fails to identify a salient object within that vicinity, ensuring the geometric reliability of $`M_{box}`$.

Double-Blind Swap Mechanism. At this stage, we obtain two heterogeneous candidates: the optimal text-derived mask $`M_{text}`$ and the box-derived mask $`M_{box}`$. To select the final segmentation $`M^*`$ without bias, we submit them to a heterogeneous arena. Acknowledging the inherent position bias in MLLMs, where models may exhibit a preference for images appearing in specific input slots, we design a double-blind swap adjudication strategy. We construct two permuted visual prompt tuples: the forward permutation $`V_{fwd} = (I, M_{text}, M_{box})`$ and the reverse permutation $`V_{rev} = (I, M_{box}, M_{text})`$. The MLLM acts as a judge $`J`$ in two independent trials. The final selection is determined by a strict consistency logic that favors the robust geometric prior in uncertain cases:

\begin{equation}
    M^* = 
    \begin{cases} 
    M_{text} & \text{if } J(V_{fwd}) = M_{text} \land J(V_{rev}) = M_{text} \\
    M_{box} & \text{otherwise (Fallback Priority)}
    \end{cases}
\end{equation}

By enforcing consistency across permutations, we mitigate the stochasticity of the VLM. Specifically, any logical conflict (e.g., the model always choosing the first image regardless of content) or preference for the box triggers the fallback to $`M_{box}`$, thereby enforcing safe performance boundaries.

Experiments

Implementation Details

We implement EVOL-SAM3 using the PyTorch framework on a server equipped with 8 NVIDIA RTX 3080 GPUs. As a training-free framework, our method focuses on inference-time optimization without updating the backbone parameters. For the Multi-modal Large Language Model (MLLM) backbone, we employ Qwen2.5-VL-Instruct, evaluating both the 3B and 7B parameter versions to verify scalability. To optimize memory efficiency while maintaining performance, the MLLM inference is conducted in FP16 precision. For the visual segmentation backend, we utilize SAM 3, with image resolution and mask generation hyperparameters strictly aligned with the SAM3Agent baseline to ensure a fair comparison. The evolutionary process is configured with a population size of $`N_{pop}=4`$ and a maximum of $`T_{max}=2`$ generations. In the Visual Arena, the Elo rating update step is set to 1.0, and the confidence threshold for mask filtering is set to $`\tau_{mask}=0.6`$. All semantic mutations and arena selections are performed in a zero-shot manner.

Datasets and Evaluation Metrics

To ensure a fair comparison with state-of-the-art methods, we strictly follow the evaluation protocols and dataset settings established in previous research. We conduct extensive experiments on four standard benchmarks. For Reasoning Segmentation, we use the ReasonSeg dataset, which contains complex queries requiring implicit reasoning, and report results on its validation set. For Referring Expression Segmentation, we evaluate on RefCOCO, RefCOCO+, and RefCOCOg. Specifically, we use the val, testA, and testB splits for RefCOCO and RefCOCO+; for RefCOCOg, we use the val (UMB) and test (UMB) partitions. Consistent with prior works, we employ gIoU and cIoU as the primary evaluation metrics.

COMPARISONS

As shown in Table [tab:main_results], we report detailed performance comparisons on the ReasonSeg benchmark. The most remarkable finding is that EVOL-SAM3 demonstrates superior performance under a fully zero-shot setting, surpassing existing learning-based methods (encompassing SFT and RL). Specifically, our 7B version achieves 70.7 gIoU on the validation set. This not only significantly outperforms the classic LISA-7B (52.9 gIoU) but, more notably, surpasses LISA-13B (65.0 gIoU), which has undergone deep optimization via supervised fine-tuning and reinforcement learning. This result prompts us to revisit the heavy reliance of high-performance reasoning segmentation on expensive supervised data or reward feedback, and demonstrates the feasibility of mining the reasoning potential of frozen models as an efficient alternative.

Furthermore, among training-free counterparts, EVOL-SAM3 proves that a refined evolutionary mechanism is more efficient than merely scaling up model parameters. Although the baseline SAM 3 Agent achieves 74.6 gIoU on the validation set by relying on a massive 72B parameter model, our method achieves a reversal on the more challenging Test Set (72.5 vs. 70.8 gIoU) using only 7B parameters. Finally, the robustness of EVOL-SAM3 in handling long and complex queries is particularly pronounced. Benefiting from the effective decomposition and error correction of complex instructions enabled by the semantic mutation and visual arena mechanisms, we achieve an outstanding 74.3 gIoU on the Test (Long) subset. This significantly outperforms RSVP (GPT-4o) at 61.9 gIoU and SAM 3 Agent (72B) at 71.0 gIoU, fully validating the unique advantages of dynamic evolutionary search in deep semantic understanding.

Table [tab:res_benchmark] presents the evaluation results on the classic RefCOCO, RefCOCO+, and RefCOCOg datasets. We observe that task-specific supervised training still yields dominant performance gains for RES tasks. As shown, fully fine-tuned methods (e.g., X-SAM, RICE, MLCD-seg) generally outperform zero-shot approaches by a large margin. This is attributed to the fact that RES tasks heavily rely on specific referring patterns (e.g., “the person on the left”), which are explicitly reinforced during supervised training, enabling SFT models to establish strong text-region mappings. Consequently, zero-shot methods, lacking this targeted pattern adaptation, naturally face a performance gap.

However, within the Zero-Shot category, EVOL-SAM3 achieves a remarkable performance breakthrough, significantly narrowing the gap with supervised methods. Compared to existing training-free baselines, EVOL-SAM3 demonstrates superior generalization. Specifically, our 7B model achieves the best zero-shot performance across all three datasets. Compared to the baseline SAM 3 Agent, which shares the identical Qwen2.5-VL-7B backbone, our method realizes a comprehensive performance leap. For instance, on the RefCOCO val and testA subsets, we achieve improvements of +9.3 cIoU ($`59.4 \to 68.7`$) and +9.4 cIoU ($`64.3 \to 73.7`$), respectively. On RefCOCO+ val, which demands strictly appearance-based grounding, the gain reaches +13.0 cIoU ($`51.4 \to 64.4`$). Similarly, on RefCOCOg val (U), which features more complex sentence structures, we obtain a significant increase of +7.5 cIoU ($`57.2 \to 64.7`$).

This significant performance improvement validates the necessity of transforming static inference into a dynamic iterative process. The results demonstrate that even in the absence of task-specific training data, introducing an adaptive optimization mechanism during the inference phase can effectively calibrate the misalignment between semantic understanding and visual perception, thereby achieving robustness and accuracy comparable to fully supervised methods in traditional referring expression segmentation tasks.

ABLATION STUDY

Evolutionary Generations

To thoroughly investigate the impact of evolutionary iterations on model performance, we conducted a sensitivity analysis on the maximum number of generations $`T_{max}`$ using the ReasonSeg validation set. As illustrated in the figure, the model exhibits a significant upward trajectory in segmentation performance as generations increase from 0 to 2, with gIoU and cIoU rising from 71.35% and 66.30% to 72.50% and 67.39%, respectively. This result strongly validates the effectiveness of our evolutionary mechanism: by introducing semantic mutation and arena selection, the model can transcend the limitations of the initial meta-planning and dynamically rectify early suboptimal solutions to approach more precise semantic descriptions. However, we observe distinct diminishing returns beyond $`T_{max}=2`$, where performance metrics saturate or even fluctuate slightly (e.g., a drop in cIoU at Gen 3). Given that the computational cost of inference increases linearly with evolutionary generations, further iterations do not yield performance gains commensurate with the additional overhead. Consequently, to achieve the optimal trade-off between reasoning accuracy and system efficiency, we adopt $`T_{max}=2`$ as the default configuration, which is sufficient for the model to achieve adequate reasoning calibration at a manageable cost.

We attribute the performance saturation and fluctuations observed beyond $`T_{max}=2`$ to two primary factors. First, the rapid convergence of the semantic search space. For most visual reasoning tasks, the number of valid logical paths from an ambiguous query to a precise referral is finite. Our experiments suggest that within two rounds of “mutation-selection” iteration, the evolutionary algorithm typically traverses and locks onto the most plausible semantic interpretation within the current context. Subsequent generations often involve minor paraphrasing rather than discovering substantive new semantic information. Second, the inherent upper bound of frozen backbones. It is crucial to note that EVOL-SAM3 is fundamentally a mining strategy rather than a learning process. It aims to elicit the maximum latent potential of the frozen MLLM and SAM but cannot surpass their intrinsic perceptual limits (e.g., recognizing extremely small or heavily occluded objects). Consequently, once the inference instruction is optimized to the boundary of the model’s understanding, further evolutionary attempts risk introducing overly complex descriptions or hallucinations, leading to semantic drift. This also explains the slight oscillation in cIoU observed at higher generations.

Effectiveness of the Visual Arena Mechanism

Table 1 presents the ablation results regarding different fitness evaluation strategies. First, the Direct Scoring strategy, which requires the MLLM to assign absolute quality scores without peer comparison, yields a suboptimal cIoU of 64.45%. This suggests that without a reference target, the model struggles to provide well-calibrated scores and is susceptible to numerical hallucinations, leading to unstable evolutionary directions. Second, introducing a Single-Pass Judge (pairwise comparison without position swapping) improves the cIoU to 66.50% but, surprisingly, degrades the gIoU to 69.42%. We attribute this drop to the inherent position bias of Large Language Models, where the model preferentially selects options at specific positions (e.g., typically favoring the first option). This bias results in the erroneous elimination of high-quality individuals during the selection phase. Finally, our Dual-Pass Arena, equipped with the Double-Switch mechanism, effectively mitigates this bias by enforcing consistency checks across swapped positions. This strategy achieves the best overall performance (72.50% gIoU and 67.40% cIoU), demonstrating that establishing a fair and robust comparison mechanism is critical for identifying superior solutions in evolutionary search.

Qualitative Analysis

To further evaluate the reasoning capability of the model in complex real-world scenarios, Figure 4 presents a detailed visual comparison between EVOL-SAM3 and the strong baseline SAM3Agent. The results reveal the inherent limitations of static reasoning models in handling functional descriptions and abstract spatial referents, while our method demonstrates superior self-correction capabilities. First, the baseline model exhibits a strong salient object bias. When the query involves “objects to propel the boat,” the baseline captures the semantic relevance but is misled by the visually dominant “boat,” failing to localize the semantically correct but smaller “oars.” In contrast, EVOL-SAM3 leverages the semantic mutation mechanism within the evolutionary loop to successfully suppress this visual bias, precisely focusing attention on the specific components that align with the functional description.

Second, when distinguishing low-saliency or implicit targets, the baseline model is highly prone to linguistic hallucinations or missed detections. Benefiting from the incorporation of geometric intuition by the heterogeneous arena module, EVOL-SAM3 not only successfully recalls the imperceptible tire valve area but also accurately grounds the abstract concept of “activity area” to the specific grass field. This provides compelling evidence that modeling the reasoning process as a dynamic search rather than a single-pass mapping effectively bridges the gap between complex semantics and visual signals.

style="width:95.0%" />

Conclusion

In this paper, we proposed EVOL-SAM3, a novel zero-shot framework that reformulates reasoning segmentation as an inference-time evolutionary search process. By synergizing a frozen MLLM with SAM 3 through a dynamic “Generate-Evolve-Arbitrate” pipeline, our method effectively navigates the semantic space to identify optimal prompts without any parameter updates. The introduction of the Visual Arena and Heterogeneous Arena mechanisms ensures robust optimization and safeguards against the hallucinations inherent in static inference. Extensive experiments on ReasonSeg and RefCOCO benchmarks demonstrate that EVOL-SAM3 not only surpasses existing training-free agents but also outperforms fully supervised state-of-the-art models in a zero-shot setting. This work provides compelling evidence that scaling inference-time computation via evolutionary strategies offers a promising and efficient alternative to traditional training paradigms for complex visual reasoning tasks.