PCA-Seg: Revisiting Cost Aggregation for Open-Vocabulary Semantic and Part Segmentation

Recent advances in vision-language models (VLMs) have garnered substantial attention in open-vocabulary semantic and part segmentation (OSPS). However, existing methods extract image-text alignment cues from cost volumes through a serial structure of spatial and class aggregations, leading to knowledge interference between class-level semantics and spatial context. Therefore, this paper proposes a simple yet effective parallel cost aggregation (PCA-Seg) paradigm to alleviate the above challenge, enabling the model to capture richer vision-language alignment information from cost volumes. Specifically, we design an expert-driven perceptual learning (EPL) module that efficiently integrates semantic and contextual streams. It incorporates a multi-expert parser to extract complementary features from multiple perspectives. In addition, a coefficient mapper is designed to adaptively learn pixel-specific weights for each feature, enabling the integration of complementary knowledge into a unified and robust feature embedding. Furthermore, we propose a feature orthogonalization decoupling (FOD) strategy to mitigate redundancy between the semantic and contextual streams, which allows the EPL module to learn diverse knowledge from orthogonalized features. Extensive experiments on eight benchmarks show that each parallel block in PCA-Seg adds merely 0.35M parameters while achieving state-of-the-art OSPS performance.

💡 Research Summary

The paper addresses a fundamental limitation in current open‑vocabulary semantic and part segmentation (OSPS) methods that rely on vision‑language models (VLMs) such as CLIP. Existing approaches extract image‑text alignment cues from a cost volume by first applying spatial aggregation and then class aggregation in a serial fashion. This sequential design creates a “cascading effect” where the output of the first aggregation biases the second, leading to knowledge interference between class‑level semantics and spatial context. The authors illustrate this problem with a concrete example: spatial aggregation improves contextual consistency but distorts the semantics of the “truck” class, and the subsequent class aggregation amplifies this distortion, causing the model to misclassify the truck region as “runway”.

To eliminate this interference, the authors propose PCA‑Seg, a parallel cost‑aggregation paradigm. Instead of feeding the cost‑volume embedding through two consecutive blocks, PCA‑Seg processes the same embedding simultaneously with a spatial aggregator Φ and a class aggregator Γ, producing two independent feature streams: B (spatial context) and E (class semantics). This decoupling preserves the integrity of each information type and prevents the bias propagation inherent in the serial design.

However, simply having two parallel streams is insufficient; the model must fuse them effectively. The paper introduces an Expert‑Driven Perceptual Learning (EPL) module comprising a Multi‑Expert Parser and a Coefficient Mapper. The parser concatenates B and E, then passes the combined tensor through Z (empirically set to 4) lightweight expert blocks. Each expert consists of two 1×1 convolutions with BatchNorm and GeLU activation, adding only 0.085 M parameters. Canonical Correlation Analysis shows that the experts learn low‑correlated, complementary representations, confirming the diversity of the extracted knowledge.

The Coefficient Mapper learns pixel‑wise weighting coefficients for each expert output. A small MLP takes the concatenated features as input and outputs Z scalar weights per pixel, which are normalized by a Softmax. The final fused embedding is a weighted sum of the expert outputs, allowing the network to emphasize the most informative expert for each spatial location dynamically.

To further reduce redundancy between the two streams, the authors propose Feature Orthogonalization Decoupling (FOD). An orthogonalization loss forces the cosine similarity between B and E to approach zero, encouraging the streams to become orthogonal in feature space. This orthogonalization makes the EPL module’s job easier, as it receives more diverse inputs, and experimentally yields a 0.9 % absolute gain in unseen‑class mIoU on the ADE20K‑Part benchmark.

From an efficiency standpoint, each parallel block adds only 0.35 M parameters and consumes an extra 0.96 GB of GPU memory, which is negligible compared with the performance gains. Extensive experiments on eight benchmarks—including PASCAL‑5ᵢ, COCO‑Stuff, ADE20K‑Part, and several “Pred‑All/Oracle‑Obj” settings—show that PCA‑Seg consistently outperforms state‑of‑the‑art methods such as DeCLIP, H‑CLIP, and PartCATSeg, achieving improvements of 1–2 % in mean IoU and notable gains on unseen categories.

In summary, PCA‑Seg revisits the cost‑aggregation stage of VLM‑based dense prediction, replacing the traditional serial pipeline with a parallel architecture, enriching the fused representation through a multi‑expert, coefficient‑weighted learning scheme, and enforcing orthogonal feature streams to maximize diversity. The result is a lightweight yet powerful framework that markedly improves open‑vocabulary semantic and part segmentation, setting a new benchmark for future research in vision‑language alignment for dense prediction tasks.