Disco: Densely-overlapping Cell Instance Segmentation via Adjacency-aware Collaborative Coloring

Accurate cell instance segmentation is foundational for digital pathology analysis. Existing methods based on contour detection and distance mapping still face significant challenges in processing complex and dense cellular regions. Graph coloring-based methods provide a new paradigm for this task, yet the effectiveness of this paradigm in real-world scenarios with dense overlaps and complex topologies has not been verified. Addressing this issue, we release a large-scale dataset GBC-FS 2025, which contains highly complex and dense sub-cellular nuclear arrangements. We conduct the first systematic analysis of the chromatic properties of cell adjacency graphs across four diverse datasets and reveal an important discovery: most real-world cell graphs are non-bipartite, with a high prevalence of odd-length cycles (predominantly triangles). This makes simple 2-coloring theory insufficient for handling complex tissues, while higher-chromaticity models would cause representational redundancy and optimization difficulties. Building on this observation of complex real-world contexts, we propose Disco (Densely-overlapping Cell Instance Segmentation via Adjacency-aware COllaborative Coloring), an adjacency-aware framework based on the “divide and conquer” principle. It uniquely combines a data-driven topological labeling strategy with a constrained deep learning system to resolve complex adjacency conflicts. First, “Explicit Marking” strategy transforms the topological challenge into a learnable classification task by recursively decomposing the cell graph and isolating a “conflict set.” Second, “Implicit Disambiguation” mechanism resolves ambiguities in conflict regions by enforcing feature dissimilarity between different instances, enabling the model to learn separable feature representations.

💡 Research Summary

The paper introduces Disco, a novel cell instance segmentation framework that reframes the problem as a graph‑coloring task and explicitly handles the topological conflicts that arise in densely packed tissues. The authors first release a large‑scale, stress‑test dataset named GBC‑FS 2025, comprising 2,839 frozen‑section images of gallbladder cancer with 864,204 annotated nuclei. Using this dataset together with three public benchmarks (PanNuke, DSB2018, CryoNuSeg), they conduct the first systematic quantitative analysis of cell adjacency graphs (CAGs). Their analysis shows that, except for the extremely sparse PanNuke, real‑world cell graphs are predominantly non‑bipartite, with odd‑length cycles—especially 3‑cycles (triangles)—accounting for over 90 % of all odd cycles. Consequently, a simple 2‑coloring model (which assumes bipartiteness) is insufficient for most histopathology images. They further define “conflict nodes” (the minimal vertex set whose removal makes the graph bipartite) and “secondary conflicts” (edges between two conflict nodes). In GBC‑FS 2025, conflict nodes constitute 30.49 % of all cells, and secondary conflicts involve 24.64 % of them, highlighting the severity of topological ambiguity.

Disco tackles these challenges with two complementary mechanisms:

1. Explicit Marking – The method recursively decomposes the CAG to isolate a conflict set and projects it back onto the image as a binary “Conflict Map”. This map is learned jointly with the standard instance mask using a dual‑head network (backbone: ResNet‑50 + FPN). The Conflict Map provides a pixel‑level supervision that tells the network exactly where 2‑coloring would fail.

2. Implicit Disambiguation – Within the identified conflict regions, Disco enforces feature‑space separation between different instances. For each pair of conflict nodes, a margin‑based loss (cosine‑distance or Euclidean) pushes their embedding vectors apart, ensuring that even if the same color label is assigned, the learned features remain discriminative. This loss is combined with the usual mask loss and the conflict‑map loss:
   L_total = L_mask + λ1·L_conflict + λ2·L_disambiguation.

The overall training objective balances pixel‑wise segmentation accuracy, correct detection of conflict zones, and feature separability. During inference, the predicted Conflict Map guides a post‑processing step that either re‑colors the conflicting nodes using a simple greedy algorithm or applies a small‑scale clustering (e.g., K‑means) on the embeddings to split merged instances.

Extensive experiments on the four datasets demonstrate that Disco consistently outperforms state‑of‑the‑art methods such as StarDist, Hover‑Net, Cellpose, and the previously proposed 4‑color FCIS model. On GBC‑FS 2025, Disco improves the Parsing Quality (PQ) metric by 7.08 % absolute (from 0.62 to 0.69). Across all datasets, the average PQ gain is 2.72 %. Ablation studies confirm that both Explicit Marking and Implicit Disambiguation are essential: removing either component leads to a noticeable drop in performance, especially in regions with high conflict‑node density.

The authors also discuss limitations. The recursive conflict‑set extraction has a computational cost that grows with graph size, potentially limiting scalability to whole‑slide images without further optimization. Moreover, the current design focuses on 3‑cycle‑dominated conflicts; more complex odd cycles (e.g., 5‑ or 7‑cycles) are not explicitly addressed and may require higher chromatic numbers. Future work is proposed to integrate Graph Neural Networks for end‑to‑end conflict prediction, to dynamically select the appropriate number of colors, and to extend the approach to other tissue types and multi‑modal imaging.

In summary, Disco provides a principled, topologically aware solution to cell instance segmentation in crowded histopathology images. By converting adjacency conflicts into a learnable supervision signal and enforcing feature‑level disambiguation, it bridges the gap between graph‑theoretic optimality and deep‑learning practicality, delivering both superior quantitative performance and a novel interpretability tool—the predicted Conflict Map—for pathology research.