mViSE: A Visual Search Engine for Analyzing Multiplex IHC Brain Tissue Images

Whole-slide multiplex imaging of brain tissue generates massive information-dense images that are challenging to analyze and require custom software. We present an alternative query-driven programming-free strategy using a multiplex visual search engine (mViSE) that learns the multifaceted brain tissue chemoarchitecture, cytoarchitecture, and myeloarchitecture. Our divide-and-conquer strategy organizes the data into panels of related molecular markers and uses self-supervised learning to train a multiplex encoder for each panel with explicit visual confirmation of successful learning. Multiple panels can be combined to process visual queries for retrieving similar communities of individual cells or multicellular niches using information-theoretic methods. The retrievals can be used for diverse purposes including tissue exploration, delineating brain regions and cortical cell layers, profiling and comparing brain regions without computer programming. We validated mViSE’s ability to retrieve single cells, proximal cell pairs, tissue patches, delineate cortical layers, brain regions and sub-regions. mViSE is provided as an open-source QuPath plug-in.

💡 Research Summary

The paper presents mViSE (multiplex Visual Search Engine), a novel query-driven, programming-free software tool designed to tackle the analytical challenges posed by whole-slide multiplex immunohistochemistry (IHC) images of brain tissue. Traditional analysis pipelines for such information-dense, high-dimensional images are often rigid and require custom coding. mViSE offers a flexible alternative by functioning as a visual search engine that allows users to interactively explore and analyze tissue based on visual and molecular similarity.

The mViSE framework operates in two main phases: Learning and Query. In the Learning phase, it employs a “divide-and-conquer” strategy. The dozens of imaging channels (each representing a specific protein marker) are organized into biologically meaningful panels (e.g., a glial panel, a vascular panel). For each panel, a dedicated multiplex encoder is trained using self-supervised learning. This encoder is based on a Vision Transformer architecture, initialized with the BEiTv3 foundation model and enhanced with an Efficient Channel Attention module to weight informative channels. Crucially, the training process incorporates techniques from unsupervised person re-identification, using cluster-contrastive and triplet losses to create an embedding space where visually and architecturally similar tissue patches are close together. The success of learning for each panel is explicitly validated by visualizing pseudo-label maps that recapitulate known brain cytoarchitecture, mitigating the “black box” problem.

In the Query phase, users can visually select a cell or tissue region of interest and specify which molecular marker panels are relevant. mViSE then processes this query by combining the encodings from the selected panels using scalable, information-theoretic community detection methods (like InfoMap) and k-nearest neighbor graphs. This hierarchical combination allows the system to retrieve biologically similar communities of individual cells or multicellular niches from the entire dataset. The retrievals can be visualized in their spatial context and their molecular expression profiles can be examined.

The authors comprehensively validate mViSE’s capabilities. They demonstrate its precision in retrieving specific single cells (e.g., neurons, microglia, vascular cells), proximal cell pairs, and tissue patches. Furthermore, they show that mViSE can successfully delineate fine anatomical structures such as cortical layers and broader brain regions and sub-regions (e.g., hippocampus), tasks that typically require extensive manual annotation or complex algorithmic pipelines. By providing mViSE as an open-source plugin for the widely used QuPath platform, the authors have created an accessible tool that enables diverse investigational goals—from exploratory tissue characterization and comparative region profiling to hypothesis testing—without the need for programming expertise, thereby democratizing the analysis of complex multiplex brain tissue images.