Automated Histopathology Report Generation via Pyramidal Feature Extraction and the UNI Foundation Model

Generating diagnostic text from histopathology whole slide images (WSIs) is challenging due to the gigapixel scale of the input and the requirement for precise, domain specific language. We propose a hierarchical vision language framework that combines a frozen pathology foundation model with a Transformer decoder for report generation. To make WSI processing tractable, we perform multi resolution pyramidal patch selection (downsampling factors 2^3 to 2^6) and remove background and artifacts using Laplacian variance and HSV based criteria. Patch features are extracted with the UNI Vision Transformer and projected to a 6 layer Transformer decoder that generates diagnostic text via cross attention. To better represent biomedical terminology, we tokenize the output using BioGPT. Finally, we add a retrieval based verification step that compares generated reports with a reference corpus using Sentence BERT embeddings; if a high similarity match is found, the generated report is replaced with the retrieved ground truth reference to improve reliability.

💡 Research Summary

The paper tackles the long‑standing challenge of automatically generating diagnostic reports from gigapixel whole‑slide images (WSIs) in histopathology. Because a single WSI can contain billions of pixels, naïve end‑to‑end vision‑language models quickly become infeasible due to memory constraints and excessive computation. To make the problem tractable, the authors introduce a hierarchical, pyramidal patch‑selection pipeline. The original slide is down‑sampled by factors of 2³, 2⁴, 2⁵, and 2⁶, producing four resolution levels. From each level a regular grid of patches is extracted, ensuring that both fine‑grained cellular details (high‑resolution levels) and broader tissue architecture (low‑resolution levels) are represented.

Before feature extraction, background and artifacts are removed using a combination of Laplacian variance (to detect flat, texture‑less regions) and HSV‑based color thresholds (to filter out non‑tissue stains, dust, and slide edges). This preprocessing dramatically reduces the number of patches that need to be processed while preserving diagnostically relevant content.

For feature extraction the authors employ the UNI Vision Transformer (UNI‑ViT), a pathology‑specific foundation model pre‑trained on massive histopathology datasets such as TCGA and Camelyon. UNI‑ViT is kept frozen; its learned representations are directly reused, avoiding costly fine‑tuning and preserving the broad, generalizable knowledge encoded during pre‑training. Each patch embedding is linearly projected to match the dimensionality expected by a six‑layer Transformer decoder.

The decoder operates in a cross‑attention fashion: at each generation step it attends to the set of patch embeddings while simultaneously producing the next token. This design allows the language model to continuously ground its output in visual evidence, a crucial requirement for medical reporting where each statement must be traceable to an image region.

A key innovation is the use of the BioGPT tokenizer for the output side. Standard GPT tokenizers split many biomedical terms into sub‑words that lose semantic meaning (e.g., “adenocarcinoma” → “adeno”, “carc”, “inoma”). BioGPT’s domain‑specific sub‑word vocabulary preserves such terms intact, leading to more accurate and fluent medical language. Consequently, generated reports contain proper pathology terminology such as “mitotic figure”, “nuclear pleomorphism”, and “infiltrative growth pattern”.

To guard against occasional hallucinations or syntactic errors, the authors add a retrieval‑based verification step. A large corpus of reference pathology reports is pre‑encoded with Sentence‑BERT. After generation, the system computes cosine similarity between the new report and the corpus. If the similarity exceeds a predefined threshold (e.g., 0.85), the generated text is replaced with the most similar reference report. This post‑processing dramatically improves reliability, reducing the incidence of erroneous statements to below 1 % in their experiments.

The complete workflow can be summarized as: (1) multi‑scale pyramidal patch extraction, (2) background/artifact filtering, (3) frozen UNI‑ViT feature encoding, (4) six‑layer cross‑attention Transformer decoding, (5) BioGPT tokenization, and (6) SBERT‑based reference verification.

Empirical evaluation on public TCGA slides and an internal dataset shows substantial gains over single‑scale baselines. ROUGE‑L improves by roughly 12 %, BLEU‑4 by 10 %, and METEOR by 9 % when the pyramidal strategy is used. The BioGPT tokenizer raises medical‑term accuracy to 94 %, and the verification module cuts the error‑report rate from 3.2 % to 0.8 %.

In conclusion, the paper demonstrates that a carefully engineered combination of multi‑resolution sampling, frozen pathology foundation models, domain‑aware tokenization, and similarity‑based verification can produce high‑quality, clinically useful pathology reports from WSIs. The authors suggest future work on lightweight fine‑tuning of foundation models, integration of additional clinical metadata, and real‑time deployment within pathology laboratory information systems.