Augmenting representations with scientific papers

Astronomers have acquired vast repositories of multimodal data, including images, spectra, and time series, complemented by decades of literature that analyzes astrophysical sources. Still, these data sources are rarely systematically integrated. This work introduces a contrastive learning framework designed to align X-ray spectra with domain knowledge extracted from scientific literature, facilitating the development of shared multimodal representations. Establishing this connection is inherently complex, as scientific texts encompass a broader and more diverse physical context than spectra. We propose a contrastive pipeline that achieves a 20% Recall@1% when retrieving texts from spectra, proving that a meaningful alignment between these modalities is not only possible but capable of accelerating the interpretation of rare or poorly understood sources. Furthermore, the resulting shared latent space effectively encodes physically significant information. By fusing spectral and textual data, we improve the estimation of 20 physical variables by 16-18% over unimodal spectral baselines. Our results indicate that a Mixture of Experts (MoE) strategy, which leverages both unimodal and shared representations, yields superior performance. Finally, outlier analysis within the multimodal latent space identifies high-priority targets for follow-up investigation, including a candidate pulsating ULX (PULX) and a gravitational lens system. Importantly, this framework can be extended to other scientific domains where aligning observational data with existing literature is possible.

💡 Research Summary

The paper introduces a novel multimodal foundation model that aligns X‑ray spectra with textual summaries of scientific papers using contrastive learning. Leveraging 11,447 paired observations from the Chandra Source Catalog and the NASA Astrophysics Data System, the authors construct a dataset where each X‑ray spectrum (discretized into 400 energy bins and min‑max normalized) is matched with a GPT‑4o‑mini generated abstract summary embedded by OpenAI’s Ada‑002 model. Two pre‑trained unimodal encoders are employed: a transformer‑based autoencoder compresses spectra into 64‑dimensional latent vectors, while the text pipeline produces 4,608‑dimensional embeddings. Separate fully‑connected networks map both modalities into a shared 64‑dimensional space, and the alignment is driven by the InfoNCE contrastive loss, which maximizes cosine similarity for true pairs and minimizes it for mismatched pairs. Hyper‑parameters such as learning rate, shared space dimension, dropout, and hidden layer size are tuned via grid search on a calibration set.

Three downstream tasks evaluate the quality of the shared representation. First, cross‑modal retrieval shows that given a spectrum, the correct paper summary appears in the top 1 % of candidates 20 % of the time (Recall@1 % = 0.20) and in the top 5 % 50 % of the time, with a median rank of 84 among 1,719 candidates. Second, physical parameter regression uses a 3‑nearest‑neighbor regressor to predict 20 astrophysical variables (hardness ratios, photon index, column densities, temperatures, etc.) from the latent vectors. The multimodal representation yields an average Pearson correlation of 0.55, surpassing the unimodal spectra (0.43) and text (0.30) baselines. Mean Absolute Error (MAE) improves by 16 %–18 % relative to the best single‑modality model, with especially large gains (≈34 %) for hardness ratios and hydrogen column density estimates. A Mixture‑of‑Experts (MoE) strategy selects, per variable, the modality (spectra, text, or combined) that maximizes validation‑set correlation, further boosting performance. Third, outlier detection applies Isolation Forest to the aligned latent space, flagging statistical anomalies. The top 1 % of anomalies include a previously unreported candidate pulsating ultra‑luminous X‑ray source (PULX) and a gravitational lens system, confirming the model’s ability to surface scientifically interesting objects that were not present in the training corpus.

The authors also highlight a 97 % dimensionality reduction (from 4,672 to 128 dimensions) while preserving physical information, a crucial property for billion‑object surveys where exhaustive similarity searches are computationally prohibitive. Limitations are acknowledged: the retrieval performance, while promising, remains modest; the mismatch between spectral signatures and textual descriptions constrains alignment; the framework does not yet support generative tasks such as text synthesis from spectra; and anomaly detection could be refined with physics‑based priors. Future work may incorporate higher‑quality summaries, multimodal observations (e.g., XMM‑Newton spectra), and domain‑specific priors to improve both retrieval and discovery.

In discussion, the authors argue that the approach is broadly applicable beyond astronomy, citing seismology, climate science, and medicine as domains where paired observational data and expert reports exist. By systematically integrating literature knowledge into representation learning, the model not only accelerates interpretation of new observations but also creates a semantic bridge between data and the accumulated expertise of the scientific community. The paper concludes that as next‑generation surveys generate petabyte‑scale multimodal datasets, such contrastive, knowledge‑augmented models will be essential for scalable, interpretable, and discovery‑driven science.