Can LLMs Reason Over Non-Text Modalities in a Training-Free Manner? A Case Study with In-Context Representation Learning

The remarkable performance of Large Language Models (LLMs) can be enhanced with test-time computation, which relies on external tools and even other deep learning models. However, existing approaches for integrating non-text modality representations into LLMs typically require additional costly supervised training, restricting on-the-fly adaptation to new domains and modalities. In this work, we explore the feasibility of integrating representations from non-text foundational models (FMs) into text-based LLMs in a training-free manner. We propose In-Context Representation Learning (ICRL) as a proof-of-concept to allow LLMs to adaptively utilize non-text modality representations with few-shot learning. Unlike traditional in-context learning, which incorporates text-label pairs, ICRL replaces text inputs with FM representations, enabling the LLM to perform multi-modal inference without fine-tuning. We evaluate ICRL on a suite of tasks in the molecular domain, investigating three core research questions: (i) how to map FM representations into LLMs in a training-free manner, (ii) what factors influence ICRL performance, and (iii) what mechanisms underlie the effectiveness of ICRL. To the best of our knowledge, ICRL is the first training-free framework for integrating non-text modality representations into text-based LLMs, presenting a promising direction for adaptable, multi-modal generalization.

💡 Research Summary

This paper investigates a fundamental question: Can text-based Large Language Models (LLMs) leverage representations from non-text modalities (e.g., molecular structures, images) during inference without any additional supervised training? Current methods for multi-modal integration typically require costly fine-tuning of projection layers or the LLM itself, limiting adaptability. To address this, the authors propose In-Context Representation Learning (ICRL), a novel, training-free framework that enables LLMs to perform multi-modal reasoning by adapting the principles of in-context learning (ICL).

The core idea of ICRL is to replace the textual input in standard ICL few-shot examples with feature vectors extracted from a modality-specific Foundation Model (FM). Instead of providing (text, label) pairs, ICRL constructs (FM representation, label) pairs and injects them into the LLM’s context. This allows the LLM to utilize non-textual information for prediction without any parameter updates.

The research is structured around three key questions:

1. How to map FM representations into an LLM training-free (RQ1)? The authors explore two levels of injection:

   • Text-Level: High-dimensional FM embeddings are reduced via Principal Component Analysis (PCA) and converted into strings, which are then inserted directly into the prompt text.
   • Embedding-Level: FM representations are directly injected into the LLM’s embedding layer. To handle dimensionality mismatch and distribution alignment, several methods are compared: simple Zero-Padding, a Random (linear) Projector, and an Optimal Transport (OT) based alignment technique. The OT method, which aligns the statistical properties (mean, variance) of the projected FM features to a target distribution (either the original text embeddings or PCA-string embeddings), proves to be the most effective.

2. What factors influence ICRL performance (RQ2)? Experiments confirm that standard ICL factors like the number of few-shot examples significantly affect ICRL. Key novel findings include: ICRL performance improves when the projected FM representation has a higher cosine similarity to its corresponding original text embedding, and it degrades if the projected representations from different few-shot examples become overly uniform (losing discriminability).

3. What mechanisms underlie ICRL (RQ3)? Through mechanistic analysis, the authors uncover that when ICRL examples are presented alongside traditional text-based ICL examples, the ICRL-injected representations undergo a mode shift and are treated similarly to “pause tokens” by the LLM. This suggests the LLM may engage a distinct processing mode for non-textual information.

The framework is rigorously evaluated on a suite of molecular domain tasks (e.g., property prediction), chosen because molecular structures can be represented both as text (SMILES strings, for ICL) and as non-textual FM features (for ICRL), enabling a clean, apples-to-apples comparison.

Theoretical Underpinning: The paper provides a theoretical analysis demonstrating that a randomly initialized linear projector preserves the norms and angles (cosine similarity) of high-dimensional vectors with high probability, justifying its use over non-linear projectors that may distort information.

In summary, this work makes significant contributions: (i) proposing ICRL, the first training-free method for integrating non-text modality representations into text-based LLMs; (ii) providing a comprehensive empirical study of design choices and their impact on performance across molecular tasks; and (iii) offering novel mechanistic insights into how LLMs process these injected representations. It presents a promising direction for building highly adaptable, multi-modal systems by dynamically composing pre-trained text LLMs and specialized FMs at inference time.