Hyperbolic Fine-Tuning for Large Language Models

Large language models (LLMs) have demonstrated remarkable performance across various tasks. However, it remains an open question whether the default Euclidean space is the most suitable choice for LLMs. In this study, we investigate the geometric characteristics of LLMs, focusing specifically on tokens and their embeddings. Our findings reveal that token frequency follows a power-law distribution, where high-frequency tokens (e.g., the, that ) constitute the minority, while low-frequency tokens (e.g., apple, dog) constitute the majority. Furthermore, high-frequency tokens cluster near the origin, whereas low-frequency tokens are positioned farther away in the embedding space. Additionally, token embeddings exhibit hyperbolic characteristics, indicating a latent tree-like structure within the embedding space. Motivated by these observations, we propose HypLoRA, an efficient fine-tuning approach that operates in hyperbolic space to exploit these underlying hierarchical structures better. HypLoRA performs low-rank adaptation directly in hyperbolic space, thereby preserving hyperbolic modeling capabilities throughout the fine-tuning process. Extensive experiments across various base models and reasoning benchmarks, specifically arithmetic and commonsense reasoning tasks, demonstrate that HypLoRA substantially improves LLM performance.

💡 Research Summary

The paper “Hyperbolic Fine‑Tuning for Large Language Models” investigates whether the conventional Euclidean embedding space is optimal for large language models (LLMs). By analyzing token frequency distributions and the geometry of token embeddings across several reasoning datasets (GSM8K, AQuA, MAWPS, SVAMP), the authors discover two key empirical facts. First, token frequencies follow a power‑law distribution: a small set of high‑frequency function words (e.g., “the”, “that”) dominate the corpus, while the vast majority of tokens are low‑frequency content words (e.g., “apple”, “dog”). Second, there is a strong inverse correlation between token frequency and embedding norm: high‑frequency tokens cluster near the origin of the embedding space, whereas low‑frequency tokens lie farther out. This pattern suggests a radial hierarchy where abstract, frequent concepts occupy central positions and specific, rare concepts occupy peripheral positions.

To probe the local geometry, the authors compute hyperbolicity (δ) for the metric space induced by token embeddings within individual prompts. Small δ values (often < 0.1) indicate that the distances among tokens approximate a tree‑like metric, confirming that token embeddings possess intrinsic hyperbolic characteristics. This observation aligns with prior theoretical work linking power‑law distributions to hyperbolic geometry, but the paper provides concrete measurements on modern LLMs.

Given these findings, the authors argue that incorporating a hyperbolic inductive bias could improve downstream adaptation. Traditional parameter‑efficient fine‑tuning (PEFT) methods such as LoRA adapt weight matrices via a low‑rank decomposition (ΔW = BA) while keeping the original weights frozen. However, applying LoRA directly in hyperbolic space typically requires mapping embeddings to the tangent space using exponential and logarithmic maps, performing Euclidean updates, and mapping back. This round‑trip cancels much of the curvature information, effectively reducing the operation to a Euclidean transformation and negating the benefits of hyperbolic geometry.

To overcome this limitation, the paper introduces HypLoRA, a hyperbolic low‑rank adaptation method that operates directly on the Lorentz (hyperboloid) model without leaving the manifold. Using Möbius addition and scalar multiplication, the low‑rank matrices A and B are defined in the hyperbolic space, and the update ΔW = B ⊙ A (where ⊙ denotes hyperbolic composition) is applied directly to the original weight matrix. This design preserves the negative curvature throughout training, avoids the cancellation effect, and retains the same parameter efficiency as Euclidean LoRA ((d + k)·r trainable parameters).

The authors evaluate HypLoRA on multiple base models (LLaMA‑3‑8B, LLaMA‑13B, Gemma‑7B) and a suite of arithmetic and commonsense reasoning benchmarks. Compared with standard LoRA, DoRA, AdaLoRA, and prompt‑tuning baselines, HypLoRA consistently yields higher accuracy, typically improving by 2–4 percentage points under the same parameter budget. Notably, gains are larger on tasks where low‑frequency tokens play a significant role, supporting the hypothesis that hyperbolic structure better captures hierarchical relationships needed for reasoning. Training time and memory consumption remain comparable to Euclidean LoRA, demonstrating practical feasibility.

The paper also discusses limitations and future directions. The choice of curvature K in the Lorentz model can affect numerical stability; adaptive curvature strategies may be explored. Extending hyperbolic adaptation beyond token embeddings to attention matrices or feed‑forward layers could further exploit hierarchical priors. Moreover, combining hyperbolic manifolds with other non‑Euclidean geometries (e.g., spherical) or developing hybrid models may capture even richer linguistic structures.

In summary, the work provides three major contributions: (1) a comprehensive empirical analysis showing that LLM token embeddings naturally exhibit a tree‑like, hyperbolic organization correlated with token frequency; (2) the design of HypLoRA, the first low‑rank adaptation method that directly operates on a hyperbolic manifold, preserving curvature while remaining parameter‑efficient; and (3) extensive experiments demonstrating that hyperbolic fine‑tuning yields consistent performance improvements on reasoning tasks without additional computational overhead. This study opens a new avenue for integrating geometric priors into LLM adaptation and suggests that future foundation models may benefit from native hyperbolic representations.