GRIT -- Geometry-Aware PEFT with K-FACPreconditioning, Fisher-Guided Reprojection, andDynamic Rank Adaptation

📝 Original Info

• Title: GRIT – Geometry-Aware PEFT with K-FACPreconditioning, Fisher-Guided Reprojection, andDynamic Rank Adaptation
• ArXiv ID: 2601.00231
• Date: 2026-01-01
• Authors: Pritish Saha, Chandrav Rajbangshi, Rudra Goyal, Mohit Goyal, Anurag Deo, Biswajit Roy, Ningthoujam Dhanachandra Singh, Raxit Goswami, Amitava Das

📝 Abstract

Parameter-efficient fine-tuning (PEFT) has become the default route for adapting LLMs to domain-and application-specific settings, yet widely used methods such as LoRA and QLoRA remain largely geometry-agnostic: they optimize within fixed, randomly oriented low-rank subspaces using first-order descent, largely ignoring local loss curvature. This can inflate the effective update budget and amplify drift along weakly constrained directions. We introduce GRIT, a dynamic, curvature-aware LoRA procedure. GRIT preserves the LoRA parameterization but: (1) preconditions gradients in rank space using K-FAC as a natural-gradient proxy; (2) periodically reprojects the low-rank basis onto dominant Fisher eigendirections to suppress drift; and (3) adapts the effective rank from the spectrum so capacity concentrates where signal resides. The net effect is to steer updates toward high-signal, low-interference directions while using fewer effective parameters. Across instruction-following, comprehension, and reasoning benchmarks on LLaMA backbones, GRIT matches or surpasses LoRA/QLoRA while reducing trainable parame...

📄 Full Content

...(본문 내용이 길어 생략되었습니다. 사이트에서 전문을 확인해 주세요.)