CLIP-Map: Structured Matrix Mapping for Parameter-Efficient CLIP Compression

Contrastive Language-Image Pre-training (CLIP) has achieved widely applications in various computer vision tasks, e.g., text-to-image generation, Image-Text retrieval and Image captioning. However, CLIP suffers from high memory and computation cost, which prohibits its usage to the resource-limited application scenarios. Existing CLIP compression methods typically reduce the size of pre-trained CLIP weights by selecting their subset as weight inheritance for further retraining via mask optimization or important weight measurement. However, these select-based weight inheritance often compromises the feature presentation ability, especially on the extreme compression. In this paper, we propose a novel mapping-based CLIP compression framework, CLIP-Map. It leverages learnable matrices to map and combine pretrained weights by Full-Mapping with Kronecker Factorization, aiming to preserve as much information from the original weights as possible. To mitigate the optimization challenges introduced by the learnable mapping, we propose Diagonal Inheritance Initialization to reduce the distribution shifting problem for efficient and effective mapping learning. Extensive experimental results demonstrate that the proposed CLIP-Map outperforms select-based frameworks across various compression ratios, with particularly significant gains observed under high compression settings.

💡 Research Summary

**
The paper addresses the pressing problem of deploying large-scale multimodal models, specifically CLIP, in resource‑constrained environments. While CLIP has demonstrated impressive zero‑shot transfer capabilities across tasks such as text‑to‑image generation, image‑text retrieval, and captioning, its massive parameter count and computational demands limit practical use. Existing compression techniques for CLIP primarily rely on selection‑based pruning: a mask or importance metric identifies “unimportant” weights, which are then removed, followed by a retraining phase to recover performance. Although effective at modest compression ratios, these methods inevitably discard portions of the pretrained knowledge, leading to severe degradation when compression ratios become extreme (e.g., 8×, 16×, 32×).

Key Innovation – Mapping‑Based Compression
The authors propose a fundamentally different paradigm called CLIP‑Map, which replaces hard weight removal with a learnable full‑mapping operation. Instead of selecting a subset of parameters, the entire weight matrix of each transformer layer is transformed into a smaller matrix via two learnable mapping matrices (F^{in}) and (F^{out}). Mathematically, for a layer weight (W \in \mathbb{R}^{D_1 \times D_1}) and a target dimension (D_2 < D_1), the compressed weight is computed as: \