Computer Science / Machine Learning

More Than Bits: Multi-Envelope Double Binary Factorization for Extreme Quantization

Reading time: 1 minute
...

📝 Original Info

  • Title: More Than Bits: Multi-Envelope Double Binary Factorization for Extreme Quantization
  • ArXiv ID: 2512.24545
  • Date: 2025-12-31
  • Authors: Yuma Ichikawa, Yoshihiko Fujisawa, Yudai Fujimoto, Akira Sakai, Katsuki Fujisawa

📝 Abstract

For extreme low-bit quantization of large language models (LLMs), Double Binary Factorization (DBF) is attractive as it enables efficient inference without sacrificing accuracy. However, the scaling parameters of DBF are too restrictive; after factoring out signs, all rank components share the same magnitude profile, resulting in performance saturation. We propose Multi-Envelope DBF (MDBF), which retains a shared pair of 1-bit sign bases but replaces the single envelope with a rank-l envelope. By sharing sign matrices among envelope components, MDBF effectively maintains a binary carrier and utilizes the limited memory budget for magnitude expressiveness. We also introduce a closed-form initialization and an alternating refinement method to optimize MDBF. Across the LLaMA and Qwen families, MDBF enhances perplexity and zeroshot accuracy over previous binary formats at matched bits per weight while preserving the same deployment-friendly inference primitive.

📄 Full Content

...(본문 내용이 길어 생략되었습니다. 사이트에서 전문을 확인해 주세요.)

Start searching

Enter keywords to search articles

↑↓
↵
ESC
⌘K Shortcut