LLMQ: Efficient Lower-Precision Pretraining for Consumer GPUs

We present LLMQ, an end-to-end CUDA/C++ implementation for medium-sized language-model training, e.g. 3B to 32B parameters, on affordable, commodity GPUs. These devices are characterized by low memory availability and slow communication compared to datacentre-grade GPUs. Consequently, we showcase a range of optimizations that target these bottlenecks, including activation checkpointing, offloading, and copy-engine based collectives. LLMQ is able to train or fine-tune a 7B model on a single 16GB mid-range gaming card, or a 32B model on a workstation equipped with 4 RTX 4090s. This is achieved while executing a standard 8-bit training pipeline, without additional algorithmic approximations, and maintaining FLOP utilization of around 50%. The efficiency of LLMQ rivals that of production-scale systems on much more expensive cloud-grade GPUs.

💡 Research Summary

**
LLMQ is a CUDA‑C++ framework that enables efficient pre‑training and fine‑tuning of medium‑sized language models (3 B–32 B parameters) on consumer‑grade GPUs with limited memory (as low as 16 GB) and relatively slow inter‑GPU communication. The system targets the two primary bottlenecks of such hardware: (1) insufficient device memory for activations, optimizer states, and residuals, and (2) low PCIe bandwidth that prevents direct GPU‑to‑GPU transfers. To address these issues, LLMQ combines a suite of engineering optimizations that are carefully layered.

First, activation checkpointing is made configurable: users can choose to recompute only non‑matrix‑multiplication layers (e.g., SwiGLU, RMSNorm) or the entire transformer block, allowing a trade‑off between memory savings and extra compute. This flexibility lets a 0.5 B model run with batch‑size 6, a 1.5 B model with batch‑size 2–12, and a 3 B model with batch‑size 24 on a single 16 GB GPU.

Second, optimizer states, which dominate memory consumption (8 bytes per parameter in FP32), are stored in BF16 (or FP8) and optionally off‑loaded to host memory. Two off‑loading strategies are provided: zero‑copy page‑locked memory (leveraging the GPU’s ability to read directly from host) and explicit double‑buffering with small GPU staging buffers. Empirically, zero‑copy performs poorly on gaming GPUs (RTX 5060Ti, RTX 4090) while double‑buffering yields higher PCIe utilization, so the framework recommends benchmarking both.

Third, LLMQ implements ZeRO‑1 style sharding of optimizer states by default, and additionally allows independent sharding of model weights and gradients. Because current consumer GPUs cannot communicate directly over PCIe, weights are cached on the host; they are transferred once per optimizer step and then reused for subsequent forward/backward passes, dramatically reducing traffic. The LM‑head and embedding layers, which have huge vocabularies, are replicated across GPUs to avoid excessive communication; only their gradients are synchronized.

Fourth, the training pipeline uses mixed‑precision: the core matrix multiplications (attention and feed‑forward) run in FP8, while non‑linearities, RMSNorm, embeddings, LM‑head, and gradient accumulation stay in BF16. Dynamic tensor‑level abs‑max scaling is applied just‑in‑time before FP8 quantization, guaranteeing no clipping even when tensor statistics change rapidly. This approach preserves the numerical stability of gradient accumulation while exploiting the compute efficiency of FP8 tensor cores on recent Ada (RTX 40xx) and Blackwell (RTX 50xx) GPUs.

Fifth, LLMQ is written entirely in C++/CUDA with deterministic kernels. Reductions are performed via a two‑step process (local accumulation followed by a global kernel) to avoid nondeterministic atomics. Embedding backward passes are made deterministic by sorting token indices on the CPU, allowing each thread block to work on a contiguous token subset without large intermediate buffers. Randomness needed for stochastic rounding is generated with counter‑based PRNGs that require no per‑thread state.

Sixth, a custom communication backend leverages the GPU copy engine to overlap data movement with computation. This “copy‑engine based collectives” approach yields higher throughput than NCCL on consumer hardware. The framework also fuses certain operations (e.g., RMS‑norm + residual addition) and returns the abs‑max of the result in the same kernel, eliminating extra passes.

Performance results demonstrate the effectiveness of these techniques. On a single RTX 4090, LLMQ trains a 14 B model at 4 300 tokens/s with 61 % Model FLOPs Utilization (MFU). On a workstation with four RTX 4090s, it reaches 7 800 tokens/s for a 14 B model (54 % MFU) and 3 400 tokens/s for a 32 B model (51 % MFU). By contrast, a professional L40S GPU achieves only ~29 % MFU under similar conditions. Even a modest RTX 5060Ti (16 GB) can pre‑train a 7 B model with 70 % MFU by off‑loading optimizer states and residuals to host memory. The authors also report successful runs on the new HP ZGX Spark system equipped with a Blackwell‑architecture GPU and unified memory.

In summary, LLMQ shows that with careful memory management, dynamic low‑precision scaling, and a GPU‑centric communication strategy, consumer‑grade GPUs can approach the efficiency of data‑center accelerators for medium‑scale LLM training. The work opens the door for researchers and developers to conduct large‑model experiments locally, reducing reliance on costly cloud resources. Future directions include scaling beyond a single node, integrating NVMe‑based off‑loading, and designing optimizers that operate natively in FP8 to further shrink the memory footprint.