Parallel Track Transformers: Enabling Fast GPU Inference with Reduced Synchronization

Efficient large-scale inference of transformer-based large language models (LLMs) remains a fundamental systems challenge, frequently requiring multi-GPU parallelism to meet stringent latency and throughput targets. Conventional tensor parallelism decomposes matrix operations across devices but introduces substantial inter-GPU synchronization, leading to communication bottlenecks and degraded scalability. We propose the Parallel Track (PT) Transformer, a novel architectural paradigm that restructures computation to minimize cross-device dependencies. PT achieves up to a 16x reduction in synchronization operations relative to standard tensor parallelism, while maintaining competitive model quality in our experiments. We integrate PT into two widely adopted LLM serving stacks-Tensor-RT-LLM and vLLM-and report consistent improvements in serving efficiency, including up to 15-30% reduced time to first token, 2-12% reduced time per output token, and up to 31.90% increased throughput in both settings.

💡 Research Summary

The paper addresses a fundamental systems bottleneck in large‑scale inference of transformer‑based large language models (LLMs): the heavy inter‑GPU synchronization required by conventional tensor parallelism. In standard tensor parallelism, each layer’s weight matrices are sharded across GPUs, and after each partial computation an all‑reduce is performed to combine the results. For a model with L transformer layers, this yields 2 L synchronization points (one for the attention projection, one for the feed‑forward projection) per forward pass, which quickly dominates latency as model size and GPU count increase.

To mitigate this, the authors introduce the Parallel Track (PT) Transformer. The key idea is to decompose the entire model into n independent “tracks”, each a complete transformer sub‑network that runs on a distinct GPU (or set of GPUs). Within a track, computation proceeds exactly as in a dense model, but tracks are only synchronized at the boundaries of “track blocks”. A track block consists of D consecutive transformer layers; after every D‑th layer, all tracks perform a single all‑reduce on their hidden states, then broadcast the fused representation back to each track. Consequently, the number of synchronization points drops from 2 L to L / D, a reduction of up to 16× when D = 8. Because each track operates on a reduced dimensionality (the full hidden size is split among tracks), the volume of data exchanged during each all‑reduce is also smaller, further easing bandwidth pressure.

The PT design is deliberately simple to adopt: it merely re‑orders existing all‑reduce calls to occur at block boundaries, without requiring new communication primitives or complex scheduling. This makes it compatible with existing serving stacks that already support tensor parallelism.

Model quality – The authors train three PT model families (6 B, 13 B, and 30 B parameters) with eight tracks (n = 8) and evaluate them on a broad suite of zero‑shot and few‑shot benchmarks (ARC‑C/E, HellaSwag, PIQA, SciQ, Winogrande, TriviaQA, MMLU, GSM8K, Math, HumanEval). For the 6 B model, increasing D from 2 to 8 leads to a noticeable drop in MMLU (from 0.56 to 0.36), indicating that very small models are more sensitive to infrequent synchronization. However, the larger 13 B and 30 B models retain comparable performance across all D values; differences are typically within 1 % of the dense baseline. This suggests that the representational capacity of larger models compensates for the reduced cross‑track communication.

Serving performance – The PT architecture is integrated into two widely used LLM serving frameworks: TensorRT‑LLM (with an internal PT‑enabled fork) and vLLM. Experiments run on an 8 × H100 GPU cluster using the 30 B model. Across a range of input lengths (1 024–63 488 tokens) and output lengths (128–4 096 tokens), PT consistently reduces time‑to‑first‑token (TTFT) by 15–30 % and time‑per‑output‑token (TPOT) by 2–12 % compared with the dense tensor‑parallel baseline. Throughput (output tokens per second) improves up to 31.9 %, especially for long sequences where synchronization latency dominates. The performance gains are observed for all three block depths (D = 2, 4, 8), with D = 8 often yielding the best throughput due to the fewest synchronization points, while still preserving acceptable latency.

Relation to other parallelism schemes – The paper contrasts PT with Mixture‑of‑Experts (MoE) and expert parallelism. Unlike MoE, PT does not perform token‑level routing; every token passes through every track, and synchronization is deterministic and regular. This regularity simplifies kernel fusion and communication overlap, making PT more amenable to inference‑time optimizations. The authors also sketch a PT‑MoE hybrid, where sparse MLP experts are placed inside each track while the cross‑track synchronization schedule remains unchanged, offering a potential path to further efficiency gains.

Conclusion and impact – Parallel Track Transformers demonstrate that re‑architecting the synchronization schedule—rather than merely overlapping communication with computation—can dramatically reduce the dominant latency source in multi‑GPU LLM inference. The approach delivers substantial latency and throughput improvements with minimal changes to existing software stacks, and it scales well to larger models where quality degradation is negligible. By providing a clear systems‑oriented design that coexists with model‑level innovations (e.g., MoE), PT opens a practical avenue for deploying ever‑larger LLMs in latency‑sensitive production environments.