Distributed Matrix-Vector Multiplication: A Convolutional Coding Approach

Distributed computing systems are well-known to suffer from the problem of slow or failed nodes; these are referred to as stragglers. Straggler mitigation (for distributed matrix computations) has recently been investigated from the standpoint of erasure coding in several works. In this work we present a strategy for distributed matrix-vector multiplication based on convolutional coding. Our scheme can be decoded using a low-complexity peeling decoder. The recovery process enjoys excellent numerical stability as compared to Reed-Solomon coding based approaches (which exhibit significant problems owing their badly conditioned decoding matrices). Finally, our schemes are better matched to the practically important case of sparse matrix-vector multiplication as compared to many previous schemes. Extensive simulation results corroborate our findings.

💡 Research Summary

**
The paper addresses the well‑known straggler problem in distributed matrix‑vector multiplication, where slow or failed worker nodes delay the overall computation. While recent works have applied Reed‑Solomon (RS) erasure codes to mitigate stragglers, they suffer from two major drawbacks: (i) the decoding matrices have very high condition numbers, leading to poor numerical stability, and (ii) the linear combinations required by RS destroy the sparsity of the original matrix, inflating computation time for sparse problems common in machine learning.

To overcome these issues, the authors propose a novel scheme based on binary cross‑parity‑check convolutional codes, denoted CP(n,k). In this framework each column of an infinite‑length sequence (represented as a formal Laurent series) corresponds to the workload assigned to a worker node. The code is defined by a set of geometric constraints (Equation 1) that enforce the sum of coefficients along lines of a given slope to be zero. These constraints are captured by a parity‑check matrix Hₙ,ₖ(D), which has a Vandermonde structure in powers of the indeterminate D. Because any (n‑k)×(n‑k) sub‑matrix of Hₙ,ₖ(D) is invertible, the original data can be recovered from any n‑k surviving workers.

A key contribution is the construction of a systematic generator matrix Gₙ,ₖ(D) that is feed‑forward rather than recursive. Theorem 1 proves that each entry of Gₙ,ₖ(D) is a finite polynomial with integer coefficients; for the practically important case of two stragglers (s = n‑k = 2) the coefficients belong to {‑1,0,1}, and for three stragglers they are bounded in magnitude by k. This property guarantees that the encoding operations consist only of additions and subtractions, preserving sparsity and requiring minimal storage.

The authors map the matrix‑vector product Ax onto the CP code as follows. The matrix A is partitioned row‑wise into Δ block‑rows (Δ is chosen to be a multiple of k). For each block‑row j they compute u_j = A_j x. They then form k polynomials ũ_i(D) = Σ_{ℓ=0}^{q‑1} u_{i q+ℓ} D^ℓ, where q = Δ/k. Stacking these polynomials into a vector U(D) and multiplying by Gₙ,ₖ(D) yields the set of encoded sub‑tasks assigned to the n workers. A storage constraint γ (the fraction of rows each worker can store) leads to a simple bound Δ ≥ (λ/γ − 1) k, where λ is the maximal exponent spread in any column of Gₙ,ₖ(D).

Decoding is performed by a peeling decoder that exploits the geometric constraints. At each iteration the decoder identifies a constraint line that contains exactly one unknown symbol, solves for it (which is just a subtraction), and proceeds iteratively. Lemma 1 quantifies how many symbols can be recovered from each straggler after each phase, showing that the process terminates after at most s − 1 phases. The decoder’s computational complexity is linear in the number of symbols and requires only elementary arithmetic, unlike RS decoding which involves solving dense linear systems.

Extensive simulations compare the proposed CP‑based scheme with RS‑based schemes for various (n,k) configurations, numbers of stragglers (s = 2,3), and matrix sparsity levels (up to 90 % zeros). Results demonstrate that CP achieves:

• Lower latency – on average 30 %–50 % faster recovery because the master node performs only simple peeling steps.
• Superior numerical stability – condition numbers close to 1, yielding 10⁴–10⁶ times smaller reconstruction error than RS.
• Sparsity preservation – when A is sparse, the CP scheme’s computation time remains comparable to the dense case, whereas RS incurs a 2×–3× slowdown due to dense linear combinations.

In summary, the paper introduces a convolutional‑coding‑based framework for straggler mitigation that simultaneously offers low‑complexity decoding, excellent numerical robustness, and compatibility with sparse matrix‑vector products. Theoretical analysis (including proofs of feed‑forward generator structure and coefficient bounds) and empirical evaluation together establish the proposed method as a compelling alternative to Reed‑Solomon coding for distributed linear algebra workloads. Future work is suggested on dynamic job reallocation under varying straggler patterns and on real‑world cloud deployments.