Flexible block-iterative analysis for the Frank-Wolfe algorithm

We prove that the block-coordinate Frank-Wolfe (BCFW) algorithm converges with state-of-the-art rates in both convex and nonconvex settings under a very mild “block-iterative” assumption. This appears to be the first result on BCFW addressing the setting of nonconvex objective functions with Lipschitz-continuous gradients and no additional assumptions. This analysis newly allows for (I) progress without activating the most-expensive linear minimization oracle(s), LMO(s), at every iteration, (II) parallelized updates that do not require all LMOs, and therefore (III) deterministic parallel update strategies that take into account the numerical cost of the problem’s LMOs. Our results apply for short-step BCFW as well as an adaptive method for convex functions. New relationships between updated coordinates and primal progress are proven, and a favorable speedup is demonstrated using FrankWolfe.jl.

💡 Research Summary

The paper addresses a fundamental limitation of the classic Frank‑Wolfe (FW) method when applied to problems with a product‑structured feasible set. In the standard setting, each iteration requires solving a linear minimization oracle (LMO) for every block of the Cartesian product, which can be prohibitively expensive if some blocks involve costly sub‑problems such as large eigenvalue decompositions or linear programs. To mitigate this, the authors study the block‑coordinate Frank‑Wolfe (BCFW) algorithm under a very mild “block‑iterative” assumption (Assumption 1.1). This assumption merely requires that there exists a positive integer K such that, over any consecutive K iterations, each block is selected at least once. Consequently, the algorithm can postpone the evaluation of expensive LMOs, perform cheap LMOs many times, and even vary the block size from iteration to iteration, allowing deterministic, cost‑aware parallel updates.

The contributions are threefold. First, the paper provides the first convergence analysis of BCFW for non‑convex objectives that only assume a Lipschitz‑continuous gradient, without any additional regularity conditions (e.g., Kurdyka‑Łojasiewicz). Theorem 3.1 shows that after t iterations the Frank‑Wolfe gap decays as O(1/√t), matching the best known rates for the classical FW method in the non‑convex regime. Second, for convex objectives the authors prove an O(1/t) primal‑gap convergence rate for an adaptive step‑size variant of BCFW that does not require prior knowledge of the smoothness constant. Corollary 2.2 further establishes that the short‑step BCFW (with a fixed step size 2/(t+2)) attains the same constant‑factor rate as the standard FW algorithm. Third, the analysis is built entirely on the flexible block‑iterative scheme, which unifies many existing selection strategies (essentially‑cyclic, random, greedy) and enables new deterministic strategies that take LMO computational cost into account.

Technical highlights include the introduction of a Huber‑type perspective function ρ(x,b) to handle the non‑monotone extra‑progress terms a_t in the recursion, and Lemma 1.3, which provides explicit bounds on sequences satisfying h_t – h_{t+1} ≥ ρ(h_t + a_t, b). These tools allow the authors to relate the progress contributed by each updated block to the overall Frank‑Wolfe gap, even when block sizes and selections vary across iterations. Lemma 1.1 connects partial gaps G_J(x) to the global primal gap, providing a lower bound that is crucial for the convergence proofs.

The experimental section, implemented in the Julia package FrankWolfe.jl, validates the theory on synthetic toy problems with heterogeneous LMO costs and on a large‑scale matrix factorization task. Cost‑aware block selection strategies—e.g., repeatedly updating cheap blocks while ensuring that expensive blocks are activated at least once every K iterations—outperform traditional cyclic or uniformly random schemes. The authors report improvements in the number of LMO calls, gradient evaluations, and wall‑clock time, with speed‑ups of up to a factor of two on multi‑core hardware.

In summary, this work significantly expands the theoretical foundations of block‑coordinate Frank‑Wolfe methods. By removing restrictive assumptions for non‑convex problems and by introducing a highly flexible, cost‑sensitive block‑selection framework, it opens the door to efficient projection‑free optimization in settings where oracle costs are heterogeneous and parallel resources are available. The analytical techniques based on the ρ‑function and non‑monotone progress terms are likely to be useful beyond the specific BCFW context, potentially influencing the design of other block‑coordinate and proximal algorithms.