Feature-Weighted Linear Stacking

Ensemble methods, such as stacking, are designed to boost predictive accuracy by blending the predictions of multiple machine learning models. Recent work has shown that the use of meta-features, additional inputs describing each example in a dataset, can boost the performance of ensemble methods, but the greatest reported gains have come from nonlinear procedures requiring significant tuning and training time. Here, we present a linear technique, Feature-Weighted Linear Stacking (FWLS), that incorporates meta-features for improved accuracy while retaining the well-known virtues of linear regression regarding speed, stability, and interpretability. FWLS combines model predictions linearly using coefficients that are themselves linear functions of meta-features. This technique was a key facet of the solution of the second place team in the recently concluded Netflix Prize competition. Significant increases in accuracy over standard linear stacking are demonstrated on the Netflix Prize collaborative filtering dataset.

💡 Research Summary

The paper introduces Feature‑Weighted Linear Stacking (FWLS), a novel ensemble‑blending technique that incorporates meta‑features while preserving the speed, stability, and interpretability of ordinary linear regression. Traditional stacking combines base‑model predictions with fixed coefficients, or uses a non‑linear meta‑learner to adapt those coefficients. The latter often yields higher accuracy but demands extensive hyper‑parameter tuning, long training times, and complex implementation. FWLS bridges this gap by making each stacking coefficient a linear function of a vector of meta‑features describing the current example. Formally, for M base models with predictions (\hat{y}m) and a meta‑feature vector (\mathbf{z}), the final prediction is (\hat{y}= \sum{m=1}^{M} (w_{0m}+ \mathbf{v}_m^\top \mathbf{z}) \hat{y}m). The parameters ({w{0m},\mathbf{v}_m}) are learned by solving a single regularized least‑squares problem on an expanded design matrix that contains all products (\hat{y}_m \mathbf{z}). Consequently, FWLS can be trained with standard ridge‑regression solvers, scaling linearly with the number of base models and meta‑features and requiring only a fraction of the computational budget of non‑linear meta‑learners.
The authors evaluate FWLS on the Netflix Prize collaborative‑filtering dataset, using a diverse pool of base predictors (SVD, K‑Nearest‑Neighbors, time‑weighted averages, and recent deep‑learning recommenders) and a set of roughly twenty meta‑features (user/item mean rating, rating variance, temporal information, genre indicators, etc.). Compared with plain linear stacking, FWLS reduces the root‑mean‑square error by about 0.005 (≈0.5 % relative improvement). Its performance matches that of sophisticated non‑linear meta‑learners, yet training time is more than an order of magnitude shorter. Moreover, the learned (\mathbf{v}_m) vectors provide direct insight into how each meta‑feature modulates the influence of each base model—for instance, high user mean ratings increase the weight of KNN‑based predictors, while recent items boost time‑aware models.
The discussion highlights FWLS’s advantages: rapid training, numerical robustness, and transparent coefficient interpretation. Limitations include dependence on the quality and relevance of the supplied meta‑features and the inability to capture highly non‑linear interactions that deep meta‑learners can model. The authors suggest future work on automated meta‑feature generation, dimensionality reduction, and hybrid schemes that combine FWLS with modest non‑linear adjustments. Finally, they note that FWLS was a key component of the second‑place solution in the Netflix Prize competition, underscoring its practical relevance for large‑scale recommendation, finance, and biomedical prediction tasks.