FedRandom: Sampling Consistent and Accurate Contribution Values in Federated Learning

Federated Learning is a privacy-preserving decentralized approach for Machine Learning tasks. In industry deployments characterized by a limited number of entities possessing abundant data, the significance of a participant’s role in shaping the global model becomes pivotal given that participation in a federation incurs costs, and participants may expect compensation for their involvement. Additionally, the contributions of participants serve as a crucial means to identify and address potential malicious actors and free-riders. However, fairly assessing individual contributions remains a significant hurdle. Recent works have demonstrated a considerable inherent instability in contribution estimations across aggregation strategies. While employing a different strategy may offer convergence benefits, this instability can have potentially harming effects on the willingness of participants in engaging in the federation. In this work, we introduce FedRandom, a novel mitigation technique to the contribution instability problem. Tackling the instability as a statistical estimation problem, FedRandom allows us to generate more samples than when using regular FL strategies. We show that these additional samples provide a more consistent and reliable evaluation of participant contributions. We demonstrate our approach using different data distributions across CIFAR-10, MNIST, CIFAR-100 and FMNIST and show that FedRandom reduces the overall distance to the ground truth by more than a third in half of all evaluated scenarios, and improves stability in more than 90% of cases.

💡 Research Summary

Federated Learning (FL) enables collaborative model training without sharing raw data, but assessing each participant’s contribution remains a challenging and often unstable problem, especially when using Shapley‑value based methods. Existing work has shown that contribution estimates can vary dramatically across different aggregation strategies, undermining fairness, incentives, and the detection of malicious behavior. This paper introduces FedRandom, a novel technique that treats contribution estimation as a statistical sampling problem and dramatically increases the number of independent contribution samples available for analysis. The authors first describe the instability of Shapley‑based contributions and propose two sampling approaches. The baseline Mean Strategy Method (MSM) aggregates contributions from a fixed set of eight well‑known aggregation algorithms (FedAvg, FedAvgM, FedAdagrad, FedAdam, FedYogi, FedMedian, FedTrimmedAvg, Krum) and takes their mean. While simple, MSM is limited by the finite number of distinct aggregation strategies, restricting the achievable sample size and thus the confidence of the estimate. FedRandom overcomes this limitation by defining a small pool of high‑performing aggregators S = {FedAvg, FedAvgM, FedAdagrad, FedAdam, FedYogi}. At each communication round, the server randomly selects one aggregator from S and uses it for model aggregation. Over r communication rounds, this yields up to r × |S| independent contribution samples, effectively providing an unbounded sample set without the need to implement additional algorithms. The contribution estimate is then taken as the arithmetic mean of these samples, mirroring the statistical principle that the sample mean converges to the true underlying value.
The experimental protocol replicates the setup of prior work, using Dirichlet‑based non‑IID splits (α = 1, 10, 100) of four computer‑vision datasets (CIFAR‑10, CIFAR‑100, MNIST, Fashion‑MNIST). A total of 324 FL scenarios (3 epoch settings × 4 datasets × 3 α values × 9 random seeds) are evaluated. For each scenario, the authors run the eight baseline aggregators (for MSM) and 30 independent FedRandom simulations, resulting in over 12,000 federated training runs on V100 GPUs.
Results show that FedRandom consistently reduces the standard deviation of contribution estimates compared to MSM—by an average of more than 70 %—and brings the mean estimates much closer to a size‑based ground truth (the proportion of data owned by each client). In half of the evaluated scenarios, FedRandom cuts the overall distance to the ground truth by more than one third, and improves stability in more than 90 % of cases. While FedRandom’s impact on model convergence speed is modest (its test‑accuracy curves lie between those of the baseline aggregators), its primary value lies in providing a low‑variance, low‑bias contribution assessment. The authors argue that in cross‑silo FL settings, where participants are few but have strong hardware, the additional computational and communication overhead of FedRandom is justified by the increased trust and fairness it enables.
In conclusion, FedRandom offers a practical, scalable solution to the contribution instability problem in federated learning. By leveraging random selection among a small set of aggregators, it generates a large pool of contribution samples, dramatically stabilizing Shapley‑based estimates without requiring new aggregation algorithms. The paper suggests future work on extending FedRandom to asynchronous FL, incorporating more diverse aggregators, and exploring its integration with incentive mechanisms and security defenses.