Testing the Efficacy of Hyperparameter Optimization Algorithms in Short-Term Load Forecasting

Accurate forecasting of electrical demand is essential for maintaining a stable and reliable power grid, optimizing the allocation of energy resources, and promoting efficient energy consumption practices. This study investigates the effectiveness of five hyperparameter optimization (HPO) algorithms – Random Search, Covariance Matrix Adaptation Evolution Strategy (CMA–ES), Bayesian Optimization, Partial Swarm Optimization (PSO), and Nevergrad Optimizer (NGOpt) across univariate and multivariate Short-Term Load Forecasting (STLF) tasks. Using the Panama Electricity dataset (n=48,049), we evaluate HPO algorithms’ performances on a surrogate forecasting algorithm, XGBoost, in terms of accuracy (i.e., MAPE, $R^2$) and runtime. Performance plots visualize these metrics across varying sample sizes from 1,000 to 20,000, and Kruskal–Wallis tests assess the statistical significance of the performance differences. Results reveal significant runtime advantages for HPO algorithms over Random Search. In univariate models, Bayesian optimization exhibited the lowest accuracy among the tested methods. This study provides valuable insights for optimizing XGBoost in the STLF context and identifies areas for future research.

💡 Research Summary

This paper investigates the relative effectiveness of five hyper‑parameter optimization (HPO) algorithms—Random Search, Covariance Matrix Adaptation Evolution Strategy (CMA‑ES), Bayesian Optimization, Partial Swarm Optimization (PSO), and Nevergrad Optimizer (NGOpt)—in the context of short‑term load forecasting (STLF). The authors use the publicly available Panama National Electricity Demand dataset, which contains 48,049 hourly observations from January 2015 to June 2020, enriched with weather variables (temperature, humidity, wind speed, precipitation) and categorical indicators (holidays, school days). After removing duplicate rows and applying min‑max scaling, the data are split into univariate (historical demand only) and multivariate (demand plus exogenous features) configurations.

XGBoost is selected as the surrogate forecasting model because of its proven predictive power in load‑forecasting tasks and its relatively short training time, which makes it suitable for evaluating the runtime impact of different HPO strategies. Six key XGBoost hyper‑parameters—max_depth, learning_rate, n_estimators, subsample, colsample_bytree, and min_child_weight—are explored over predefined grids. Each HPO algorithm conducts 50 independent optimization runs, using early stopping with a patience of 20 (except Random Search, which relies on a 5‑fold cross‑validation scheme). The objective function is the root‑mean‑square error (RMSE); the final performance is assessed with three metrics: Mean Absolute Percentage Error (MAPE), coefficient of determination (R²), and total runtime required to locate the best hyper‑parameter set.

Experiments are performed on Google Colab Pro+ (NVIDIA A100 GPU). Sample sizes are varied from 1,000 to 20,000 in increments of 1,000; for each size the three metrics are recorded, normalized to the 0‑1 interval, and plotted to illustrate scalability. To test whether observed differences are statistically significant, a Kruskal‑Wallis non‑parametric test is applied at each sample size, followed by pairwise comparisons with Bonferroni correction (α = 0.05). Results are presented as mean‑rank differences and associated p‑values.

The findings can be summarised as follows. First, all “intelligent” HPO methods (CMA‑ES, Bayesian, PSO, NGOpt) achieve substantially lower runtimes than the baseline Random Search. Bayesian Optimization and NGOpt reduce the total search time by roughly 35‑45 % on average, reflecting their ability to guide the search efficiently and to stop early when improvements plateau. Second, in the multivariate setting, increasing the sample size consistently improves forecasting accuracy for every HPO algorithm: MAPE declines and R² rises, and the Kruskal‑Wallis test shows no significant differences among the algorithms once enough data are available. This suggests that when rich exogenous information is present, the choice of HPO technique matters less for predictive performance. Third, in the univariate case Bayesian Optimization performs worst in terms of both MAPE and R², and its performance is statistically inferior to NGOpt (particularly for R²). The authors attribute this to the limited information content of a single‑feature time series, which hampers the Gaussian‑process surrogate used by Bayesian Optimization. Fourth, CMA‑ES is slower than PSO, while Random Search remains the slowest overall.

The paper acknowledges several limitations. The study is confined to a single national dataset and a single base learner (XGBoost), so the generalisability of the conclusions to other regions, other forecasting models (e.g., LSTM, Transformer), or different hardware environments is not established. Moreover, the impact of the chosen hyper‑parameter search spaces and initialisation strategies on each algorithm’s performance is not exhaustively examined.

Future research directions proposed include (i) extending the benchmark to multiple electricity markets and to other time‑series domains, (ii) evaluating HPO on deep‑learning based forecasters, (iii) incorporating multi‑objective optimisation that balances accuracy, computational cost, and possibly interpretability, and (iv) developing adaptive HPO schemes that dynamically reshape the search space based on intermediate results or meta‑learning.

In conclusion, the study demonstrates that sophisticated HPO algorithms can dramatically cut the time needed to tune XGBoost for STLF, while delivering comparable or superior forecasting accuracy, especially when multivariate inputs are available. Bayesian Optimization, despite its popularity, may be unsuitable for pure univariate load‑forecasting tasks, highlighting the importance of matching the optimisation strategy to the data characteristics and model complexity.