Whale Optimization Algorithm (WOA) suffers from limited global search ability, slow convergence, and tendency to fall into local optima, restricting its effectiveness in hyperparameter optimization for machine learning models. To address these issues, this study proposes a Nonlinear Adaptive Whale Optimization Algorithm (NAWOA), which integrates strategies such as Good Nodes Set initialization, Leader-Followers Foraging, Dynamic Encircling Prey, Triangular Hunting, and a nonlinear convergence factor to enhance exploration, exploitation, and convergence stability. Experiments on 23 benchmark functions demonstrate NAWOA's superior optimization capability and robustness. Based on this optimizer, an NAWOA-XGBoost model was developed to predict academic potential using data from 495 Computer Science undergraduates at Macao Polytechnic University (2009-2019). Results show that NAWOA-XGBoost outperforms traditional XGBoost and WOA-XGBoost across key metrics, including Accuracy (0.8148), Macro F1 (0.8101), AUC (0.8932), and G-Mean (0.8172), demonstrating strong adaptability on multi-class imbalanced datasets.
With the rapid development of Educational Data Mining (EDM) [1], predicting students' future academic potential based on their early learning performance has become an important research direction for quality assurance, resource allocation, and personalized instruction in higher education. In the field of Computer Science (CS), students' academic records and behavioral data during their first and second years often provide strong indications of their foundational knowledge, learning capability, and future development potential. Therefore, establishing a reliable early academic potential prediction model is of significant importance.
In recent years, machine learning methods have been extensively applied to educational prediction tasks. Among these methods, XGBoost has been regarded as a suitable model due to its strong generalization ability and superior performance on structured data. However, the performance of XGBoost highly depends on its hyperparameter settings. Manual tuning is inefficient and prone to local optima, which restricts further improvements in predictive performance. To address this limitation, researchers started to pay attention on metaheuristic algorithms. Metaheuristic algorithms, which are famous of their strong optimization abilities, have been introduced to path planning [2][3] [4], engineering design, antenna design [5], and optimizing the hyperparameters of machine learning models. Among them, the Whale Optimization Algorithm (WOA) has attracted attention due to its simplicity, efficiency, and global search capability.
Nevertheless, the conventional WOA tends to suffer from premature convergence and loss of population diversity in later iterations, limiting its ability to effectively explore complex parameter spaces.
Previous studies have integrated the classical WOA with XGBoost and achieved promising prediction results [6]. However, there remains considerable room for improvement. Building upon this foundation, the present study proposes a Nonlinear Adaptive Whale Optimization Algorithm (NAWOA), which incorporates strategies such as Good Nodes Set initialization, Leader-Followers Foraging, and Dynamic Encircling Prey to enhance global exploration ability and optimization stability. Experiments conducted on 23 benchmark functions validate the strong optimization capability of the proposed algorithm [7] [8]. Based on this enhanced optimizer, we construct an NAWOA-XGBoost academic potential prediction model and evaluate it using a real dataset of 495 undergraduate students majoring in Computer Science at Macao Polytechnic University (MPU) from 2009 to 2019. The dataset was preprocessed and provided by the institution, enabling direct use for training and comparative analysis.
Experimental results indicate that NAWOA can more effectively optimize the hyperparameters of XGBoost, thereby enhancing model stability and predictive performance in multi-class educational prediction tasks. The proposed model provides a valuable reference for early academic potential identification and personalized educational support in higher education institutions.
The Whale Optimization Algorithm (WOA), proposed by Mirjalili et al. in 2016, is a metaheuristic optimization technique inspired by the foraging behavior of humpback whales [9]. The algorithm models the whales’ cooperative hunting strategies observed in nature, particularly their distinctive bubble-net feeding mechanism. WOA simulates three core behavioral patterns of humpback whales: search-for-prey, spiral-updating during bubble-net feeding, and encircling the prey. Although WOA is structurally simple, easy to implement, and has demonstrated good performance in various optimization applications, it still exhibits several limitations. First, WOA tends to suffer from a decline in population diversity during the later stages of iteration, which weakens its global search capability. Second, the algorithm often shows slow convergence, limited optimization accuracy in complex or high-dimensional problems, and a tendency to become trapped in local optima. Moreover, achieving a proper balance between global exploration and local exploitation highly depends on parameter settings, making WOA sensitive to parameter variations and reducing its stability in real-world applications.
To address these shortcomings, this study proposes an enhanced multi-strategy Whale Optimization Algorithm (NA-WOA). The improved algorithm strengthens the balance between exploration and exploitation, increases search efficiency, and enhances optimization accuracy by integrating several newly designed strategies tailored to overcome the weaknesses of the original WOA.
3 The Proposed NAWOA
In the classical WOA, the initial population is typically generated using pseudo-random numbers, as illustrated on the left side of Figure 1. Although this method is easy to implement and can quickly produce an initial set of solutions, the resulting individuals are often unevenly distributed, leading to low population diversity and noticeable clustering.
To overcome these limitations, NAWOA adopts the Good Nodes Set method for population initialization [10]. The concept of the Good Nodes Set was introduced by Hua Luogeng, aiming to generate point sets with more uniform spatial distribution. A key advantage of this method is that its construction is dimension-independent-meaning that it maintains uniformity not only in two-dimensional spaces but also in high-dimensional environments. As shown on the right side of Figure 1, when the population size is set to N =200, the population generated by the Good Nodes Set exhibits a more homogeneous distribution compared to pseudo-random number initialization. This effectively avoids population clustering, increases coverage of the search space, and strengthens global exploration performance during early iterations.
Let U D denote the unit hypercube in a D-dimensional Euclidean space. The Good Nodes Set can be defined as:
where {x} denotes the fractional part of x, M is the number of nodes, r > 0 is an offset parameter, C(r, ฮต) is a constant depending on r, and ฮต > 0 is a given constant. , respectively. The mapping is performed as follows:
Through this mapping, the Good Nodes Set enables the generation of more uniformly distributed initial solutions within the search space, effectively enhancing the global exploration ability and search stability of NAWOA.
In the traditional WOA, the ‘search for prey’ process relies on randomly selecting an individual to update positions. Although this strategy increases stochasticity and diversity to some extent, it also introduces several limitations: the search directions of individuals lack consistency, convergence becomes unstable in later iterations, and the algorithm depends excessively on random factors, resulting in insufficient utilization of population information. Particularly in the late stage of optimization, individual updates often exhibit irregular oscillations, making the algorithm prone to premature convergence or entrapment in local optima.
To address these issues, this study draws inspiration from the natural hunting behavior of humpback whales, in which a leader guides the group toward the prey [11]. Accordingly, we propose the Leader-Followers Foraging strategy. This strategy introduces a Leader (the current global best individual) as the primary driving force and incorporates the population mean position to guide individuals toward more structured and direction-aware searches, thereby improving the stability and effectiveness of the optimization process.
The position update model of the Leader-Followers Foraging strategy is defined as follows:
where t is the current iteration number; T is the maximum number of iterations; X i is the position of the whale individual; X * denotes the position of the current best solution.
X R is the average position of all whale individuals:
where N is the population size.
Under this update mechanism, each whale individual is jointly influenced by the Leader’s position and the overall population distribution. The Leader provides a clear global convergence direction, while the population mean position constrains and regulates the search range, ensuring that the update process maintains both directionality and stability. This strategy effectively reduces instability caused by excessive randomness in the search process, enabling individuals to rely more on population structural information and preventing overly weak or overly strong mutual influence among individuals. As a result, the overall coordination of the population search is significantly enhanced. The Leader-Followers Foraging strategy strengthens global exploration while improving convergence stability, enabling a more precise and efficient optimization process.
In the original WOA, during the encircling prey phase, individual positions are updated mainly based on the Euclidean distance between the current best solution and each whale. Although this linear shrinking mechanism is simple, its search trajectories are relatively monotonic and lack flexibility. In the later stages of iteration, the movement amplitude becomes too small, making individuals prone to being trapped in local optima. Moreover, when the contraction coefficient A becomes small, the update range of individuals is significantly restricted, further reducing the algorithm’s ability to escape local regions and negatively affecting overall convergence performance.
To enhance the diversity and dynamic behavior of the encircling phase, this study introduces a spiral flight mechanism and proposes the Dynamic Encircling Prey strategy [12] [13]. This strategy incorporates a spiral flight step into the traditional encircling mechanism, enabling whale individuals to follow diverse and nonlinear trajectories when approaching the prey. This improves both the exploration capability and the ability to escape from local optima. The modeling of the Dynamic Encircling Prey strategy is as follows.
First, the distance between the whale individual and the Leader is calculated:
where A and C are coefficient vectors.
Then, a spiral-flight-based disturbance term is constructed using random variables:
where Z represents the Spiral flight step size; s and k are spiral coefficients; and r is a random number between 0 and 1.
Finally, the position update model is formulated as:
In this strategy, the spiral step size Z dynamically changes with the iterations, allowing whale individuals to exhibit different disturbance magnitudes at different optimization stages. The random variable L introduces directional diversity, generating rich encircling trajectories. Through this nonlinear flight mechanism, individuals can not only perform thorough local exploration when approaching the prey but also gain a higher probability of escaping local optima, thus avoiding premature convergence. The Dynamic Encircling Prey strategy significantly enhances flexibility during the encircling phase, equipping the algorithm with stronger dynamic exploration ability when approaching the optimal solution, thereby contributing to improved convergence accuracy and robustness.
In the original WOA, the spiral updating mechanism adjusts individual positions around the current best solution, guiding whales to approach the prey along a fixed spiral trajectory. Although this strategy mimics the bubble-net spiral feeding behavior of humpback whales, it becomes overly dependent on the current optimal solution in the later stages of iteration. The disturbance range becomes limited, restricting the ability of individuals to perform more complex spatial exploration. As the algorithm approaches the optimal region, the use of a single spiral path increases the risk of falling into local optima, limiting both fine-grained search capability and the ability to escape suboptimal regions.
To enhance the spatial exploration ability during the spiral hunting phase, this study proposes the Triangular Hunting strategy. By introducing dynamically scaled random disturbances, multi-directional angular deviations, and a triangular step length, whale individuals can generate richer and more complex movement trajectories when approaching the prey. This significantly improves both local search precision and the ability to escape local optima. The modeling of the Triangular Hunting strategy is as follows.
First, a dynamic scaling factor r 1 and a random scaling factor R are constructed:
where Rand denotes a random number between 0 and 1; r 1 gradually decreases as iterations progress, enabling wide-range exploration in early stages and more stable local exploitation in later stages.
The distance term consists of two parts: the straight-line distance to the optimal individual and a random step length:
A random angle ฮณ is introduced to generate asymmetric perturbations:
Then the triangular step length L is computed:
Finally, the position update model of Triangular Hunting is expressed as:
In this strategy, the triangular step L enables individuals to deviate from the traditional spiral path and explore in multiple directions, forming more flexible hunting trajectories. The dynamic scaling factor r 1 ensures a smooth transition from broad early exploration to refined late exploitation. The random angle ฮณ increases the asymmetry of perturbations, preventing individuals from collapsing into a single spiral path, thereby enhancing diversity and robustness. By combining triangular disturbances with spiral update behavior, the Triangular Hunting strategy enables the population to maintain a convergent trend while performing more comprehensive and flexible neighborhood exploration. As a result, the algorithm achieves significantly improved local optimization capability and a stronger ability to escape local optima.
Building upon the aforementioned enhancement strategies, although the search capability of the algorithm has been significantly improved, the linear convergence factor a used in the original WOA remains insufficient for meeting the regulatory requirements of the enhanced NAWOA. In the classical method, a decreases linearly from 2 to 0, and this relatively smooth transition results in limited differentiation of convergence behavior across different stages of the iteration process. In particular, the slow decline in the later iterations may restrict the algorithm’s convergence efficiency and search precision.
To achieve more flexible control over the exploration-exploitation balance and better align with the newly introduced mechanisms of NAWOA, this study adopts a nonlinear convergence factor update strategy based on a Sigmoid curve. This update pattern enables a to follow a ‘slow-fast-slow’ transition during the optimization process, which better matches the dynamic requirements of real-world optimization behavior. The nonlinear convergence factor is modeled as follows:
This Sigmoid-based update mechanism allows a to adjust dynamically according to the needs of different optimization phases, resulting in a more adaptive control of search intensity. It establishes a more reasonable balance among global exploration, convergence speed, and local exploitation, thereby enhancing the overall optimization performance and stability of the algorithm.
To evaluate the optimization capability of NAWOA, this study selected WOABAT [14], IWOA [15], and WOA [9] for comparison with NAWOA on the 23 benchmark functions listed in Table 1. The parameter settings for each algorithm were shown in Table 2. The number of iterations was uniformly set to T =500, and the population size was set to N =30. Each algorithm was independently executed 30 times on the 23 benchmark functions, and the average fitness (Ave) and standard deviation (Std) were recorded for performance analysis. The experimental results are shown in Figure 2 and Table 3.The results indicate that NAWOA achieves the overall best performance among all algorithms. Compared to the original WOA, NAWOA significantly improves overall performance. As shown in Figure 2 and Table 3, NAWOA attains the best results in terms of both average fitness and standard deviation for the majority of benchmark functions, demonstrating its strong adaptability and robustness in solving various types of optimization problems. To ensure fairness, all models were trained and evaluated under the same training-test split conditions.
The three methods compared in the experiments are:
โข XGBoost: Baseline model without hyperparameter optimization [16];
โข WOA-XGBoost: Hyperparameters optimized using the traditional WOA;
โข NAWOA-XGBoost: Hyperparameters optimized using the proposed NAWOA.
All models were evaluated using Accuracy, Macro Precision, Macro Recall, Macro F1, AUC, and G-Mean to comprehensively assess their adaptability to multi-class educational prediction tasks.
Table 4, Figure 3, Figure 4 and Figure 5 present the experimental results of the three methods. It can be observed that NAWOA-XGBoost achieves the best performance across all evaluation metrics, outperforming both XGBoost and WOA-XGBoost. These results indicate that strategies introduced in NAWOA, including Good Nodes Set initialization, Leader-Followers Foraging, Dynamic Encircling Prey, Triangular Hunting, and a nonlinear convergence factor, significantly enhance the algorithm’s exploration capability, ability to escape local optima, and convergence stability. Consequently, NAWOA-XGBoost demonstrates superior performance in predicting academic potential, accurately distinguishing students with different levels of potential. The experimental results fully validate the effectiveness of NAWOA in optimizing XGBoost, confirming the practical value and superior performance of the proposed method.
This study addresses the limitations of the traditional Whale Optimization Algorithm (WOA) in terms of search capability, convergence speed, and ability to escape local optima by proposing an improved algorithm, NAWOA. The algorithm incorporates multiple strategies, including Good Nodes Set initialization, Leader-Followers Foraging, capability on multi-class imbalanced datasets. Overall, the results demonstrate that the proposed NAWOA not only significantly enhances hyperparameter optimization but also improves the stability and generalization ability of predictive models, underscoring its research significance and practical value.
Despite NAWOA’s promising performance, several directions warrant further investigation. First, this study primarily validated the algorithm on continuous optimization problems and structured educational data; future work could extend NAWOA to more complex problem types, such as combinatorial optimization, multi-objective optimization, and dynamic optimization environments, to further assess its generality. Second, the strategies in NAWOA are currently based on fixed mechanism designs; future research could explore adaptive parameter tuning or deep learning-assisted dynamic strategy generation to enhance the algorithm’s adaptability across diverse task environments. In addition, the current study relies on a single dataset; extending the validation to multi-institutional, multi-major, or cross-regional educational datasets would improve the model’s universality and robustness. Finally, NAWOA-XGBoost could be further combined with graph neural networks, temporal models, or more advanced ensemble methods to explore additional potential applications in educational assessment, student behavior analysis, and learning path prediction.
4.2 Comparison experiment on metaheuristic algorithms combining XGBoost 4.2.1 Dataset Description This study uses the student dataset from the Computer Science program at Macao Polytechnic University (MPU) covering the years 2009-2019, consistent with datasets used in related literature. The dataset contains multi-dimensional information of 495 undergraduate students, including academic performance, learning behavior, and course outcomes, and classifies students into different academic potential levels. The data were preprocessed by the university and are ready for model training and validation. The dataset has the following characteristics: โข Multi-class classification problem: Different academic potential levels constitute a three-class prediction task; โข Diverse feature dimensions: Includes structured features such as course grades and learning trajectories; โข Real educational scenario data: Sourced from authentic teaching environments, providing representativeness and research value.
