REBEC: Robust Evolutionary-based Calibration Approach for the Numerical Wind Wave Model

REBEC: Robust Ev olutionary-based Calibration Approac h for the Numerical Wind W a v e Mo del P av el Vyc huzhanin 1 , Nik olay O. Nikitin 1 , and Anna V. Kalyuzhnay a 1 ITMO Univ ersity , 49 Kron verksky Pr. St. P etersburg, 197101, Russian F ederation { pavel.vychuzhanin, nikolay.o.nikitin } @gmail.com Abstract. The adaptation of numerical wind w av e mo dels to the local time-spatial conditions is a problem that can be solv ed b y using v arious calibration tec hniques. Ho wev er, the obtained sets of ph ysical parameters b ecome ov er-tuned to sp ecific even ts if there is a lac k of observ ations. In this pap er, we prop ose a robust evolutionary calibration approach that allo ws to build the sto c hastic ensemble of p erturbed mo dels and use it to achiev e the trade-off b et w een quality and robustness of the target mo del. The implemen ted robust ensemble-based ev olutionary calibration (REBEC) approach was compared to the baseline SPEA2 algorithm in a set of exp erimen ts with the SW AN wind wa ve mo del configuration for the Kara Sea domain. Provided metrics for the set of scenarios confirm the effectiveness of the REBEC approach for the ma jorit y of calibration scenarios. Keyw ords: · Evolutionary algorithm · SW AN wind w av e model · en- sem ble mo delling · robust optimisation · mo del calibration 1 In tro duction The v arious tasks of offshore dev elopment and coastal shipping mak e it neces- sary to use the regional configurations of the n umerical wind wa ve models to repro duce historical extreme even ts and predict p oten tial hazards. T o obtain the forecasts and hindcasts of desired quality , the suitable physical parameters of mo dels should b e iden tified for the sp ecific simulation conditions. The n umerical mo del calibration of o cean wind wa ve mo del inv olves the fitting of sim ulation results with the in-situ and satellite wa v e measuremen ts. The purp ose of calibration is the identification of the physical parameters set that allows minimising the discrepancy b et w een the mo del and observ ations. Ho wev er, it is a sophisticated task to calibrate the mo del man ually even with the meto cean experts’ inv olvemen t. The mo dern wind wa ve mo dels are computationally in tensive, and each simulation run can take hours to compute. Also, a dramatically low time-spatial co verage of the av ailable historical wa ve measuremen ts and lo w quality of atmospheric reanalyses in some regions (like the Arctic seas, in particular the Kara Sea region describ ed in the pap er) makes it hard to v alidate the parameter set reliably . The obtained parameters with minimal discrepancy can be v ery sp ecialised in case of o ver-fitting to the low 2 P av el Vych uzhanin et al. n umber of observed data points and can actually decrease the quality of long- term simulation results in non-observed lo cations or time ranges [4]. There are man y well-kno wn optimisation approac hes that can b e applied to automate the parameters’ tuning for environmen tal models as w ell as [9]. Despite this, in the pap er a task-sp ecific robust ev olutionary algorithm is prop osed. It al- lo ws to make reliable calibration decisions in situations with high en vironmental uncertain ty and tries to ensure a tolerable solution iden tification. A t the momen t, the mo dern atmospheric reanalysis still has qualit y issues in the Arctic region [8]. W e prop osed an algorithm that establishes artificial div ersity for wind v elo cit y fields. It w as used to generate the probabilistic en- sem ble of input wind fields to take the impact of the surface forcing uncertaint y in to account. Then, the m ulti-ob jectiv e fitness function was used to achiev e the trade-off b etw een robustness and p erformance of the optimised mo del. W e conducted a set of exp erimen ts to verify the effectiveness of the prop osed approac h against the baseline SPEA2 algorithm using the Kara Sea domain and the SW AN (Simulating W Aves Nearshore)[3] mo del as the case study . The nine spatially distributed points w ere chosen to analyse the p erformance and robustness of the mo del’s configurations obtained after calibration in one-month training runs of the mo del. The several configurations with different subsets of calibration and v alidation p oin ts w ere compared to estimate the statistical metrics of optimisation effectiveness for b oth algorithms. This pap er is structured as follows. Sec. 2 describ es the problem statement and mathematical formalisation of the robust optimisation task. Sec. 3 provides an ov erview of v arious calibration approaches and their applicability for the problem. Sec. 4 con tains a detailed description of the baseline SPEA2 algorithm and the proposed robust algorithm. Sec. 5 is dedicated to the experimental stud- ies (mo del configuration, datasets, results and metrics). Sec. 6 summarises the obtained results and highlights of the key findings. 2 Problem statemen t As it was noted in introduction, co verage of observ ed met-o cean data (espe- cially o ceanic observ ations) is extremely sparse. Although, reliable information ab out met-o cean c haracteristics is needed in many regions (e.g. Arctic seas). That’s why during last decades it b ecame a common practice to obtain the in- formation ab out met-ocean even ts and pro cesses from forecasting or hindcasting (retrosp ectiv e) simulation results from numerical hydrodynamic mo dels. Never- theless, for solving such task the numerical mo dels should b e fitted (through mo del parameters) to the certain water area. T aking in to accoun t few spatial p oin ts and small sizes of datasets with observ ations, there is a serious risk of mo del ov erfitting when model fits to sp ecific features of observed data instead of fitting to common features of the target region. Description of the solution to this problem is the main goal of this article. Hydro dynamic mo del fitting through the tuning of mo del parameters (or mo del calibration) can b e form ulated as an optimisation task. F or this purp ose, REBEC: Calibration Approach for the Numerical Wind W av e Mo del 3 it is reasonable to present the simulation pro cess in a general mathematical notation (1). Y = { Y 1 , Y 2 , ..., Y k } = M ( ξ | θ ) , (1) where Y = { Y 1 , Y 2 , ..., Y k } denotes multiv ariate output data (simulated fields, e.g. wa ve heights), M ( • ) is the mo del op erator, ξ is the input data (b oundary and initial conditions), θ is the set of mo del parameters. With that, the tuning of model parameters (or mo del calibration) can be formalized in terms of multi-ob jective optimisation in the model parameter space and written as: θ opt = arg min θ F ( θ ) , F ( θ ) = G ( f i ( θ , Y , { x, y } )) , (2) where G ( • ) is an op erator for multiob jective transformation to F , f i is the ob jective function, i = 1...n , { x, y } are spatial co ordinates of a p oin t-of-interest. In a case of wind wa ves hindcasting, the p oor time and spatial cov erage of observ ations make the mo del optimisation m uch harder. The ov er-fitting of the solution to the specific ev ents represented in small data samples can cause a non-optimal mo del configuration with lo wer qualit y under different external conditions. One of the wa ys to improv e the robustness of optimisation results is to enlarge training dataset with new instances with relatively small artificial disturbances. This issue mak es it necessary to tak e the sim ulation uncertain ty factors into account. The uncertaint y in the wind wa v e mo del can b e represen ted not only by dis- turbances in design v ariables [19]. There are deviations in the environmen t v ari- ables that can b e represented through input data sets diversit y (for the SW AN mo del the wind forcing obtained from atmospheric reanalysis is most imp or- tan t). In this case input data ξ should b e transformed to ensem ble realisation { ξ } n = { ξ 1 , ..., ξ n } b y addition of artificial disturbance (or noise) and equation (2) transforms into equation (3). A detailed description of the ensemble pro ce- dure is given in Sec. 4.3. θ rob = arg min θ ˜ F ( θ | { ξ } n ) , ˜ F ( θ | { ξ } n ) = G ( ˜ f i ( θ | { ξ } n , Y , { x, y } )) . (3) An ensemble ob jective function ˜ f i defines landscap e of ob jective function o ver the space of parameters considering ensem ble of input states { ξ } n . As an example, ensemble fitness function can b e represented by the exp ected function for the ensemble of runs with small disturbances in input data (shown in equation (4)). This approac h can b e used to pro duce better solutions for the set of diverse en vironmental scenarios and increase the exp ected p erformance. ˜ f ( θ | { ξ } n ) = Z ∞ −∞ f ( x , ξ + δ ) · p ( δ ) d δ (4) 4 P av el Vych uzhanin et al. As an example of the h ydro dynamic mo del for experimental studies, third- generation wind wa ve mo del SW AN [3] was chosen. The wind wa v es are surface w av es in the o ceans and seas that caused by the interaction b et ween w ater masses and sea-level wind. Wind wa v es mo dels of third-generation (e.g. SW AN) allo w to sim ulate the w av e sp ectra and to reconstruct c haracteristics of w av es (e.g. heights, p erio ds, directions). The SW AN mo del can b e describ ed with the action balance equation (5). ∂ ∂ t N + ∂ ∂ x c x N + ∂ ∂ y c y N + ∂ ∂ σ c σ N + ∂ ∂ θ c θ N = S σ , (5) where on the left-hand side N = E σ denotes the wa ve action density and E is an energy of wa ve sp ectrum, σ is the relative frequency , θ is the group w av e direction, c is the group velocity in corresponding space. The right-hand side represen ts the source and sink term in a form equation (6). S = S in + S ds + S nl , (6) where S in is the input energy obtained by wind, S ds is the energy of dissipation and S nl denotes the energy of wa v e-wa v e nonlinear interaction. These three terms represent the genesis of wa ve energy sources/sinks and are a pow erful handle for wa ve model fitting. F rom this p oint of view, it is con venien t to express energy sources through mo del parameters. Wind energy is c haracterised b y the drag function (DR G), wa v e dissipation — by the w av e breaking (STMP) and b ottom friction (CFW) functions. Energy flow from non- linear interactions is relativ ely small and wasn’t taken in to accoun t in the current pap er. In the frame of this article, the experimental study (Sec. 5) was provided to assess the practical effectiveness of the prop osed robust calibration method in comparison with the general-purp ose calibration algorithms. The SW AN mo del configuration for the Kara Sea w as c hosen as a case study b ecause of v alue of this region for offshore industrial developmen t and extremely low densit y of sensors in areas of interest. 3 Related w ork Mo del calibration or tuning is a sub ject with extensive literature [25,9]. The conserv ativ e approach is to estimate the parameters in an exp ert w ay [11,17]. It includes the dev elopment of several candidate sets of paramete rs based on previous simulation exp erience and man ual individual adjustment of each pa- rameter. The qualit y metrics for the model quality assessmen t are calculated with the comparison of model time series and historical v alues obtained from the reanalyses and observ ations. Since the man ual ”trials-and-errors” metho d is time-consuming and giv es so- lution only for particular mo del setup, the automatic calibration of mo dels is widely used for different asp ects of environmen tal simulations lik e atmospheric REBEC: Calibration Approach for the Numerical Wind W av e Mo del 5 [7] and o cean [26] forecasting tasks. As a basic approac h, the space-filling de- sign for the parameter space can b e used [24] for model calibration. How ever, the high-resolution configurations of wind wa v e mo dels of 3rd generation are computationally-in tensive and require a lot of time to pro cess the appropriate date range and spatial domain. This problem makes it necessary to reduce the n umber of runs required for calibration. There are many well-kno wn optimisation metho ds applied to en vironmen- tal models like deriv ativ e-free optimisation [23], v arious Bay esian optimisation metho ds [6] and surrogate-assisted metho ds [10]. Ho wev er, the evolutionary (genetic) algorithms are efficient enough to per- form a robust solution searc h [15] in a complex parameter space with a lack of historical data for qualit y assessmen t [21]. The applicabilit y of evolutionary algorithms for SW AN wa ve mo del calibration is demonstrated in [13]. The robust optimal design approac hes hav e a lot of applications in many fields [27]. They are often based on Monte Carlo metho ds that allow representing the uncertain ty from different sources [5]. The p erturbation-based ensemble allows sampling the mo delling uncertaint y in a more systematic w ay [20]. A set of sim ulation with small differences induced by sto chastic mo difications allows to increase the v ariability of the calibration dataset and improv e the qualit y of mo dels [18]. Nev ertheless, the discrepancy usually simulated as additional noise in mo del output and observ ations[2] without taking the actual sources of external uncer- tain ty (e.g. wind forcing for reanalysis) in to accoun t. The task of a reliable cali- bration of a wa ve mo del for a sp ecific domain with p oor observ ational cov erage mak es it necessary to implement the approac h that combines the ensem ble-based div ersity of environmen tal v ariables with multi-ob jectiv e evolutionary optimisa- tion. 4 Ev olutionary algorithms for mo dels fitting W e compared the robust wa ve mo del calibration with a baseline solution — the m ulti-ob jectiv e evolutionary algorithm that estimates the most suitable solution without taking uncertaint y in to accoun t. The other approach is based on the same algorithm with mo dified fitness functions — it estimates the p erformance and robustness of the solution with the ensemble of forecasts obtained from sev eral mo del runs with noised inputs. The source co de of b oth algorithms was implemen ted in Python and av ailable in [1]. 4.1 Baseline approach The commonly used SPEA2 m ulti-ob jective optimisation algorithm [28] was cho- sen as a baseline solution for the calibration task. In terms of evolutionary al- gorithms, in our case, each individual corresp onds to a genot yp e represented b y a certain set of mo del parameter and the phenotype (v alues of the ob jec- tiv e function) are the errors of the mo del predictions, corresp onding to these 6 P av el Vych uzhanin et al. parameters. At each iteration of ev olution, the Pareto-optimal set of individuals is selected according to the v alues of the fitness function, and all non-dominated solutions are sa ved in the archiv e. Then the mating p o ol is filled with a binary tournamen t selection and recombination and a mutation op erator are applying for each individual. The resulting mating p ool b ecomes a new p opulation at the next iteration of the algorithm. Despite the fact that some modern evolutionary algorithms outperform SPEA2 in some syn thetic tasks [14], w e decided to base the exp eriments on a well-studied [13] algorithm to separate the impact from the prop osed ensemble-based mo di- fications from other features’ influence. 4.2 Robust ensem ble-based ev olutionary calibration (REBEC) approac h The main disadv antage of the baseline algorithm is that the mo del v ariables optimise exactly for the sp ecific conditions that w ere used for the fitness function ev aluation. It allows to maximise the p erformance for the observed case, but the solution found can b e unstable even after small changes in external conditions. The lack of the time-spatial co verage of observ ational data for wa v e parameters in target regions mak es it complicated to tak e the differen t external uncertain ties (e.g. forcing-induced, resolution-induced, etc) into accoun t. The more robust approac h to mo del parameters optimisation can b e imple- men ted using the ensemble of wa ve mo dels configured using different input data sets. W e can form the sto c hastic ensem ble of wa ve mo dels with the p erturb ed wind forcings and search for more robust mo del parameters using this ensemble instead of a single mo del with certain forcing. F or this purp ose, w e can adapt the baseline SPEA2 algorithm (that w as in tro duced ab o ve) b y changing the fitness assignment strategy: for a giv en geno- t yp e, the set of phenot yp es corresp onding to the elemen ts of the ensemble is estimated and based on its v alues the robust metric is calculated. The flow chart of the prop osed algorithm is presented in Fig 2. It is imp ortan t to find a compromise b etw een p erformance and robustness of the obtained solution [12], so the fitness function for the algorithm is based on the comp osite estimation robustness and p erformance metrics. The performance can b e calculated as a v ector of ro ot-mean-square errors (RMSE) against obser- v ations for a set of target p oin ts, and the robustness can b e simulated in v arious w ays [16]. Fig. 1 depicts the set of the ensemble error surfaces that are used for metric calculation. W e tried to use the mean-v ariance as a robust metric, but it causes the domination of the solutions with low wind drag and, consequen tly , near-zero wind-induced v ariability . So, the ensem ble mean w as chosen as a trade-off metric. REBEC: Calibration Approach for the Numerical Wind W av e Mo del 7 Fig. 1. The landscap e of an ob jective function for a probabilistic ensemble The pseudo code of the final implementation of the robust algorithm is presen ted in Alg. 1. Input: Initialised ensemble, p opulationSize, arc hiveSize, crossov erRate, m utationRate Result: b est individual from archiv e p op ← InitPopulation( p opulationSize ) a rchive ← ∅ while not Conver genc eCriterion() do for individ in p op do ensObjectives ← ∅ for mo del in ensemble do ensObjectives [ mo del ] ← CalculateObj( individ , mo del ) end b estByObjectives ← TakeBestByMean( ensObjectives , ensAmount ) individ .ob jectives ← Mean( b estByObjectives ) end union ← archive + p op for individ in union do individ .fitness ← CalculateFitness( individ ) end a rchive ← TakeNonDominated( union , ar chiveSize ) matingP o ol ← BinaryTournamentSelection( archive , p opulationSize ) p op ← CrossoverAndMutation( matingP o ol , cr ossoverR ate, mutationR ate ) end Algorithm 1: The pseudo code of the implemen ted REBEC algorithm 8 P av el Vych uzhanin et al. Fig. 2. The main logical blo c ks and interconnections of prop osed robust evolutionary algorithm 4.3 Syn thetic input data generation with artificial noise T o implemen t the proposed probabilistic optimisation method, we developed the supplemen tary algorithm that allo ws to add sp ecific noise to wind velocity v ari- ables — U (eastw ard) and V (northw ard) vector comp onen ts from atmospheric reanalysis data that is used b y the wa v e mo del as an external forcing. The algorithm starts from uniform scattering of the randomly-located sources of artificial noise in the gridded data. T o obtain the realistic wind field after the application of noise, the time-spatial correlation terms are added to control the noise spreading from the source. The noise function for the wind vector comp onen t U pro duced by one noise p oin t can b e written as: f ∗ ( j, t ) = N (0 , σ ) · corr ( U j , V j ) · corr ( U t , U t − 1 ) (7) where j is the spatial index of source p oin ts, t is the time step index, σ is the standard deviation parameter of the Gaussian distribution, U is the matrix of wind U-comp onents. Then, the aggregated noise from N source p oin ts for sp ecific data p oin t in- duced by all source p oin ts can b e obtained as: f ( i, t ) = N X j =1 f ∗ ( j, t ) · corr ( U i , U j ) (8) where i is the spatial index of the data p oin t, j is the spatial index of the noise p oin t, t is the time step index, N is the n umber of noise points, U is the matrix of wind U-comp onen ts. The example of the wind field augmented with noise b y the describ ed metho d (with σ equal to 25% of basic v alue magnitude) is presented in Fig. 3. It can b e seen that the common wind patterns are similar but some wind sp eed v ariabilit y exists. The additional p ost-pro cessing procedure w as applied to the perturb ed model runs output to suppress the non-realistic wind heigh t peaks in the observ ed calm p eriods. How ev er, the near-p eaks v ariability w as preserved. REBEC: Calibration Approach for the Numerical Wind W av e Mo del 9 Fig. 3. The example of comparison of (a) basic ERA-Interim for Kara Sea region and (b) wind field augmented with noise for the same region. The blue marks depicts the noise source p oints lo cations. In this wa y , the ten wind data sets augmented with artificial noise were generated and used in an exp erimen tal study . 5 Exp erimen tal study The case study for the calibration task is based on the SW AN mo del configu- ration for the Kara Sea region. The significant wa v e heigh t (Hsig) v ariable was c hosen as a target v ariable. Moreov er,the results in nine represen tative p oin ts w ere analyzed (P1-P9 presented in Fig. 4) to take into account p ossible spatial v ariabilit y of the optimal solution. 5.1 Syn thetic data for w a ve observ ations The w av e observ ations data are required for the v alidation of the mo del qual- it y and calibration algorithm effectiveness. How ever, such data often cannot b e obtained from op en data sources. T o p erform a repro ducible exp erimen t with Kara Sea configuration, we used the simulation results from the high-resolution W a veW atc h I I I mo del [22] configuration. The systematic biases of synthetic ob- serv ations against mo del were remov ed. Then we analysed the error metrics for the significant w av e height v ariable against real observ ations in p oin ts 1-3 (RMSE is 0.29m and MAE is 0.21m). W e accept the quality of the W av eW atc h I II output as sufficient to b e used as the reference dataset for the optimisation algorithms’ ev aluation. T o main tain the v ariability of exp erimental scenarios, we prepared 18 subsets of syn thetic observ ational p oin ts to b e used for calibration. They consist of 10 P av el Vych uzhanin et al. observ ation p oin ts lo cated in v arious spatial areas with differen t depths and distances to the coast. T o repro duce v arious scenarios, the calibration subsets w ere initialised with random p oint groups of a certain size: from a single-p oin t situation to the all-p oin ts-instead-one case. F or eac h subset, the p oints that were not used during the calibration w ere assigned to v alidation sets. 5.2 Mo del configuration The SW AN mo del was configured with the regular curvilinear grid in cartesian co ordinates. The initial conditions w ere obtained for a preliminary mon thly spin- up run. The boundary conditions were not set (since the control p oin ts were distanced from the grid b oundaries, see Fig. 4). The sim ulation dates range was set from 20140814.120000 to 20140915.000000. The time step for in tegration w as defined as 120 min and output time step is 3 hours. The parameterisations GEN3, COLLINS, QUADR UPL, TRIAD and DIFFRA Ction were enabled. The output was configured to obtain the significan t wa ve height (HS) v alues in 9 spatial p oints. Their locations are sp ecified in Fig. 4. Fig. 4. The part of the bathymetry of the simulation domain: the Kara Sea and Ob ba y . The land cells are shaded with a gray mask. The locations of observ ation p oin ts and their indices are sp ecified with green marks. 5.3 Sensitivit y analysis The sensitivity analysis of model parameters described in Sec. 2 w as p erformed to estimate their significance. W e ran the set of exp erimen ts with ev ery parameter REBEC: Calibration Approach for the Numerical Wind W av e Mo del 11 indep enden tly modified b y additiv e noise with Gaussian distribution with σ / µ = 0.25 assumption. The comparison diagram of model parameters’ relative v ariance and relativ e av eraged RMSE metric are presen ted in Fig. 5. The b o xplots are obtained from 50 exp erimen ts with sequential noising of each v ariable from the c hosen set. Fig. 5. The b o xplots of relativ e parameter v ariance (IN prefix) and relative a veraged RMSE of mo del results (OUT prefix) obtained after 50 runs. The CFW is the Collins b ottom friction co efficien t, DRG is the wind drag function, STMP is wa ve steepness. It can b e seen that the wind drag is the most sensitive parameter with a high relative output and input v ariance ratios, the wa v e steepness is the second one and the sensitivity b ottom friction co efficien t is quite low for most of the comparison p oin ts. In can b e concluded that the SW AN model error function has a wide ”plateau” with similar error v alues and many lo cal minim ums in the ”v alley” area that can affect the algorithm’s conv ergence and robustness. 5.4 V alidation of REBEC approac h A set of exp erimen ts was conducted to compare the results of optimisation ex- p erimen ts. The initial p opulation for b oth calibration approaches was pro duced using Latin hypercub e sampling (LHS) in the parameter space. The parameters for b oth calibration algorithms was c hosen as: p opulation size is set to 20 indi- viduals, the num b er of generations — 60, the archiv e size — 5 individuals, the probabilit y of mutation and crossov er — 0.2. Tw o ob jective functions w ere chosen for mo del results quality assessment: the mean absolute error (MAE) and ro ot mean square error (RMSE). The calibration for every scenario was rep eated 100 times to obtain the distri- bution of the relativ e impro vemen t of RMSE and MAE against the mo del config- uration with default parameter v alues (DRF=1.0, CFW=0.015, STPM=0.00302). Also, the mean relativ e standard deviation of the calibrated parameters set is pro vided. The b o xplots for the scenario 3 are presen ted in Fig. 6. It can b e seen that the v ariance of metrics for the robust algorithm is low er and the quality is b etter. The detailed metrics for all scenarios and stations sets are provided in T able 5.4. As can b e seen, the robust approach pro vides a b etter or equiv alen t improv e- men t of mo del p erformance for the v alidation p oin ts in all groups of scenarios. 12 P av el Vych uzhanin et al. Fig. 6. The comparison of the baseline and robust algorithms’ p erformance on the v alidation set of stations in all scenarios. The RMSE, MAE, peak-RMSE and p eak- MAE metrics are presen ted as an improv emen t against the corresp onding v alues for the default configuration. T able 1. Error metrics for the baseline and robust algorithms. The ”test” blo c k con- tains the metrics for the verification p oin ts. The ”train” blo c k contains the metrics for the calibration p oin ts. The b oldface num b ers indicate the b est metrics for all station sets (the higher improv ement and low er standard deviation is better) Scenario Algorithm V alidation p oin ts Calibration p oints Impro vemen t, % P ar. SD Impro vemen t, % P ar. SD RMSE MAE RMSE MAE Mean Max SD Mean Max SD Mean Max SD Mean Max SD 1-9 BL 11 22.1 7.8 -9.2 1.8 7.1 3.3 6.6 26.4 13.7 2.1 13.6 6.8 3.3 RB 11 15 2.7 4 6.9 2 2.7 11.3 16.4 3 2.5 6.5 3 2.7 10-14 BL 16.6 23.6 4.9 -7.2 0.7 5.2 2.7 24.4 29.1 4.1 7.9 12.8 2.7 2.7 RB 18.1 21 2.8 4.1 9.4 2.5 2.4 22.9 26.9 3.9 8.7 14.4 2.2 2.4 15-18 BL 17.6 25.1 5.7 -6.6 2.8 6.5 3.1 27.2 34.2 6.2 11.2 17.1 2.8 3.1 RB 18.7 24.7 4.1 5.4 10 4.1 3 23.4 33.4 5.1 11.2 17.9 3.2 3 All BL 14.0 23 6.6 -8.4 1.7 6.4 3.2 14.5 27.9 9.7 4.7 13.5 4.9 3.2 RB 14.6 18.3 3.1 4.6 8.7 2.7 2.8 15.8 21.6 3.9 5.2 10.5 2.9 2.8 REBEC: Calibration Approach for the Numerical Wind W av e Mo del 13 The standard deviation for b oth mo del parameters and relative improv ement v alues are also low er than the baseline. In can be concluded that the optimal algorithm choice for v alidation p oints v aries in different scenarios. The scenarios 1-9 operate with a single-point calibration set. The performance of the robust algorithm for this group of v alidation p oin ts is similar to baseline RMSE (but outp erforms it for the MAE metric and calibration p oin ts metrics). F or the other scenarios, the gain are near 1-2% RMSE and 10% MAE against the baseline. Also, the calibration set quality av eraged for all scenarios for the robust approac h also outp erforms the baseline. The standard deviation of the obtained metrics is smaller for all scenarios, as well as the mean standard deviation for mo del parameters. W e can claim that a robust approach is effective for the cases with several spatially scattered p oints that can b e applied for calibration. It is imp ortan t to notice that the c alibration points’ quality is not affected in a negativ e wa y . 6 Conclusion In the pap er, the practical approac h to the calibration of numerical w av e mo dels under data quality and av ailability constrain ts w as prop osed. The algorithm for the sim ulation of artificial data diversit y w as implemen ted and applied to the ERA-Interim reanalysis wind data. The regional configuration of the SW AN mo del was used as a case study for the parameters tuning algorithm effectiv eness ev aluation. The prop osed REBEC approach was compared with the baseline SPEA2 al- gorithm in a set of exp erimen ts. The low er v ariabilit y and b etter p erformance metrics for the spatially distributed calibration and v erification p oin ts w ere ob- tained. It confirms the effectiveness of the robust calibration approac h for the sim ulation domains with a small num b er and p oor cov erage of real observ a- tions. How ever, the negativ e impact of the prop osed approach for computa- tional p erformance (several simulations should b e p erformed for each candidate parameters set) makes the robust optimisation p otentially non-preferable for the mo del configurations with the sufficient spatial co verage of observ ations and high-qualit y atmospheric reanalyses. The source co de of the algorithms for calibration, pre- and p ost- pro cessing as well as the configuration files for SW AN are av ailable in an op en rep ository [1]. 7 Ac kno wledgements The European Center for Medium-Range W eather F orecasts (ECMWF) is ac- kno wledged for providing ERA-In terim surface wind data. References 1. The source code of the robust evolutionary algorithm for swan mo del calibration (2019), https://github.com/J3FALL/SwanEvolution 14 P av el Vych uzhanin et al. 2. Bhat, K.S., Haran, M., Go es, M., Chen, M.: Computer mo del calibration with m ultiv ariate spatial output: A case study . F ron tiers of Statistical Decision Making and Ba yesian Analysis pp. 168–184 (2010) 3. Bo oij, N., Ris, R.C., Holthuijsen, L.H.: A third-generation wa v e mo del for coastal regions: 1. mo del description and v alidation. Journal of geophysical research: Oceans 104 (C4), 7649–7666 (1999) 4. Brynjarsd´ ottir, J., OHagan, A.: Learning ab out physical parameters: The imp or- tance of mo del discrepancy . Inv erse Problems 30 (11), 114007 (2014) 5. Che, J., W ang, J., Li, K.: A monte carlo based robustness optimization metho d in new pro duct design pro cess: a case study . American Journal of Industrial and Business Managemen t 4 (07), 360 (2014) 6. Cornejo-Bueno, L., Garrido-Merch´ an, E.C., Hern´ andez-Lobato, D., Salcedo-Sanz, S.: Ba yesian optimization of a hybrid system for robust ocean wa ve features pre- diction. Neurocomputing 275 , 818–828 (2018) 7. Duan, Q., Di, Z., Quan, J., W ang, C., Gong, W., Gan, Y., Y e, A., Miao, C., Miao, S., Liang, X., et al.: Automatic mo del calibration: a new w ay to impro ve n umerical w eather forecasting. Bulletin of the American Meteorological So ciet y 98 (5), 959– 970 (2017) 8. F redriksen, L.E.: An ev aluation of the reanalyses ERA-Interim and ERA5 in the Arctic using N-ICE2015 data. Master’s thesis, UiT The Arctic Universit y of Nor- w ay (2018) 9. Hourdin, F., Mauritsen, T., Gettelman, A., Golaz, J.C., Bala ji, V., Duan, Q., F olini, D., Ji, D., Klock e, D., Qian, Y., et al.: The art and science of climate mo del tuning. B ulletin of the American Meteorological Society 98 (3), 589–602 (2017) 10. James, S.C., Zhang, Y., O’Donncha, F.: A machine learning framework to forecast w av e conditions. Coastal Engineering 137 , 1–10 (2018) 11. Jin, K.R., Ji, Z.G.: Calibration and verification of a sp ectral wind–wa ve mo del for lak e okeec hob ee. Ocean Engineering 28 (5), 571–584 (2001) 12. Jin, Y., Sendhoff, B.: T rade-off b et ween p erformance and robustness: an ev olution- ary multiob jectiv e approac h. In: international conference on Evolutionary Multi- Criterion Optimization. pp. 237–251. Springer (2003) 13. Ko v alch uk, S.V., Metsker, O.G., F unkner, A.A., Kisliako vskii, I.O., Nikitin, N.O., Kalyuzhna ya, A.V., V agano v, D.A., Bochenina, K.O.: A conceptual approach to complex mo del managemen t with generalized mo delling patterns and evolutionary iden tification. Complexity 2018 (2018) 14. Li, M., Y ao, X.: An empirical in vestigation of the optimality and monotonicity prop erties of multiob jectiv e arc hiving metho ds. Pro c. Evolutionary Multi-Criterion Optimization (EMO) (2019) 15. Liu, Y., Khu, S.T.: Automatic calibration of n umerical mo dels using fast optimi- sation by fitness approximation. In: Neural Netw orks, 2007. IJCNN 2007. Interna- tional Join t Conference on. pp. 1073–1078. IEEE (2007) 16. McPhail, C., Maier, H., Kwakk el, J., Giuliani, M., Castelletti, A., W estra, S.: Ro- bustness metrics: How are they calculated, when should they b e used and wh y do they giv e different results? Earth’s F uture 6 (2), 169–191 (2018) 17. Mortlo c k, T.R., Go odwin, I.D., T urner, I.L.: Calibration and sensitivities of a nearshore sw an mo del to measured and mo delled wa ve forcing at wam b eral, new south w ales, australia 18. O’Donnc ha, F., Zhang, Y., Chen, B., et al.: Ensem ble model aggregation using a computationally ligh tw eight machine-learning mo del to forecast o cean w av es. arXiv preprin t arXiv:1812.00511 (2018) REBEC: Calibration Approach for the Numerical Wind W av e Mo del 15 19. P aenke, I., Branke, J., Jin, Y.: Efficient search for robust solutions by means of ev olutionary algorithms and fitness approximation. IEEE T ransactions on Ev olu- tionary Computation 10 (4), 405–420 (2006) 20. Rougier, J., Sexton, D.M., Murphy , J.M., Stainforth, D.: Analyzing the climate sensitivit y of the hadsm3 climate mo del using ensembles from differen t but related exp erimen ts. Journal of Climate 22 (13), 3540–3557 (2009) 21. Sc hmitt, C., Rey-Co yrehourcq, S., Reuillon, R., Pumain, D.: Half a billion sim- ulations: Ev olutionary algorithms and distributed computing for calibrating the simp oplocal geographical mo del. Environmen t and Planning B: Planning and De- sign 42 (2), 300–315 (2015) 22. T olman, H.L., et al.: User manual and system do cumentation of wa vew atch iii tm v ersion 3.14. T echnical note, MMAB Contribution 276 , 220 (2009) 23. v an Vledder, G.P .: Calibration of swan 40.20 for field cases p etten, slotermeer and w esterschelde (2003) 24. W ainwrigh t, J., Mulligan, M.: En vironmental modelling: finding simplicity in com- plexit y . John Wiley & Sons (2005) 25. Williams, J.J., Esteves, L.S.: Guidance on setup, calibration, and v alidation of h ydro dynamic, wa ve, and sediment mo dels for shelf seas and estuaries. Adv ances in Civil Engineering 2017 (2017) 26. Williamson, D.B., Blaker, A.T., Sinha, B.: T uning without o ver-tuning: parametric uncertain ty quantification for the nemo o cean mo del. Geoscientific Mo del Dev el- opmen t 10 (4), 1789–1816 (2017) 27. Zang, C., F riswell, M., Mottershead, J.: A review of robust optimal design and its application in dynamics. Computers & structures 83 (4-5), 315–326 (2005) 28. Zitzler, E., Laumanns, M., Thiele, L.: Sp ea2: Improving the strength pareto evo- lutionary algorithm. TIK-rep ort 103 (2001)