Estimating the distance at which narwhal $( extit{Monodon monoceros})$ respond to disturbance: a penalized threshold hidden Markov model

Understanding behavioural responses to disturbances is vital for wildlife conservation. For example, in the Arctic, the decrease in sea ice has opened new shipping routes, increasing the need for impact assessments that quantify the distance at which marine mammals react to vessel presence. This information can then guide targeted mitigation policies, such as vessel slow-down regulations and delineation of avoidance areas. Using telemetry data to determine distances linked to deviations from normal behaviour requires advanced statistical models, such as threshold hidden Markov models (THMMs). While these are powerful tools, they do not assess whether the estimated threshold reflects a meaningful behavioural shift. We introduce a lasso-penalized THMM that builds on computationally efficient methods to impose penalties on HMMs and present a new, efficient penalized quasi-restricted maximum-likelihood estimator. Our framework is capable of estimating thresholds and assessing whether the disturbance effects are meaningful. With simulations, we demonstrate that our lasso method effectively shrinks spurious threshold effects towards zero. When applied to narwhal $\textit{(Monodon monoceros)}$ movement data, our analysis suggests that narwhal react to vessels up to 4 kilometres away by decreasing movement persistence and spending more time in deeper waters (average maximum depth of 356m). Overall, we provide a broadly applicable framework for quantifying behavioural responses to stimuli, with applications ranging from determining reaction thresholds to disturbance to estimating the distances at which terrestrial species, such as elephants, detect water.

💡 Research Summary

This paper addresses a pressing conservation need: quantifying the distance at which marine mammals, specifically narwhals (Monodon monoceros), respond to vessel disturbance in a rapidly changing Arctic environment. Traditional approaches rely on controlled exposure experiments, which are rarely feasible in the wild, and on threshold hidden Markov models (THMMs) that require computationally intensive grid searches for each candidate threshold. Moreover, existing THMMs lack a principled way to determine whether the “disturbed” regime truly reflects a behavioural shift or is merely a statistical artefact.

The authors propose a novel penalized THMM that integrates an L1 (lasso) penalty on both the regression coefficients governing state transitions and on the threshold parameter itself. By treating the threshold as a continuous parameter and approximating the step function with a smooth logistic curve, the need for exhaustive grid searches is eliminated. The lasso penalty is interpreted as a Laplace prior, encouraging sparsity: spurious threshold effects are shrunk to zero, while genuine effects remain. To estimate the penalty strength efficiently, the authors embed the model within a quasi‑restricted maximum likelihood (qREML) framework and employ a Laplace approximation of the marginal likelihood, dramatically reducing computational overhead compared with bootstrap likelihood ratio tests.

Mathematically, the model comprises two regimes—baseline (B) and disturbed (D)—each with its own transition probability matrix (TPM). Hidden states Sₜ follow a Markov chain; transition probabilities Γ^{(k)}{t,ij} are modeled via a multinomial logit link with covariates ω{m,t} (e.g., vessel distance) and coefficients α^{(k)}{m,ij}. The regime‑mixing probability ν{β₀}(uₜ) = 1_{uₜ > 1/β₀} (or its logistic approximation) determines when the process switches from B to D based on the exposure variable uₜ. The full likelihood combines the two regimes weighted by ν_{β₀}(uₜ).

A comprehensive simulation study explores scenarios with no true threshold, true thresholds at 2 km, 4 km, and 6 km, and varying sample sizes (T = 500–2000). Results show that the lasso‑penalized estimator accurately recovers true thresholds, while shrinking nonexistent thresholds to zero, thereby avoiding false positives. The method also demonstrates robustness to different penalty strengths, with qREML efficiently selecting the optimal λ.

The empirical application uses telemetry from 11 narwhals tagged in the Qikiqtaaluk (Baffin) region during summer 2017, yielding 8,603 location points at 30‑minute intervals. For each location, step length, turning angle, and maximum dive depth (sampled every 75 s) are derived. Vessel positions are obtained from AIS data, corrected to 1‑minute resolution, and the inverse distance (km⁻¹) to the nearest vessel is calculated for each whale fix.

Applying the penalized THMM, the estimated disturbance threshold is β₀ ≈ 0.25 km⁻¹, corresponding to a physical distance of about 4 km. When vessels are within this range, narwhals exhibit a statistically significant reduction in movement persistence (i.e., lower probability of remaining in the same behavioural state) and an increase in deep diving, with average maximum depth rising to 356 m. These findings align with earlier, more qualitative reports of narwhal avoidance of vessel noise but provide the first quantitative, model‑based distance metric.

Key contributions of the paper are:

1. Methodological Innovation – Integration of lasso penalization into THMMs, eliminating costly grid searches and enabling simultaneous variable selection for transition coefficients and the disturbance threshold.
2. Efficient Estimation – Use of Laplace approximation within a qREML framework to estimate the penalty parameter, offering a computationally tractable alternative to bootstrap likelihood ratio tests.
3. Model Validation – Demonstration that the penalized approach can distinguish genuine behavioural regime shifts from spurious mixture components, addressing a long‑standing limitation of mixture‑model selection.
4. Ecological Insight – Provision of a concrete, species‑specific reaction distance for narwhals, directly informing mitigation measures such as speed limits, vessel routing, and designated avoidance zones.

The authors acknowledge limitations: the model assumes a single, abrupt threshold, which may oversimplify more gradual or multi‑threshold responses; the choice of λ, while efficiently selected, can still be data‑dependent; and measurement error in location and depth is not explicitly modeled. Future work could extend the framework to multiple thresholds, incorporate non‑linear smooth functions, embed observation error models, or adopt a fully Bayesian implementation with MCMC to obtain posterior distributions for all parameters.

In summary, the lasso‑penalized THMM offers a powerful, computationally efficient tool for extracting disturbance thresholds and associated behavioural changes from high‑resolution telemetry data. Its application to narwhals yields actionable conservation information and sets the stage for broader use across taxa and disturbance types.