A Bayesian latent class reinforcement learning framework to capture adaptive, feedback-driven travel behaviour

Many travel decisions involve a degree of experience formation, where individuals learn their preferences over time. At the same time, there is extensive scope for heterogeneity across individual travellers, both in their underlying preferences and in how these evolve. The present paper puts forward a Latent Class Reinforcement Learning (LCRL) model that allows analysts to capture both of these phenomena. We apply the model to a driving simulator dataset and estimate the parameters through Variational Bayes. We identify three distinct classes of individuals that differ markedly in how they adapt their preferences: the first displays context-dependent preferences with context-specific exploitative tendencies; the second follows a persistent exploitative strategy regardless of context; and the third engages in an exploratory strategy combined with context-specific preferences.

💡 Research Summary

The research presents a groundbreaking approach to modeling travel behavior by introducing the Latent Class Reinforcement Learning (LCRL) framework. Traditional Discrete Choice Models (DCM) often fail to account for the dynamic nature of human decision-making, assuming that preferences remain static over time. However, travelers constantly update their expectations based on experiential feedback, such as discrepancies between predicted and actual travel times. To address this, the authors integrated the Rescorla-Wagner reinforcement learning model with a latent class structure, allowing for the simultaneous capture of adaptive learning processes and individual heterogeneity.

Technically, the model utilizes the Rescorla-Wagner mechanism to update the expected value ($Q$) of selected alternatives based on a learning rate ($\alpha$). The utility function is formulated as $U = \gamma \pm \beta Q + \epsilon$, where $\beta$ serves as a critical parameter regulating the trade-off between exploration and exploitation, and $\gamma$ represents fixed effects. To handle the computational complexity of estimating high-dimensional latent variables, the study employs Variational Bayes. This approach provides an efficient approximation of the posterior distribution, enabling the quantification of uncertainty while maintaining computational feasibility for large-scale simulation datasets.

The empirical study was conducted using a driving simulator dataset, which provided a unique opportunity to observe feedback-driven learning in a controlled yet realistic environment. Unlike traditional survey-based methods, the simulator allowed researchers to track how participants modified their route preferences through repeated trials and real-time feedback.

The analysis revealed three distinct classes of travelers characterized by their learning strategies. The first class exhibits context-dependent preferences with context-specific exploitative tendencies. The second class demonstrates a persistent exploitation strategy, maintaining initial preferences regardless of the context due to a low learning rate and high $\beta$. The third class engages in a continuous exploratory strategy, characterized by a moderate learning rate and low $\beta$.

Despite its contributions, the paper acknowledges certain limitations, such as the “selection-only” nature of the Rescorla-Wagner update, which neglects value updates for non-selected alternatives, and the lack of intra-class individual variation. Furthermore, the reliance on simulator data necessitates future validation against real-world exogenous shocks like weather or accidents. Future research directions include incorporating value decay for non-selected alternatives and integrating the model with real-time traffic data to enhance its predictive power for intelligent transportation systems. This study offers significant implications for designing personalized transportation policies and information services tailored to different traveler profiles.