Inpatient Overflow Management with Proximal Policy Optimization

Problem Definition: Managing inpatient flow in large hospital systems is challenging due to the complexity of assigning randomly arriving patients – either waiting for primary units or being overflowed to alternative units. Current practices rely on ad-hoc rules, while prior analytical approaches struggle with the intractably large state and action spaces inherent in patient-unit matching. Scalable decision support is needed to optimize overflow management while accounting for time-periodic fluctuations in patient flow. Methodology/Results: We develop a scalable decision-making framework using Proximal Policy Optimization (PPO) to optimize overflow decisions in a time-periodic, long-run average cost setting. To address the combinatorial complexity, we introduce atomic actions, which decompose multi-patient routing into sequential assignments. We further enhance computational efficiency through a partially-shared policy network designed to balance parameter sharing with time-specific policy adaptations, and a queueing-informed value function approximation to improve policy evaluation. Our method significantly reduces the need for extensive simulation data, a common limitation in reinforcement learning applications. Case studies on hospital systems with up to twenty patient classes and twenty wards demonstrate that our approach matches or outperforms existing benchmarks, including approximate dynamic programming, which is computationally infeasible beyond five wards. Managerial Implications: Our framework offers a scalable, efficient, and explainable solution for managing patient flow in complex hospital systems. More broadly, our results highlight that domain-aware adaptation is more critical to improving algorithm performance than fine-tuning neural network parameters when applying general-purpose algorithms to specific applications.

💡 Research Summary

The paper tackles the challenging problem of inpatient overflow management in large hospital systems, where patients arriving randomly must be assigned either to their primary ward or to an alternative “overflow” ward when capacity is exceeded. Traditional practice relies on ad‑hoc rules, and existing analytical approaches such as Approximate Dynamic Programming (ADP) struggle with the combinatorial explosion of both state and action spaces, limiting their applicability to systems with only a handful of specialties and wards.

To overcome these limitations, the authors formulate the problem as a time‑periodic, long‑run average‑cost Markov Decision Process (MDP) and apply Proximal Policy Optimization (PPO), a state‑of‑the‑art reinforcement‑learning algorithm. Three key methodological innovations enable PPO to scale to realistic hospital sizes (up to 20 patient classes and 20 wards):

1. Atomic Action Decomposition – Instead of treating a multi‑patient routing decision as a single high‑dimensional action, the authors break it down into a sequence of “atomic” actions that assign one patient at a time. This reduces the effective action space from combinatorial (exponential in the number of patients and wards) to linear, making policy learning tractable.

2. Partially‑Shared Policy Network – Patient arrivals and discharges exhibit daily and weekly periodic patterns. The policy network therefore shares a core set of parameters across all time periods while attaching small, time‑specific subnetworks. This design captures periodic dynamics without exploding the number of trainable parameters, balancing generalization and temporal specificity.

3. Queue‑Informed Value Function Approximation – The value function is enriched with features derived from queueing theory (e.g., current queue lengths, service rates, expected waiting times). By embedding domain knowledge directly into the critic, sample efficiency improves dramatically, reducing the amount of simulation data required for stable learning.

The authors also provide a theoretical guarantee that PPO’s policy‑improvement property holds under the periodic MDP setting, extending existing PPO convergence results to the infinite‑horizon average‑cost regime.

Empirical Evaluation
A suite of simulation experiments is conducted, ranging from small (5 × 5) to large (20 × 20) configurations. The proposed “Atomic‑PPO” is benchmarked against ADP, linear programming‑based heuristics, and simple rule‑based policies. Results show:

• Comparable or superior average cost performance across all sizes, with a 5‑15 % improvement over the best baselines in the 10‑20 ward regimes.
• Substantial computational advantage: while ADP becomes infeasible beyond five wards, Atomic‑PPO converges within a few hours even for the 20‑ward case.
• Sample efficiency gains of 30‑50 %: the queue‑informed critic enables the algorithm to learn effective policies with far fewer simulated trajectories.

Policy visualizations reveal intuitive behavior: the learned policy tends to defer overflow assignments when a surge of discharges is imminent, and proactively overflows patients during anticipated high‑arrival periods. This aligns with managerial intuition and demonstrates the interpretability of the approach.

Managerial Implications
The framework offers a scalable, data‑driven decision‑support tool that can replace ad‑hoc overflow rules in complex hospital networks. By reducing unnecessary overflow placements, hospitals can lower coordination costs, improve patient outcomes, and increase bed utilization efficiency. Moreover, the study underscores that tailoring a generic RL algorithm to the problem’s structural properties (action atomization, periodicity, queueing dynamics) yields larger performance gains than exhaustive hyper‑parameter tuning.

Limitations and Future Work
The work is validated through simulation; real‑world pilot deployments are needed to assess robustness to unmodeled factors such as patient acuity, staffing constraints, and emergency department surge dynamics. Extending the model to incorporate multi‑stage treatment pathways, heterogeneous service times, and stochastic resource availability constitutes promising avenues for further research.

In sum, the paper delivers a novel, theoretically grounded, and empirically validated PPO‑based solution that scales to realistic inpatient overflow settings, while offering insights transferable to other large‑scale matching and queueing problems in logistics, ride‑hailing, and cloud resource allocation.