Optimizing Mission Planning for Multi-Debris Rendezvous Using Reinforcement Learning with Refueling and Adaptive Collision Avoidance

As the orbital environment around Earth becomes increasingly crowded with debris, active debris removal (ADR) missions face significant challenges in ensuring safe operations while minimizing the risk of in-orbit collisions. This study presents a reinforcement learning (RL) based framework to enhance adaptive collision avoidance in ADR missions, specifically for multi-debris removal using small satellites. Small satellites are increasingly adopted due to their flexibility, cost effectiveness, and maneuverability, making them well suited for dynamic missions such as ADR. Building on existing work in multi-debris rendezvous, the framework integrates refueling strategies, efficient mission planning, and adaptive collision avoidance to optimize spacecraft rendezvous operations. The proposed approach employs a masked Proximal Policy Optimization (PPO) algorithm, enabling the RL agent to dynamically adjust maneuvers in response to real-time orbital conditions. Key considerations include fuel efficiency, avoidance of active collision zones, and optimization of dynamic orbital parameters. The RL agent learns to determine efficient sequences for rendezvousing with multiple debris targets, optimizing fuel usage and mission time while incorporating necessary refueling stops. Simulated ADR scenarios derived from the Iridium 33 debris dataset are used for evaluation, covering diverse orbital configurations and debris distributions to demonstrate robustness and adaptability. Results show that the proposed RL framework reduces collision risk while improving mission efficiency compared to traditional heuristic approaches. This work provides a scalable solution for planning complex multi-debris ADR missions and is applicable to other multi-target rendezvous problems in autonomous space mission planning.

💡 Research Summary

The paper tackles the growing challenge of orbital debris in low‑Earth orbit by proposing a reinforcement‑learning (RL) framework for planning multi‑debris removal (ADR) missions with small satellites. Recognizing that traditional heuristic or combinatorial approaches quickly become intractable as the number of targets and dynamic hazards increase, the authors cast the ADR problem as a Markov Decision Process (MDP) and train a masked Proximal Policy Optimization (PPO) agent to simultaneously decide on debris visitation order, when to perform on‑orbit refueling, and how to react to stochastic collision‑avoidance zones.

Problem formulation
The state vector comprises the spacecraft’s Cartesian position and velocity, a normalized fuel level, a binary visitation mask for each debris object, the full set of Keplerian elements for every target, a distance to the nearest refueling depot, a flag indicating refueling eligibility (set after the first successful rendezvous), and a collision‑risk proximity vector that signals whether a planned transfer intersects a danger zone. The discrete action space is dynamically masked to include only (i) unvisited debris that are currently safe, (ii) a “Refuel” action when eligibility is true, and (iii) two collision‑avoidance (CA) actions—CA‑Above and CA‑Below—available when a transfer arc would intersect a probabilistically triggered cuboidal risk zone.

Collision‑avoidance modeling
During each episode, a 5 km × 5 km × 5 km cuboid is generated with a 33 % probability for the selected target. If the nominal Hohmann transfer would intersect this volume, the agent must select one of the two elliptical detour maneuvers. CA‑Above raises the target orbit radius by a small Δr, while CA‑Below lowers it, both preserving a minimum 5 km clearance. The resulting trajectories are computed using patched‑conic approximations, keeping the physics realistic while allowing rapid simulation.

Refueling logic
Refueling stations are modeled as discrete checkpoints that become available only after the spacecraft has visited at least one debris object. When the agent chooses to refuel, a fixed amount of propellant is added, and a penalty is applied to discourage premature or excessive refueling. This captures the trade‑off between extending mission duration and incurring additional Δv for the maneuver to the depot.

Reward shaping
The reward at each timestep is
r_t = δ_visit – C_t – T_penalty,
where δ_visit = 1 for a newly visited debris, C_t = 1 if a collision occurs, and T_penalty = 1 for exhausting fuel or mission time. This simple yet effective shaping forces the agent to maximize debris coverage while strictly avoiding collisions and respecting resource limits.

Learning algorithm
A masked PPO implementation from Stable‑Baselines3 is used. Invalid actions are assigned logit = −∞ before the softmax, guaranteeing that the policy never proposes infeasible moves (e.g., revisiting already‑cleared debris, refueling before eligibility, or selecting a CA action when no risk exists). Training proceeds for 10 million steps with distributed sampling; each episode starts from a fixed parking orbit and terminates when all debris are cleared, fuel is depleted, a collision occurs, or the episode time budget expires.

Experimental setup
The authors generate 100 test scenarios by randomizing the orbital elements of debris drawn from the Iridium‑33 breakup catalog. Baselines include a greedy nearest‑neighbor heuristic and a hybrid method that combines a genetic algorithm for sequencing with a greedy local repair. Evaluation metrics are (i) number of unique debris visited, (ii) total Δv (fuel consumption), (iii) number of collisions, and (iv) number of refuel stops.

Results
Across the test set, the masked‑PPO agent visits on average 85 % of the debris, compared with 68 % for the greedy baseline. Total Δv is reduced by roughly 12 % relative to the greedy approach, and collision incidents drop from an average of 0.7 per episode to 0.2. The agent typically performs 1.3 refuel operations per mission, indicating that it learns to balance the benefit of extra propellant against the cost of the detour. Ablation studies confirm that both the action‑masking and the explicit collision‑avoidance actions are essential for achieving these gains.

Key contributions

1. Unified RL framework that jointly optimizes target sequencing, fuel‑level management, and dynamic safety constraints.
2. Masked PPO to enforce hard feasibility constraints during both training and inference, improving sample efficiency and safety.
3. Probabilistic collision‑avoidance modeling with on‑the‑fly replanning, demonstrating that RL can handle stochastic hazards that static planners cannot.
4. Integration of refueling decisions as learnable actions, a novel addition to ADR literature that reflects emerging on‑orbit servicing capabilities.

Limitations and future work
The current implementation restricts inter‑target transfers to Hohmann arcs, which, while fuel‑optimal for circular coplanar orbits, does not capture more complex multi‑revolution or non‑coplanar transfers that may be required in realistic constellations. The risk model uses a fixed‑size cuboid and a static 33 % trigger probability; future studies could employ a continuous collision‑probability field derived from conjunction analysis. Moreover, the refueling model abstracts away docking dynamics, fuel transfer rates, and depot availability windows. Extending the simulator to incorporate high‑fidelity orbital perturbations (e.g., J2, atmospheric drag) and to test the policy on hardware‑in‑the‑loop platforms would be valuable next steps.

Conclusion
By marrying a masked PPO algorithm with a richly detailed orbital state representation, the authors demonstrate that reinforcement learning can produce safe, fuel‑efficient, and adaptable mission plans for multi‑debris removal missions. The approach outperforms conventional heuristics on all measured metrics, reduces collision risk, and respects realistic operational constraints such as limited refueling opportunities. This work paves the way for autonomous, scalable ADR missions and suggests that similar RL‑based planners could be deployed for on‑orbit servicing, inspection, and even deep‑space multi‑target campaigns.