Deep Reinforcement Learning for Drone Conflict Resolution Under Uncertainty

Abstract

Drone operations are expected to become a key facet of urban life, through contributions to emergency response missions, alleviation of urban traffic through parcel delivery, and aerial inspection. High demand is predicted to cause traffic densities never before seen in classical aviation, exceeding human capabilities and requiring autonomous solutions for separation management. In flight, drones are to perform maneuvers to avoid other vehicles - this is Conflict Resolution. In most research, the Conflict Detection and Resolution (CD&R) component is modeled without sensor error, which is unrealistic, as the State-Based conflict detection is to rely on GNSSderived position information. Previous work has shown that conventional methods such as the Modified Voltage Potential (MVP) are negatively impacted by this error. This paper examines the applicability and performance of Reinforcement Learning (RL) for this conflict detection and resolution task, with positional error modeled. The RL model outputs velocity and heading commands (actions) on the basis of its state information, and that of other vehicles. Two models are trained - one with, and one without gaussian noise applied to the observed vehicle position. Both of these are tested against the MVP algorithm as a benchmark. The difference between the RL model trained with and without noise is minor in terms of losses of separation (safety). Compared to MVP, the model consistently shows a lower loss of separation count, demonstrating better noise robustness. The RL models favor pure speed changes to resolve conflicts, while staying on the same course.