3D Autocomplete: Enhancing UAV Teleoperation with AI in the Loop

Abstract

Manually teleoperating a flying robot can be a demanding task, especially for users with limited levels of experience. This is primarily due to the nonlinear properties of such robots in addition to the difficulty of controlling various degrees of freedom at the same time. To help mitigate such limitations, this thesis proposes a framework named ‘3D Autocomplete’ that aids users in teleoperation. It uses artificial intelligence to predict in real-time the operator’s intended motion, and mixed reality to convey the predicted motion to the user. Previous Autocomplete systems focused on different 2D motions in the same plane (line, arc, sine). However, since many drone tasks take place in a three-dimensional environment, 3D Autocomplete primarily assists users in navigating challenging 3D motions around 3D geometric primitives (cylinder, cone, and box). During teleoperation, the framework uses a real-time change point detection algorithm called ‘just-in-time’ to monitor the user’s input, and deep learning to early predict the motion type as one of predefined 3D motions. Then, the predicted motion is augmented into the first person view in real-time using a virtual reality headset. Finally, if the users accept the proposed trajectory, 3D Autocomplete completes their desired motion autonomously. We validate the proposed mixed reality teleoperation approach by conducting different experiments on a simulated quadrotor. The results illustrate 3D Autocomplete advantages over traditional teleoperation methods through both subjective and objective evaluations conducted via human subject experiments. The system achieves its primary goal of reducing the users workload, and improves task completion time and covered distance by at least 30% compared to traditional teleoperation. Moreover, it enhanced the system performance and trajectory smoothness by approximately 50%.