Abstract:
Manually teleoperating a flying robot can be a demanding task, especially for users
with limited levels of experience. This is primarily due to the nonlinear properties of
such robots in addition to the difficulty of controlling various degrees of freedom at
the same time. To help mitigate such limitations, this thesis proposes a framework
named ‘3D Autocomplete’ that aids users in teleoperation. It uses artificial intelligence
to predict in real-time the operator’s intended motion, and mixed reality to
convey the predicted motion to the user. Previous Autocomplete systems focused on
different 2D motions in the same plane (line, arc, sine). However, since many drone
tasks take place in a three-dimensional environment, 3D Autocomplete primarily
assists users in navigating challenging 3D motions around 3D geometric primitives
(cylinder, cone, and box). During teleoperation, the framework uses a real-time
change point detection algorithm called ‘just-in-time’ to monitor the user’s input,
and deep learning to early predict the motion type as one of predefined 3D motions.
Then, the predicted motion is augmented into the first person view in real-time
using a virtual reality headset. Finally, if the users accept the proposed trajectory,
3D Autocomplete completes their desired motion autonomously. We validate the
proposed mixed reality teleoperation approach by conducting different experiments
on a simulated quadrotor. The results illustrate 3D Autocomplete advantages over
traditional teleoperation methods through both subjective and objective evaluations
conducted via human subject experiments. The system achieves its primary goal of
reducing the users workload, and improves task completion time and covered distance
by at least 30% compared to traditional teleoperation. Moreover, it enhanced
the system performance and trajectory smoothness by approximately 50%.
