Deep Learning and Mixed Reality for Autocomplete Teleoperation

Abstract

Teleoperation of robots can be challenging, especially for novice users with little to no experience at such tasks. The difficulty is largely due to the numerous degrees of freedom users must control and their limited perception bandwidth. Although humans can become skilled teleoperators, the amount of training time required to acquire such skills is typically very high. To help mitigate these challenges, this thesis proposes a solution (named Autocomplete) which relies on artificial intelligence to understand user intended motion and then on mixed real- ity to communicate the estimated trajectories to the users in an intuitive manner. User intended motion is estimated using a deep learning network trained on a dataset of motion primitives. During teleoperation, the estimated motions are augmented onto a first-person live video feed from the robot. Finally, if a suggested motion is accepted by the user, the robot is driven along that trajectory in an autonomous manner. We validate our proposed mixed reality teleoperation scheme with simulation experiments on a drone and demonstrate, through sub- jective and objective evaluation, its advantages over other teleoperation methods.