Scalable distributed lifelong multi-task reinforcement learning

Abstract

Multi-task reinforcement learning (MTRL) suffers from scalability issues when the number of tasks or trajectories per task grows-large. One of the main reasons behind this limitation is the reliance on centralized solutions. Recent methods exploited the connection between MTRL and general consensus to propose scalable solutions with linear convergence guarantees. In this work, we improve over state-of-the-art by presenting a distributed solver for MTRL with quadratic convergence guarantees. Our algorithm exploits a novel connection between MTRL and Laplacian-based general consensus that leads to an efficient solver. We further extend our work to the lifelong settings where we propose the first distributed lifelong MTRL solver who exhibits vanishing regret. We analyze both the theoretical and empirical properties of our method. In set of extensive experiments, we also show that the novel algorithm outperforms state-of-the-art on a variety of dynamical systems, including a simulated humanoid robot.