Machine Learning from Limited Time-series Data

Abstract

Time-series presents an important class of data in our everyday life and is becoming predominant with the abundance of sensors and IoT devices, which as a result has created opportunities for new machine learning (ML) applications. Unfortunately, the vast amounts of data collected are unlabeled and annotation of such data for ML becomes a challenge as it demands high monetary cost, labor, and time. This work aims at developing methods to overcome data limitations for time-series and advancing transfer learning approaches.

In the first objective of the work, we address the labeled data deficiency bottleneck when building personalized models for a group of target tasks. Most researchers have approached the problem of learning personalized models through learning a unique model per task. We present a new systematic approach for designing, evaluating, and improving Multitask Learning models, which learn multiple target tasks simultaneously and leverage information transfer across all tasks. We consider three primary design components: features capturing the time dynamics in data, similarity metrics reflecting degrees of commonality and uniqueness across entities, and generalization metrics to prevent overfitting. The framework enables the introduction of efficient new MTL models and advances the prior state-of-the-art. The approach is successfully applied and tested resulting in an MTL deep learning approach that makes use of Convolutional Neural Networks (CNN) and Gated Recurrent Units (GRU).

In the second objective of the work, we consider the case where new tasks emerge for models that had been previously trained on sufficient labeled data (referred to as ‘source’ data) but where the source data is no longer accessible, a common scenario that arises due to lack of resources or privacy and security constraints. The goal in such scenarios is to transfer knowledge from a pre-trained model to the emerging new target tasks without sacrificing performance on the source data tasks, a problem known as catastrophic forgetting. We propose a novel multi-objective learning approach with three loss functions to minimize catastrophic forgetting, prediction error, and generalization error where label shifts exist across the source and target tasks. Under the first objective, the contributions of this dissertation are two folds. We introduce an architecture supporting the multi-objective method and targeting multitask prediction for time-series data.

In the third objective of the work, we investigate the task transferability estimation problem, which aims to provide an a priori estimate of the success of knowledge transfer between a given pre-trained model of a source task and a dataset of a new target task. Previous work has explored empirical and analytical solutions to the transferability estimation problem with remaining limitations in how the transferability relationship is defined between the source and target tasks. We present a method to show which representations extracted from a source pre-trained model are most descriptive of source and target task transferability. We build an interpretable attention network that learns the optimal combination of pre-trained model representations that hold the highest contribution to transferability across the source and target tasks. Under the second objective, the contributions of the dissertation are two folds. We introduce an attention-based transferability measure improving on state-of-the-art transferability estimation and presenting an interpretable technique to define the relationship between a source and target task.

We evaluate our proposed methods on a range of benchmark human activity recognition datasets as a sensing application as well as benchmark computer vision object detection datasets for evaluation against the state-of-the-art. Our proposed work is shown to advance state-of-the-art methods under both dissertation objectives.