Machine Learning-Based Unobtrusive Intake Gesture Detection via Wearable Inertial Sensors

Abstract

Eating is one of the most basic activities observed in sentient animals, a behavior so natural that humans often eating without giving the activity a second thought. Unfortunately, this often leads to consuming more calories than expended, which can cause weight gain, a leading cause of diseases and death. This thesis describes research in methods to automatically detect periods of eating by tracking wrist motion so that calorie consumption can be tracked. We first briefly discuss how obesity is caused due to an imbalance in calorie intake and expenditure. Calorie consumption and expenditure can be tracked manually using tools like paper diaries; however it is well known that human bias can affect the accuracy of such tracking. Researchers in the upcoming field of automated dietary monitoring (ADM) are attempting to track diet using electronic methods in an effort to mitigate this bias.

State-of-the-art wearable sensor technologies can play a critical role in assisting patients in ADM by providing meaningful statistics on critical parameters such as the onset, duration, and frequency of eating. In this thesis a wearable watch fitted with an inertial measurement unit (IMU) is adopted to measure and detect eating gestures. However, IMU readings are useless without proper calibration. Thus, an effective and easy to implement calibration scheme that only requires collecting IMU data during random orientations and under any angular velocity is presented. In the proposed method no predefined trajectory or path is required for the IMU. Unlike other methods, where the sensor is placed in precise predefined orientations using a turn table or a robotic arm, and where the parameters are calculated offline, in this method the IMU is randomly rotated by hand, and the parameters are estimated during calibration, using an online Kalman filter algorithm without the need for predefined orientations.

In addition, for an accurate yet fast food intake recognition, this work presents a novel feature engineering framework by leveraging optimized support vector machine (SVM) classifiers. The SVM classifiers are trained on three comprehensive datasets: OREBA, CLEMSON, and FIC. The developed framework outperforms existing algorithms by achieving F1-scores of 94.8% ( 94.0% precision, 95.7% recall) for the SVM classifier, on the OREBA-DIS dataset, and 96.0% (95.4% precision, 96.7% recall) on the OREBA-SHA dataset, respectively. Moreover, the same algorithm achieves an F1-score 95.7% on the FIC (95.5% precision, 97.9% recall), and 85.9% on the CLEMSON (92% precision, 80.7% recall). Our experiments also train on one of the three databases and test on the remaining two to assess generalization effectiveness. In all cases, we achieve the highest F1 scores. To this end, the proposed algorithm presents a suitable solution for real-time applications, where inference is made using a few support vector parameters, compared to thousands of parameters in peer deep neural networks (DNN) models. Therefore, the proposed solution can beat parameter heavy DNN approaches in terms of a more accurate eating detection algorithm at lower computational costs.

By visualizing periods of eating in the collected datasets we learn that people often conduct secondary activities while eating, such as walking, watching television, working, and doing household chores. These secondary activities cause wrist motions that obfuscate wrist motions associated with eating, which increases the difficulty of detecting periods of eating (meals). Subjects reported conducting secondary activities in 72% of meals. Analysis of wrist motion data revealed that the wrist was resting 12.8% of the time during self-reported meals, compared to only 6.8% of the time in a cafeteria dataset. Walking motion was found during 5.5% of the time during meals in free-living, compared to 0% in the cafeteria. This suggests that future data collections for eating activity detection should also collect detailed ground truth on secondary activities being conducted during eating.

Finally, we present SLAC, a new method to detect periods of eating by tracking wrist motion during everyday life. Prior work on eating detection aggregated individual hand to mouth gestures to detect meals. As an alternative, we think of the eating detection problem as a regression to spatially scattered eating intervals and associated class probabilities. The meal bounding boxes and their class (breakfast, lunch, dinner, and snack) probabilities may be predicted for a whole day by a single convolutional neural network (NN) in a single step. In contrast to sliding window approaches that analyses only a small portion of available data, SLAC analyzes a full day at a time, encoding contextual information such as preparing and manipulating food, as well as cleanup, which are valuable for the task of detecting eating episodes. Compared to state-of-the-art detection systems on the CLEMSON and FIC all-day datasets, SLAC outperforms other approaches detecting 95% of eating episodes, with 1.2 false positives (FP) for every true positive (TP). Finally, since SLAC is a regression-based problem, it is extremely fast and thus suitable for embedded systems with low computational resources.