Estimating Calories and Macronutrients from Lebanese Food Images Using Deep Learning–Based Classification and Single-View Portion Size Estimation

Abstract

Assessing the dietary healthiness of a population is essential for understanding nutrition-related risks, yet existing dietary assessment methods in Lebanon remain costly, intrusive, and difficult to scale. This thesis investigates whether deep learning applied to food images can support automated estimation of calories and macronutrients for Lebanese meals, with a particular focus on single-view portion size estimation. We construct a 174-class taxonomy of Lebanese and regionally relevant dishes and train two complementary classifiers, InceptionV3 and ViT-B/16, on 7,067 cleaned real images collected through web crawling and public datasets, augmented with 2,657 synthetic samples generated using Gemini, Sora (OpenAI), and a food-domain GAN (FooDI-ML). Their late-fusion ensemble achieves 78.66% accuracy and 0.77 macro-F1 on a held-out real test set. Building on this classifier, we develop a multi-modal portion-size estimator that integrates the original image, a food crop, a segmentation mask, a monocular depth map, and a 3D shape descriptor. On curated gram-labeled data, the best configuration, the Original + Cropped + Mask 3-model ensemble, consistently outperforms other ensembles, while in the wearable-camera case study on 225 free-living images, a Cropped + Mask + Shape ensemble provides the best trade-off, yielding a portion-size MAE of 108.3 g alongside a classification accuracy of 61.3%. Finally, predicted classes and weights are mapped to nutritionist-provided composition tables to compute calories and macronutrients, and the thesis concludes with a practical data-collection protocol for future deployable systems aimed at improving population-level nutritional monitoring in Lebanon.