Developing a Machine Learning–Based System for Nutrient Profiling of Pakistani Cuisine

Abstract

Abstract

Background: Long-term health conditions such as diabetes and heart disease are rising quickly, especially in South Asia, and poor diet is a major contributor. Existing dietary tracking apps are mostly built for Western food and struggle with mixed, culturally diverse dishes such as those found in Pakistani cuisine. Objective: This study aimed to build a machine learning-based system that can recognise Pakistani dishes, estimate portion sizes, and calculate their nutritional value. Methods: We developed a multi-stage automated dietary assessment pipeline for Pakistani cuisine using a newly compiled, multi-source dataset spanning 85 dish classes across 10 categories. Following data cleaning, class balancing, and targeted augmentation, multiple deep learning architectures were evaluated for classification and segmentation tasks. Specifically, ResNet-50, MobileNetV2, and YOLOv8 were used for food recognition, while YOLOv8-Seg and Mask R-CNN provided pixel-level segmentation for downstream portion and nutrient estimation. All models were optimized using transfer learning and evaluated on standard classification and mask-quality metrics. At the end, usability testing of mobile application was conducted. Results: A dataset of Pakistani food images across ten categories and 85 dishes was collected from multiple sources and used to train and compare several deep learning models for food classification and segmentation. Portion size was estimated using a segmentation and depth-based approach that does not need any reference object in the photo, and nutritional values were calculated using standardised recipes matched against established food databases. The models were then combined into a mobile application and tested with users for real-world usability. Among the classification models, a YOLO-based architecture performed best, and a related YOLO model gave the most practical results for segmentation, offering a good balance between accuracy and speed suited to real-time use on a phone. Portion size estimates were reasonably close to actual measured values, and early usability testing showed that users found the app easy to use, although some improvements were suggested. Conclusion: Overall, this work shows that a culturally adapted, AI-based dietary assessment system for Pakistani food is achievable, and it lays the groundwork for further development, particularly around reducing processing demands and improving the reliability of depth-based portion estimation.