PhD Dissertation Defense by WANG Qing | Food Computing: Domain Adaptation and Causal Inference

ABOUT THE TALK

Food computing, a field dedicated to collecting and analyzing diverse food data from various sources, plays a key role in tasks such as food recognition, detection and segmentation, cross-modal food retrieval, food recommendation, and calorie and nutrition estimation. It has gained increasing attention due to its various applications in human health and disease, including automated meal logging, dietary tracking, diet planning and nutritional guidance. Despite its significance and potential, food computing faces several challenges: (1) domain gaps: discrepancies between training data and real-world testing images limit the generalization of models; (2) data imbalance: food datasets often exhibit long-tailed distributions, leading to models trained with these datasets being biased toward certain popular dishes and ingredients; (3) fine-grained characteristics: the existence of visually similar classes presents subtle variation that complicate accurate differentiation; (4) cross-modal alignment: aligning different modalities, such as food images and recipe texts, is a non-trivial task. 

The thesis contributes to a deeper understanding of how food datasets can impact the recognition and retrieval performances. It addresses two challenges in food computing: food recognition and food image-to-recipe retrieval. The main research ideas are: (1) leveraging large language models (LLMs) to augment food image representations to mitigate the combined challenges of domain gaps and data imbalance in fine-grained food recognition; (2) proposing a causal-theory inspired cross-modal representation learning formulation for reducing the bias introduced by the ingredient confounder for cross-modal recipe retrieval; and (3) extending the framework to incorporate multiple confounders, particularly ingredients and cooking actions, allows for more comprehensive modeling of the food image-to-recipe retrieval problem.

ABOUT THE SPEAKER

Qing is a fourth-year PhD candidate in Computer Science at Singapore Management University, under the supervision of Prof. Chong-Wah Ngo (main supervisor) and Assoc. Prof. Qianru Sun (co-supervisor). He completed his master’s degree at Louisiana State University. His research interests include food computing, imbalanced learning, and causal inference. Outside of research, Qing enjoys music, and is passionate about activities related to music and language learning.