PhD Dissertation Defense by ZHOU Kankan | Evaluation of Pre-trained Vision Language Models in Challenging Contexts

Evaluation of Pre-trained Vision Language Models in Challenging Contexts

ABOUT THE TALK

This thesis presents a comprehensive evaluation of pre-trained vision-language models (VLMs), aiming to dissect their performance limitations, especially in challenging contexts that push their boundaries. We delve into three primary concerns: stereotypical bias, reasoning beyond common sense, and underspecification. A new dataset, VLStereoSet, is introduced to comprehensively measure stereotypical bias, which reveals prevalent biases through empirical analysis on six VLMs. Furthermore, we present ROME, a dataset challenging VLMs with counter-intuitive scenarios to test their reasoning against human-like interpretation. Our findings from this dataset demonstrate a significant gap between machine reasoning and nuanced human thought. Finally, we propose FOCUS, a vision-language dataset designed to evaluate how VLMs handle underspecified statements by requiring context-sensitive disambiguation through paired images. Through rigorous experimentation and analysis, we aim to unravel the complexities of these models, identify their shortcomings, and pave the way for future advancements that bring us closer to realizing the full potential of AI.

SPEAKER BIOGRAPHY

Kankan ZHOU received his bachelor’s degree in computer science from Nanyang Technological University (NTU), Singapore, in 2014 and master’s degree in computing from National University of Singapore (NUS), in 2016. Kankan has more than 10 years working experience in AI & Analytic field with different companies such as Oracle Singapore, Aon Singapore, and Accenture Singapore, etc. Now he is pursuing the part time Ph.D. degree in computer science in Singapore Management University(SMU) under the supervision of Prof. Jing Jiang and work as full time SCIS undergraduate instructor in SMU. His research focuses on natural language processing.