PhD Dissertation Proposal by ZHOU Kankan | Evaluation of Pre-trained Vision Language Models in Challenging Contexts

This proposal embarks on a critical evaluation of pre-trained vision-language models (VLMs), aiming to dissect their performance limitations, especially in challenging contexts that push their boundaries. We delve into two primary concerns: investigating the embedded biases in these models and scrutinizing their reasoning abilities in contrast to human cognition. A new dataset, VLStereoSet, is introduced to comprehensively measure stereotypical bias, which reveals prevalent biases through empirical analysis on six VLMs. Furthermore, we present ROME, a dataset challenging VLMs with counter intuitive scenarios to test their reasoning against human-like interpretation. Our findings from this dataset demonstrate a significant gap between machine reasoning and nuanced human thought. Through rigorous experimentation and analysis, we aim to unravel the complexities of these models, identify their shortcomings, and pave the way for future advancements that bring us closer to realizing the full potential of AI.

Kankan Zhou received his bachelor’s degree in computer science from Nanyang Technological University (NTU), Singapore, in 2014 and master’s degree in computing from National University of Singapore (NUS), in 2016. Kankan has more than 10 years working experience in AI & Analytic field with different companies such as Oracle Singapore, Aon Singapore, and Accenture Singapore, etc. Now he is pursuing the part time Ph.D. degree in computer science in Singapore Management University(SMU) under the supervision of Prof. Jing Jiang and work as full time SCIS undergraduate instructor in SMU. His research focuses on natural language processing.