PhD Dissertation Defense by TRUONG Quoc Tuan | Modeling Sentiments and Preferences from Multimodal Data

Online reviews are prevalent in many modern Web applications, such as e-commerce, crowd-sourced location and check-in platforms. Fueled by the rise of mobile phones that are often the only cameras on hand, reviews are increasingly multimodal, with photos in addition to textual content. In this thesis, we focus on modeling the subjectivity carried in this form of data, with two research objectives.

In the first part, we tackle the problem of detecting sentiment expressed by a review. This is a key unlocking many applications, e.g., analyzing opinions, monitoring consumer satisfaction, assessing product quality. Traditionally, the task of sentiment analysis primarily relies on textual content.  We focus on visual sentiment of review images and develop models to systematically analyze the impact of three factors: image, user, and item. Further investigation leads to a notion of concept-orientation generalizing visual sentiment analysis for Web images. We also develop an effective method to deal with the scenario of one document being associated with multiple images, such as online reviews, blog posts, social networks, and media articles. Furthermore, we study the utilization of sentiment as an independent modality in the context of cross-modal retrieval.

In the second part, we focus on developing models for capturing user preferences from multimodal data. Preference modeling is crucial to recommender systems which are core to modern online user-based platforms. The need for recommendations is to guide users in browsing the myriad of options offered to them. First, we propose an approach that captures user preferences via simultaneously modeling a rating prediction component and a review text generation component. Second, we introduce a new generative model of preferences, inspired by the dyadic nature of the preference signals. The model is bilateral making it more apt for incorporation of auxiliary data from both sides of user and item. Third, we develop a probabilistic framework for modeling preferences involving logged bandit feedback. The proposed framework is effective for recommendation and ads placement systems.

In general, we present multiple approaches to modeling various aspects of sentiment and preference signals from multimodal data. Our work contributes a set of techniques that could be broadly extensible for mining Web data. Additionally, this research facilitates the development of recommender systems, which play a significant role in many online user-based platforms.

Quoc-Tuan Truong is currently a PhD student in Computer Science at Singapore Management University. His advisor is Associate Professor Hady W. Lauw. Tuan’s research aims at developing efficient machine learning algorithms and systems for information retrieval and Web data mining. In particular, he is interested in preference modeling for recommender systems and sentiment/opinion analysis from Web online reviews. His work has been published in established conferences and journals in the fields of data mining and machine learning, such as ACMMM, AAAI, WWW, MLJ, and JMLR.

Previously, Tuan graduated with a Bachelor’s degree in Computer Science from Vietnam National University, Hanoi. During his PhD, he received the Presidential Doctoral Fellowship twice, and the Dean’s List thrice. He also did a research internship with Amazon, US in the summer of 2021.