Thesis Defense by LIU Suyu

Multi-Modal Alignment Via Hyperbolic Geometry

About The Talk

Strong capabilities of generalization to unseen domains are vital for deep neural networks. While existing methods have shown promising results without source domain access, they mostly rely on extensive pre-training or overlook the intricate hierarchical structures inherent in visual and textual features. Such limitations can hinder performance, especially on datasets with numerous classes. To overcome this, we propose a novel approach that projects the model onto hyperbolic geometry and employs optimal transport to align cross-modal features in an unsupervised manner. Unlike Euclidean geometry, hyperbolic geometry is characterized by hierarchical data structures, which can facilitate understanding diverse classes. To fully capture hierarchical information from text, we enrich the model with fine-grained concepts from WordNet, enhancing its understanding of diverse classes. Extensive experiments on standard benchmarks demonstrate the superior performance of our method compared to strong baselines.

About The Speaker

Suyu LIU is currently a master student at the School of Computing and Information Systems, Singapore Management University, supervised by Professor Lizi Liao (Main Supervisor) and Professor Zhiguang Cao (Co-Supervisor). His research interests mainly lie in multi-modal learning, geometric deep learning and efficient combinatorial optimization.