Research Seminar by Gary ANG | Learning Multimodal Networks for User Interface Tasks

User interfaces (UI) of desktop, web, and mobile applications can be viewed as inter-linked objects (e.g., applications, UI screens and elements) with multimodal (e.g., numerical, textual, visual) attributes. Representing a set of UIs as a network of design objects with multimodal attributes not only represents how users understand the visual layout of UIs, but also influences how users would interact with applications through these UIs. Learning the semantics of such multimodal networks could be useful for a range of different UI-related tasks, such as UI layout assistance, UI design evaluation, automatic annotation of UI elements for accessibility, and UI retrieval and search. In this talk, we will cover the key motivations, proposed network learning models, and experimental results in a series of works that model such multimodal UI networks for different predictive tasks that have been accepted at the ACM International Conference on Intelligent User Interfaces and ACM Transactions on Interactive Intelligent Systems.