PhD Dissertation Proposal by NGUYEN Duc Thien | Collective Multi-agent Planning and Inference

Collective Multi-agent Planning and Inference

About The Talk

Multiagent sequential decision making has seen rapid progress with formal models such as decentralized MDPs and POMDPs. However, scalability to large multiagent systems and applicability to real-world problems remain limited. To address these challenges, we study multiagent problems where the collective behavior of a population of agents affects the joint-reward and environment dynamics. Our research is twofold: to develop collective planning framework and to develop efficient collective inference solver. For collective planning, we develop a Collective Decentralized POMDP model (CDec-POMDP) where policies can be computed based on counts of agents in different states. Exploits recent advances in graphical models for modeling and inference with a population of individuals such as collective graphical models and the notion of finite partial exchangeability in lifted inference, we show that count variables are sufficient statistics in CDec-POMDP, which motivates us to develop count-based reinforcement learning algorithms. A major challenge in CDec-POMDP is the problem of credit assignment to individual agents given the team reward signal. To address this, we propose 1) a suitable form of value function approximation in CDec-POMDP; 2) a principled count-based mechanism to distribute the system empirical return into individual credit to train approximate value function in CDec-POMDP; 3) an Expected Maximization to optimize tabular policy for CDec-POMDP agents; 4) an actor-critic algorithm to train neural network policy function for CDec-POMDP. We show advantages of our proposed algorithms by experiments on synthetic instances and a real-world taxi dataset modeling supply-demand matching. For collective inference, we address the problem of incomplete count data in real-world applications due to imperfect sensors or privacy-preserving policy. We notice that all of the state-of-the-art collective inference solvers cannot handle high tree-width graphical structure. This motivates us to propose an efficient count inference algorithm for the problem based on Bethe entropy approximation and convex-concave optimization. We show that our proposed algorithm outperforms others in a well-known benchmark in the count inference task.

Speaker Biography

NGUYEN Duc Thien is a fourth-year PhD candidate in Information Systems. Since 2014, he has been working under the supervision of Professor Lau Hoong Chuin and Assistant Professor Akshat Kumar in his PhD thesis topic "Collective Multi-agent Planning and Inference", i.e. to find the agent policy in a (large) population. Before joining SMU as a PhD student, he had his Master degree in Information Systems from SMU in 2013 and Bachelor degree in Mathematics from Vietnam National University in 2010.