Pre-Conference Talk by GENG Minghong

Talk #1: Scaling up Cooperative Multi-agent Reinforcement Learning Systems

Cooperative multi-agent reinforcement learning methods aim to learn effective collaborative behaviours of multiple agents performing complex tasks. However, existing MARL methods are commonly proposed for fairly small-scale multi-agent benchmark problems, wherein both the number of agents and the length of the time horizons are typically restricted. My initial work investigates hierarchical controls of multi-agent systems, where a unified overarching framework coordinates multiple smaller multi-agent subsystems, tackling complex, long-horizon tasks that involve multiple objectives. Addressing another critical need in the field, my research introduces a comprehensive benchmark for evaluating MARL methods in long-horizon, multi-agent, and multi-objective scenarios. This benchmark aims to fill the current gap in the MARL community for assessing methodologies in more complex and realistic scenarios. My studies focus on proposing and evaluating methods for scaling up multi-agent systems in two aspects: structural-wise increasing the number of reinforcement learning agents and temporal-wise extending the planning horizon and complexity of problem domains that agents are deployed in.

Talk #2: Benchmarking MARL on Long Horizon Sequential Multi-Objective Tasks

Current MARL benchmarks fall short in simulating realistic scenarios, particularly those involving long action sequences with sequential tasks and multiple conflicting objectives. Addressing this gap, we introduce Multi-Objective SMAC (MOSMAC), a novel MARL benchmark tailored to assess MARL methods on tasks with varying time horizons and multiple objectives. Each MOSMAC task contains one or multiple sequential subtasks. Agents are required to simultaneously balance between two objectives — combat and navigation — to successfully complete each subtask. Our evaluation of nine state-of-the-art MARL algorithms reveals that MOSMAC presents substantial challenges to many state-of-the-art MARL methods and effectively fills a critical gap in existing benchmarks for both single-objective and multi-objective MARL research. The codes of the MOSMAC benchmark will be publicly available on our GitHub repository https://github.com/smu-ncc/MOSMAC soon.

Talk #3: Explaining Sequences of Actions in Multi-agent Deep Reinforcement Learning Models

Multi-agent Deep Reinforcement Learning (MADRL) has proven effective in solving complex real-world problems, but understanding the learned behaviors of the agents from MADRL is still a great challenge. Explainable AI (XAI) is a growing field offering predictions in an explainable form and easily understood by humans. However, most current XAI methods for MADRL are still limited to explaining individual agents’ decisions without the broader context regarding their strategic and coordinated actions. This paper introduces a novel method to explain the MADRL agents’ behaviors by abstracting their sequences of actions into high-level abstract strategies. Particularly, a spatio-temporal neural network model is applied to encode the agents’ sequences of actions as memory episodes wherein an aggregating memory retrieval can generalize them into a concise abstract representation of collective strategies. To assess the effectiveness of our method, we applied it to explain the actions of QMIX MADRL agents playing a StarCraft Multi-agent Challenge (SMAC) video game as a benchmark task. A user study on the perceived explainability of the extracted strategies from the game indicates that our method can provide comprehensible explanations at various levels of granularity. It is also demonstrated that agents adopting the resulting strategies from our method have comparable performance to those of the original MADRL agents in playing the game.

GENG Minghong is a Ph.D. candidate at SCIS, under the supervision of Prof. TAN Ah Hwee. Minghong's research focuses on multi-agent reinforcement learning. He is currently exploring hierarchical approaches for scaling up multi-agent learning systems. Minghong is a member of the Neural and Cognitive Computing Group. Please visit his homepage https://gengminghong.github.io/ for more information.