Pre-Conference Talk by SHAO Qian, LI Dexun and LING Jiajing

Talk #1: Preference-Aware Delivery Planning for Last-Mile Logistics
by SHAO Qian, PhD Candidate

Optimizing delivery routes for last-mile logistics service is challenging and has attracted attention from many researchers. These problems are usually modeled and solved as variants of vehicle routing problems (VRPs) with challenging real-world constraints (e.g., time windows, precedence).  Although decades of research have been devoted to solving vehicle routing problems (VRPs), there are still significant gaps between optimized and preferred routes. To address this, we propose a novel hierarchical route optimizer that combines optimization and machine learning. Using a real-world dataset, we demonstrate the importance of both components and identify instances where we may encounter difficulties.

Talk #2: Avoiding Starvation of Arms in Restless Multi-Armed Bandits
by LI Dexun, PhD Candidate

Restless multi-armed bandits (RMAB) is a popular framework for optimizing performance with limited resources under uncertainty. It is an extremely useful model for monitoring beneficiaries (arms) and executing timely interventions using health workers (limited resources) to ensure optimal benefit in public health settings. Due to the limited resources, typically certain individuals, communities, or regions are starved of interventions, which can potentially have a significant negative impact on the individual/community in the long term. To that end, we first define a soft fairness objective which entails an algorithm never probabilistically favors one arm over another if the long-term cumulative reward of choosing the latter arm is higher. Then we provide a scalable approach to ensure long-term optimality while satisfying the proposed fairness constraints in RMAB. Finally, we demonstrate the utility of our approaches on simulated benchmarks and show that the soft fairness objective can be handled without a significant sacrifice on the optimal value.

Talk #3: Knowledge Compilation for Constrained Combinatorial Action Spaces in Reinforcement Learning
by LING Jiajing, PhD Candidate

Action-constrained reinforcement learning (ACRL) has several practical applications in resource allocation and path planning. However, enforcing constraints in discrete and combinatorial action spaces is challenging. To overcome this, we propose encoding valid actions that satisfy all action constraints using a probabilistic sentential decision diagram (psdd), a recently proposed knowledge compilation framework. Parameters of the psdd encode the probability distribution over all valid actions, making the learning task become optimizing psdd parameters for maximizing RL objective. We show how to embed the psdd parameters using deep neural networks and optimize them using a deep Q-learning based algorithm. Finally, we show how practical resource allocation constraints can be encoded using a psdd. By design, our approach guarantees constraint satisfaction without expensive projection steps and outperforms previous ACRL methods in terms of scalability and constraint satisfaction.

Shao Qian is a Ph.D. student in Computer Science at the SMU SCIS, supervised by Prof. Cheng Shih-Fen. Her research interests are last-mile logistics, imitation learning and approximate dynamic programming.

Dexun is a Ph.D. candidate in Computer Science at the SMU School of Computing and Information Systems, supervised by Prof. Pradeep VARAKANTHAM. His research interests are Reinforcement Learning and Optimization. His current research focuses on unsupervised environment design.

Jiajing is a Ph.D. candidate in Computer Science. He started his Ph.D. studies at SMU in 2018 under the supervision of Prof. Akshat KUMAR. Prior to this, he obtained his Bachelor's degree in Electronics Engineering from Guangzhou University and his Master's degree in Quantitative Finance from Singapore Management University. His research interests are primarily focused on reinforcement learning, multi-agent systems, and neuro-symbolic AI. He has made significant contributions to the field by developing neuro-symbolic methods for solving constrained RL problems, which have been published in various prestigious conferences and workshops, including ICAPS, AAMAS, ECML/PKDD, and AAAI. He was awarded the SMU Presidential Doctoral Fellowship (2021-2022) for outstanding research.