Multi-agent reinforcement learning with a hybrid sequential reward feedback strategy for dynamic multi-modal traffic assignment

Abstract

• Propose a multi-agent reinforcement learning framework (MARL) for multi-modal traffic assignment (MMTA) • Avoid to predefine choice sets of travel modes/routes for capturing combined multi-modal paths. • Use multi-edges to represent multi-modal links for cars and buses, exclusive bus lanes, and railways. • Develop a sequential reward feedback strategy to accelerate the convergence of the MARL-based MMTA algorithm. • Use hybrid reward function to reduce policy fluctuations and adjust the optimum between system optimum and user equilibrium. Urbanization and the expansion of transportation modes have exacerbated the challenges of understanding travelers’ decision-making processes regarding route choice across various transportation modes. This paper proposes a novel macroscopic hybrid sequential game method using multi-agent reinforcement learning (MARL) to address issues of computational efficiency and behavioral complexity in multi-modal transportation network simulations. Specifically, agents’ perception behaviors are modeled as a sequential decision-making process considering road capacity constraints, which helps estimate travel time under congestion effects in the multi-modal traffic assignment. In addition, a hybrid reward framework is proposed, providing system-level reward to guide the multi-agent system towards different Nash equilibria, thereby reducing policy fluctuations. To simulate interactions between agents of different transportation modes, a multi-edge representation and reward structures designed for car, bus, priority bus, and metro modes are adopted to handle the mixed traffic flow through the same road. Furthermore, our approach uses a mean-field multi-agent deep Q-learning method to consider both mode and route choice, simplifying agent interactions through mean-field theory and clustering agents with the same origin–destination (OD) demands. Experimental results demonstrate that the hybrid sequential feedback strategy outperforms the simultaneous feedback strategy regarding convergence speed, agent reward distribution, and network flow distribution. Furthermore, the proposed method is tested on the Sioux-Falls network to verify its computational efficiency in three network change scenarios (disruption, road reconstruction, and new road construction). These findings highlight the potential of the proposed MARL method for large-scale multi-modal transportation network analysis, particularly under various incident scenarios, providing an effective tool for urban transportation planning and project evaluation.