Traffic Signal Control with Cell Transmission Model Using Reinforcement Learning for Total Delay Minimisation

Pitipong Chanloha, Jatuporn Chinrungrueng, Wipawee Usaha, Chaodit Aswakul

Abstract


This paper proposes a new framework to control the traffic signal lights by applying the automated goal-directed learning and decision making scheme, namely the reinforcement learning (RL) method, to seek the best possible traffic signal ac- tions upon changes of network state modelled by the signalised cell transmission model (CTM). This paper employs the Q-learning which is one of the RL tools in order to find the traffic signal solution because of its adaptability in finding the real time solu- tion upon the change of states. The goal is for RL to minimise the total network delay. Surprisingly, by using the total network delay as a reward function, the results were not necessarily as good as initially expected. Rather, both simulation and mathemat- ical derivation results confirm that using the newly proposed red light delay as the RL reward function gives better performance than using the total network delay as the reward function. The investigated scenarios include the situations where the summa- tion of overall traffic demands exceeds the maximum flow capacity. Reported results show that our proposed framework using RL and CTM in the macroscopic level can computationally efficiently find the proper control solution close to the brute-forcely searched best periodic signal solution (BPSS). For the practical case study conducted by AIMSUN microscopic traffic simulator, the proposed CTM-based RL reveals that the reduction of the average delay can be significantly decreased by 40% with bus lane and 38% without bus lane in comparison with the case of currently used traffic signal strategy. Therefore, the CTM-based RL algorithm could be a useful tool to adjust the proper traffic signal light in practice.

Keywords


Traffic Signal Control (TSC), Cell Transmission Model (CTM), Rein- forcement Learning (RL).

Full Text:

PDF

References


R. Sutton, A.G. Barto (1998); Reinforcement Learning: An Introduction. The MIT Press, Cambridge, Massachusetts.

B. Abdulhai, L. Kattan (2003); Reinforcement learning: Introduction to theory and potential for transport applications. Canadian Journal of Civil Engineering, 30(6), 981–991.
http://dx.doi.org/10.1139/l03-014

C. Jacob, B. Abdulhai (2005); Integrated traffic corridor control using machine learning. International Conference on Systems, Man & Cybernetics, 3460–3465.
http://dx.doi.org/10.1109/ICSMC.2005.1571683

D.D. Oliveira et al (2006); Reinforcement learning-based control of traffic lights in non- stationary environments: a case study in a microscopic simulator. Forth European Workshop on Multi Agent Systems.

C.F. Daganzo (1995); The cell transmission model part II: Network traffic. Transportation Research Part B: Methodological, 29b(2), 79–93.
http://dx.doi.org/10.1016/0191-2615(94)00022-R

H.K. Lo et al (2001); Dynamic network traffic control. Transportation Research Part A: Policy and Practice, 35(8), 721–744.
http://dx.doi.org/10.1016/s0965-8564(00)00014-8

M. Maher, O. Feldman (2002); The application of the cell transmission model to the optimi- sation of signals on signalised roundabouts. European Transport Conference, 1–13.

H.K. Lo, A.H.F. Chow (2004); Control strategies for oversaturated traffic. Journal of Transportation Engineering, 466–478.
http://dx.doi.org/10.1061/(ASCE)0733-947X(2004)130:4(466)

W.H. Lin, C. Wang (2004); An enhanced 0-1 mixed-integer LP formulation for traffic signal control. IEEE Transactions on Intelligence Transportation Systems, 5(4): 238–245.
http://dx.doi.org/10.1109/TITS.2004.838217

K. Tueprasert, C. Aswakul (2010); Multiclass cell transmission model for heterogeneous mobility in general topology of road network. Journal of Intelligent Transportation Systems, 14(2): 68–82.
http://dx.doi.org/10.1080/15472451003719715

G. Flotterod, K. Nagel (2005) Some practical extensions to the cell transmission model. Proceedings of the 8th Internationall IEEE Conference on Intelligent Transportation Systems.
http://dx.doi.org/10.1109/itsc.2005.1520042

A. Sadek, N. Basha (2006); Self-learning intelligent agents for dynamic traffic routing on transportation networks. International Conference on Complex Systems, 503–518.

N.H. Gartner et al (1995); Development of advanced traffic signal control strategies for Intelligent Transportation Systems : multilevel design, Transportation Research Record, 98– 105.




DOI: https://doi.org/10.15837/ijccc.2015.5.2025



Copyright (c) 2017 Pitipong Chanloha, Jatuporn Chinrungrueng, Wipawee Usaha, Chaodit Aswakul

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

CC-BY-NC  License for Website User

Articles published in IJCCC user license are protected by copyright.

Users can access, download, copy, translate the IJCCC articles for non-commercial purposes provided that users, but cannot redistribute, display or adapt:

  • Cite the article using an appropriate bibliographic citation: author(s), article title, journal, volume, issue, page numbers, year of publication, DOI, and the link to the definitive published version on IJCCC website;
  • Maintain the integrity of the IJCCC article;
  • Retain the copyright notices and links to these terms and conditions so it is clear to other users what can and what cannot be done with the  article;
  • Ensure that, for any content in the IJCCC article that is identified as belonging to a third party, any re-use complies with the copyright policies of that third party;
  • Any translations must prominently display the statement: "This is an unofficial translation of an article that appeared in IJCCC. Agora University  has not endorsed this translation."

This is a non commercial license where the use of published articles for commercial purposes is forbiden. 

Commercial purposes include: 

  • Copying or downloading IJCCC articles, or linking to such postings, for further redistribution, sale or licensing, for a fee;
  • Copying, downloading or posting by a site or service that incorporates advertising with such content;
  • The inclusion or incorporation of article content in other works or services (other than normal quotations with an appropriate citation) that is then available for sale or licensing, for a fee;
  • Use of IJCCC articles or article content (other than normal quotations with appropriate citation) by for-profit organizations for promotional purposes, whether for a fee or otherwise;
  • Use for the purposes of monetary reward by means of sale, resale, license, loan, transfer or other form of commercial exploitation;

    The licensor cannot revoke these freedoms as long as you follow the license terms.

[End of CC-BY-NC  License for Website User]


INTERNATIONAL JOURNAL OF COMPUTERS COMMUNICATIONS & CONTROL (IJCCC), With Emphasis on the Integration of Three Technologies (C & C & C),  ISSN 1841-9836.

IJCCC was founded in 2006,  at Agora University, by  Ioan DZITAC (Editor-in-Chief),  Florin Gheorghe FILIP (Editor-in-Chief), and  Misu-Jan MANOLESCU (Managing Editor).

Ethics: This journal is a member of, and subscribes to the principles of, the Committee on Publication Ethics (COPE).

Ioan  DZITAC (Editor-in-Chief) at COPE European Seminar, Bruxelles, 2015:

IJCCC is covered/indexed/abstracted in Science Citation Index Expanded (since vol.1(S),  2006); JCR2018: IF=1.585..

IJCCC is indexed in Scopus from 2008 (CiteScore2018 = 1.56):

Nomination by Elsevier for Journal Excellence Award Romania 2015 (SNIP2014 = 1.029): Elsevier/ Scopus

IJCCC was nominated by Elsevier for Journal Excellence Award - "Scopus Awards Romania 2015" (SNIP2014 = 1.029).

IJCCC is in Top 3 of 157 Romanian journals indexed by Scopus (in all fields) and No.1 in Computer Science field by Elsevier/ Scopus.

 

 Impact Factor in JCR2018 (Clarivate Analytics/SCI Expanded/ISI Web of Science): IF=1.585 (Q3). Scopus: CiteScore2018=1.56 (Q2);

SCImago Journal & Country Rank

Editors-in-Chief: Ioan DZITAC & Florin Gheorghe FILIP.