V2V Routing in VANET Based on Heuristic Q-Learning

  • Xiaoying Yang School of Information Engineering, Suzhou University, Suzhou 234000, China
  • Wanli Zhang School of Information Engineering, Suzhou University, Suzhou 234000, China
  • Hongmei Lu School of Information Engineering, Suzhou University, Suzhou 234000, China
  • Liang Zhao School of Computer Science, Shenyang Aerospace University, Shenyang 110136, China

Abstract

Designing efficient routing algorithms in vehicular ad hoc networks (VANETs) plays an important role in the emerging intelligent transportation systems. In this paper, a routing algorithm based on the improved Q-learning is proposed for vehicle-to-vehicle (V2V) communications in VANETs. Firstly, a link maintenance time model is established, and the maintenance time is taken as an important parameter in the design of routing algorithm to ensure the reliability of each hop link. Aiming at the low efficiency and slow convergence of Q-learning, heuristic function and evaluation function are introduced to accelerate the update of Q-value of current optimal action, reduce unnecessary exploration, accelerate the convergence speed of Q-learning process and improve learning efficiency. The learning task is dispersed in each vehicle node in the new routing algorithm and it maintains the reliable routing path by periodically exchanging beacon information with surrounding nodes, guides the node’s forwarding action by combining the delay information between nodes to improve the efficiency of data forwarding. The performance of the algorithm is evaluated by NS2 simulator. The results show that the algorithm has a good effect on the package delivery rate and end-to-end delay.

References

[1] Abbas, N.I.; Ilkan, M.; Ozen, E. (2015). Fuzzy approach to improving route stability of the AODV routing protocol, EURASIP Journal on Wireless Communications and Networking, (1), 1-11, 2015.
https://doi.org/10.1186/s13638-015-0464-5

[2] Agrawal, S.; Tyagi, N.; Iqbal, A.; Rao, R.S. (2020). An intelligent greedy position-based multi-hop routing algorithm for next-hop node selection in VANETs, Proceedings of the National Academy of Sciences, India Section A: Physical Sciences, 90(1), 39-47, 2020.
https://doi.org/10.1007/s40010-018-0556-9

[3] Cao, W.J. (2016). Optimization of AODV protocol for vehicular network, Nanjing University of Aeronautics and Astronautics, 2016.

[4] Ghazzai, H.; Ghorbel, M.B.; Kadri, A.; Hossain, M.J.; Menouar, H. (2017). Energy-efficient management of unmanned aerial vehicles for underlay cognitive radio systems, IEEE Transactions on Green Communications and Networking, 1(4), 434-443, 2017.
https://doi.org/10.1109/TGCN.2017.2750721

[5] Karp, B.; Kung, H.T. (2000). GPSR: Greedy perimeter stateless routing for wireless networks, In Proceedings of the 6th annual international conference on Mobile computing and networking, 243-254, 2020.
https://doi.org/10.1145/345910.345953

[6] Kim, S. (2019). Effective crowdsensing and routing algorithms for next generation vehicular networks, Wireless Networks, 25(4), 1815-1827, 2019.
https://doi.org/10.1007/s11276-017-1632-9

[7] Kumar, I.; Sachan, V.; Shankar, R.; Mishra, R.K. (2018). An investigation of wireless S-DF hybrid satellite terrestrial relaying network over time selective fading channel, Traitement Du Signal, 35(2), 103-120, 2018.
https://doi.org/10.3166/ts.35.103-120

[8] Li, C.; Han, J.H.; Wei, Z.C. (2015). GPSR-R routing algorithm in VANET Scenario, Journal of Hefei University of Technology, 38(2), 181-185, 2015.

[9] Li, F.; Song, X.; Chen, H.; Li, X.; Wang, Y. (2018). Hierarchical routing for vehicular ad hoc networks via reinforcement learning, IEEE Transactions on Vehicular Technology, 68(2), 1852- 1865, 2018.
https://doi.org/10.1109/TVT.2018.2887282

[10] Li, Y.; Shi, D.; Bu, F. (2019). Automatic recognition of rock images based on convolutional neural network and discrete cosine transform, Traitement du Signal, 36(5), 463-469, 2019.
https://doi.org/10.18280/ts.360512

[11] Li, G.; Sun, Q.; Boukhatem, L., Wu, J.S.; Yang, J. (2019). Intelligent vehicle-to-vehicle charging navigation for mobile electric vehicles via VANET-based communication, IEEE Access, 7, 170888- 170906, 2019.
https://doi.org/10.1109/ACCESS.2019.2955927

[12] Lin, D.; Kang, J.; Squicciarini, A.; Wu, Y.; Gurung, S.; Tonguz, O. (2016). MoZo: A moving zone based routing protocol using pure V2V communication in VANETs, IEEE Transactions on Mobile Computing, 16(5), 1357-1370, 2016.
https://doi.org/10.1109/TMC.2016.2592915

[13] Perkins, C.; Belding-Royer, E.; Das, S. (2003). RFC3561: Ad hoc on-demand distance vector (AODV) routing, IEEE Personal Communication, 36-45, 1997.
https://doi.org/10.17487/rfc3561

[14] Safiulina, A.M.; Ivanets, D.V.; Kudryavtsev, E.M.; Baulin, D.V.; Baulin, V.E.; Tsivadze, A.Y. (2019). Liquid- and solid-phase extraction of uranium(VI), thorium(IV), and rare earth elements( III) from nitric acid solutions using acid-type phosphoryl-containing podands, Russian Journal of Inorganic Chemistry, 64(4), 536-542, 2019.
https://doi.org/10.1134/S0036023619040181

[15] Schoeneich, R.O.; Prus, P. (2018). Improving DTNs performance by reduction of bundles redundancy using clustering algorithm, International Journal of Computers Communications & Control (IJCCC), 13(4), 550-565, 2018.
https://doi.org/10.15837/ijccc.2018.4.3180

[16] Senthilkumar, R.; Tamilselvan, G.M.; Kanithan, S.; Arun Vignesh, N. (2019). Routing in WSNs powered by a hybrid energy storage system through a CEAR protocol based on cost welfare and route score metric, International Journal of Computers Communications & Control (IJCCC), 14(2), 233-252, 2019.
https://doi.org/10.15837/ijccc.2019.2.3184

[17] Venkatramana, D.K.N.; Srikantaiah S.B.; Moodabidri, J. (2018). CISRP: Connectivity-aware intersection-based shortest path routing protocol for VANETs in urban environments, Iet Networks, 7(3), 152-161, 2018.
https://doi.org/10.1049/iet-net.2017.0012

[18] Wu, C.; Kumekawa, K.; Kato, T. (2010). Distributed reinforcement learning approach for vehicular ad hoc networks, IEICE transactions on communications, 93(6), 1431-1442, 2010.
https://doi.org/10.1587/transcom.E93.B.1431

[19] Wu, Z.; Chen, J.; Sun, X.Y.; Qu, L.D. (2017). AODV routing method based on prediction of nodes' moving direction, Computer Engineering and Design, 38(9), 2296-2301, 2017.

[20] Xiao, D.G.; Peng, L.X.; Song, D. (2012). Improved GPSR routing algorithm in hybrid VANET environment, Journal of Software, 23(S1), 100-107, 2012.

[21] Xiao, D.G.; Peng, L.X.; Song, D. (2012). Improved GPSR routing algorithm in hybrid VANET environment, Journal of Software, 23(S1), 100-107, 2012.

[22] Xiao, L.; Lu, X.; Xu, D.; Tang, Y.; Wang, L.; Zhuang, W. (2018). UAV relay in VANETs against smart jamming with reinforcement learning, IEEE Transactions on Vehicular Technology, 67(5), 4087-4097, 2018.
https://doi.org/10.1109/TVT.2018.2789466

[23] Yao, L.; Wang, J.; Wang, X.; Chen, A.L.; Wang, Y.Q. (2017). V2X routing in a VANET based on the hidden Markov model, IEEE Transactions on Intelligent Transportation Systems, 1-11, 2017.

[24] Yuan, M. (2017). Research on VANET routing algorithm based on reinforcement learning, Xi'an: Xi'an University of Electronic Science and technology, 2017.

[25] Zhang, X.L.; Zhao, Q.; Zhang, T. (2016). Improving GPSR routing protocol in vehicular Ad Hoc network, Journal of Highway & Transportation Research & Development, 11(4), 98-103, 2016.
https://doi.org/10.1061/JHTRCQ.0000601

[26] Zhang, D.G.; Ge, H.; Liu, X.H.; Zhang, X.D.; Li, W.B. (2018). A new adaptive routing algorithm based on Q-Learning strategy, Journal of Electronics, 46(10), 2325-2332, 2018.
Published
2020-07-06
How to Cite
YANG, Xiaoying et al. V2V Routing in VANET Based on Heuristic Q-Learning. INTERNATIONAL JOURNAL OF COMPUTERS COMMUNICATIONS & CONTROL, [S.l.], v. 15, n. 5, july 2020. ISSN 1841-9844. Available at: <http://univagora.ro/jour/index.php/ijccc/article/view/3928>. Date accessed: 28 sep. 2020. doi: https://doi.org/10.15837/ijccc.2020.5.3928.

Keywords

V2V communication, VANETs, ITS, Q-learning