This chapter addresses the problem of energy-limited RSUs in a vehicular net- work. A Markov Decision Process is formulated and solved using a reinforcement learning technique, namely the Q-learning algorithm. The resolution is a Protocol for Energy-efficient Adaptive scheduling using Reinforcement Learning (PEARL), which is proposed for the purpose of increasing the number of downloaded bits per unit time as well as avoiding the undesired event of a vehicle departing from the RSU’s communication range with an incomplete service request. After a sufficient training period, PEARL exploits the realized optimal scheduling policy, which outperforms three benchmark scheduling algorithms in terms of several QoS metrics. In partic- ular, the deployment of PEARL complements the RSU with the required intelligent identity, which serves to maintain the RSU’s operation throughout the whole dis- charge period as well as decrease the number of vehicles departing from the RSU’s coverage range with an incomplete service request.
This work is the first step of introducing machine learning techniques, in par- ticular, reinforcement learning, in order to optimize RSU scheduling in a vehicular network. Our future research is directed towards studying an energy-constrained ve- hicular network which is composed of a tandem of RSUs deployed on a long roadway segment. The feasibility of Q-learning as well as other machine learning techniques will be investigated in order to solve such a large-scale problem.
Chapter 6
Optimizing Downlink Traffic
Scheduling - The Multi RSU case
6.1
Introduction
The IoV is foreseen to support a full-fledged, smart and efficient ITS by provid- ing real-time traffic information, context-aware advertising as well as drive-through Internet access, provisioned through the help of stationary IoT GateWays (IoT-GW) deployed along roadways. The previous chapter presented supporting evidence about the fact that the significant barrier to the widespread deployment of IoT-GWs is the cost of provisioning electrical grid power connections, [6], as well as their remarkable energy consumption.Following the emerging need for energy-efficient wireless commu- nications as well as the fact that grid-power connection is sometimes unavailable for IoT-GWs, [69], it becomes clear and more desirable to deploy green energy-efficient IoT-GWs, which are equipped with large batteries rechargeable through renewable energy sources such as solar and wind power [70, 71]. Energy-efficient and QoS- oriented scheduling policies must be employed at the IoT-GW in order to guarantee a desired level of performance in an eco-friendly environment. The objective of this
chapter is to establish a smart visionary vehicular networking infrastructure similar to the one illustrated in Figure 6.1.
The major entangled challenge associated with the proper inauguration of a full- fledged connected vehicular network is the efficient control and management of the operation of multiple RSUs deployed in tandem along roadways. Indeed, the highly dynamic and stochastic nature of vehicular networks, the randomness in the vehicle arrival process as well as the diversity of the requested services give rise to a par- ticularly challenging scheduling problem for the efficient operation of the IoT-GWs. Multiple studies in the literature have addressed the scheduling problem in the con- text of V2I communications. For instance, the work in [28, 72, 73, 11, 80] proposed novel V2I scheduling algorithms for a single RSU equipped with an infinite power source. Other studies addressed the energy consumption issue in the single-RSU (e.g., [44] and [75]) and multi-RSU (e.g., [76]) scenarios, where the RSU is privileged with a priori knowledge of vehicle arrival instances and requests in order to resolve a complex optimization problem. The objective of this present work is to establish a universal, green, intelligent and scalable scheduling policy which acclimates to the random characteristics of a vehicular environment, overcomes the limiting assump- tions and deficiencies of previous studies and finally, establish a vigilant backbone ITS that supports the development of the IoV. Precisely, herein, we consider a long road segment where several IoT-GWs are deployed in tandem. The scenario is illustrated in Figure 6.1. Each IoT-GW is connected using fiber or cellular links to a backend ITS central server, which is the acting agent for all the communications that take place in the network. IoT-GWs collect high-dimensional inputs corresponding to the network characteristics, forwards the collected data to the central agent that devises appropriate actions. In this work, the central agent is trained to realize an optimal scheduling policy that meets the following objectives:
Backend ITS Server Large Batteries WE HAVE A SITUATION!! GG
I will leave the road on Exit 2 Exit 1 Entry 1 I am leaving the roadway! I am entering the roadway! I am downloading a song! I am checking my email! DC I am taking pictures of speeding vehicles! 1 Exit 2 Entry 2 2 G2 G1
. . .
Collection of Input Devised ActionFigure 6.1: Energy-Limited Multi-RSU Vehicular Network
1. Communicate safety messages with minimum latency.
2. Minimize the mean response time as well as the mean total delay of non-safety- related download requests.
3. Satisfy the vehicles’ download requirements before their departure from the road.
4. Maintain the entire vehicular network up and running by balancing the power consumption at each IoT-GW.
In the study presented in [12], the authors considered a vehicular network com- posed of a single battery-powered RSU and developed an MDP framework with dis- cretized states in order to establish an optimal RSU-controlled scheduling policy. Therein, the resolution of the MDP was realized using reinforcement learning tech- niques [34]. The scalability of the solution proposed therein is poor especially when the number of state and action variables increase. Truly, the size of the table used to store Q-values grows exponentially and the system fails to distinguish between sim- ilar states and actions. Furthermore, the time required for the Q-table to converge increases intractably. In this present work, the state space dimension is enormously
large. Hence, the required computational time and effort to realize an optimal schedul- ing policy is prohibitively large, a phenomenon commonly referred to as the curse of
dimensionality [35]. Consequently, recent advances in training deep neural networks
(i.e. function approximation techniques) are exploited herein in order to overcome this complexity and, thus, promote the feasibility of instantiating a fictitious artifi- cial agent that will: a) learn a scheduling policy from high-dimensional inputs using end-to-end deep reinforcement learning, b) derive efficient representations of the en- vironment, and c) progress towards the development of a successful scheduling policy which meets the above-detailed objectives.
The remainder of this chapter is structured as follows. Section II lays out the prob- lem statement and motivation as well as the novel contributions of this present study. Section III presents an overview of the most relevant related work in the context of IoV, V2I scheduling algorithms as well as deep reinforcement learning. A description of the V2I communication scenario is presented in Section IV. Section V presents the adopted vehicular mobility model. Section VI lays out a detailed presentation of the deep reinforcement learning model. An MDP model is formulated in Section VII. The performance of the proposed deep reinforcement learning algorithm is examined and compared to other existing scheduling heuristics in Section VIII. Finally, concluding remarks are presented in Section IX.