WP3: Communication model integration and evaluation/validation framework

(1)

Deliverable Lead: NTU

Author(s): Abdallah Namoun, Javad Akhlaghinia (UNIMAN), Johan Philips (KU LEUVEN), Stylianos Papanastasiou (NTU)

Contact: [email protected] Nottingham Trent University

School of Computing and Technology Clifton Lane, Nottingham, NG11 8NS United Kingdom Contributing partners: TML, UNIMAN, KU LEUVEN, NCC Delivery Date: 28 July 2014 Dissemination Level: Public Version: 1.2 (final approved) Disclaimer

The views represented in this document only reflect the views of the authors and not the views of the European Union. The European Union is not liable for any use that may be made of the information contained in this document. The user of the information provided in this document uses it at her/his sole risk and liability.

picture size: 65x90 mm at 300dpi (2480x1110 px) picture size: 65x43,5 mm at 300dpi (2480x1110 px) picture size: 65x43,5 mm at 300dpi (2480x1110 px)

D3.3.1: Report on analysis and evaluation of the final integrated simulation model

(2)

Document Information

Document Status

Deliverable Lead NTU

Type Report

Work Package WP3: Communication model integration and evaluation/validation framework

ID D3.3.2: Report on analysis and evaluation of the final integrated simulation

model

Due Date 31/12/2013

Delivery Date 28/07/2014

Status Final Approved

Deliverable History

Version Date Output Responsible

V1.0 19/12/2013 Initial draft version Stylianos Papanastasiou, Abdallah Namoun, Javad Akhlaghinia, Johan Philips V1.1 20/12/2013 Final edit Sven Maerivoet V1.2 28/07/2014 Split to comply with the DoW Sven Maerivoet

(3)

D3.3.1 Report on analysis and evaluation of the final integrated simulation model page 3

List of Figures

Figure 1 Testing turn probabilities in SUMO ...12 Figure 2 Detector location on the road network. Traffic flow is indicated by arrows ...15

(5)

List of Tables

Table 1 Number of vehicles choosing edge at intersection ...13 Table 2 Traffic estimator model validation at a particular corridor for two time points ...16

(6)

1 Introduction

Transport congestion problems contribute around 70% of pollutants to urban environments. The transport sector by itself consumes up to some 30% of the total energy in the EU. These figures suggest that if Europe is to reduce its CO2 emissions by making an efficient use of

energy while improving the quality of life in European cities, novel approaches for the optimal management of urban transport complexity must be developed and adopted in the transport sector.

MODUM addresses the environmental footprint in the transport sector by aiming to develop a new approach for pro-active demand-responsive management of traffic to enable energy-efficient multi-modal transport choices accommodating dynamic variations, minimising the environmental impact and improving the quality of life in urban environments. Moreover, MODUM considers commuters, in combinations of both private and public transport, facing dynamic conditions such as unexpected disturbances typical of urban environments.

In particular, MODUM employs the multi-agent system paradigm, which is used in a novel setting, i.e., for distributed coordination and forecasting (of, e.g., travel times) by means of self-organising virtual ants. In addition, multi-modal solutions are provided through a noticeboard and bidding approach using real-time data and declared destinations.

Both mechanisms have proven successful in other application domains and have the potential of utilising vehicles’ computational power and networking capabilities for achieving their active participation in the demand-response management of urban traffic.

The metrics for the comparison are extracted from real needs of traffic control centres and from transport users in our selected cities. Once the metrics are defined, a series of simulation experiments of realistic complexity is constructed using real-time data feeds available from transport sensing infrastructure. Results from these profile the two approaches against certain scenarios of traffic disturbances causing rapid changes in conditions. A synthesis of the two approaches is then developed by the academic partners.

Software implementation of the synthesised approach is embarked upon, focusing on the telecommunication challenges of a realistic demonstrator. The developed prototype is being validated on the initial scenarios by staging real-life experiments, which the relevant traffic management structures within the traffic control centres will evaluate. Such experiments include historical data and simulations in combination with real-time data feeds from existing infrastructure and vehicles going through a section of a city in a number of congestion profiles. Analogous experiments include people moving in a city by different means of transport. The prototype provides users in the field trials with personalised eco-friendly route advice. Consequently, in a scale-up with a higher degree of participation this might lead to lower carbon emissions in an urban complex environment.

(7)

D3.3.1 Report on analysis and evaluation of the final integrated simulation model page 7 The following diagram presents the use of the MODUM system in a nutshell:

Here we can see how a user request is processed internally in order to return a suggestion on a possible route and accompanying travel mode. It also clarifies the link between the various components in the system, as well as the relation with the external information (e.g. loop detectors, floating cars, …).

From the system point of view, the following diagram presents how the internals of the systems interoperate. We can clearly see where communication takes place, and how the MODUM core system itself is shielded from the outside (where user requests take place) via a firewall.

(8)

1.1 Purpose of this document

This report describes the testing conducted for each individual core component in the Integrated Experimental Environment. The term “core” refers to the multi-mode and ant agents models and implementation, as well as the traffic estimator. These interoperable components were tested, verified, and validated individually before integrating them into a working system whole. The complete integrated system is in turn evaluated in deliverable D5.3.

1.2 Structure of the document

This document is organised as follows. The following three sections describe the testing conducted for the three “core” components of the Integrated Experimental Environment in terms of the software development phase evaluation criteria outlined in deliverable D3.2 which contains MODUM’s evaluation and validation framework, namely in the areas of debugging, model verification, and validation. Specifically, Section 1.4 outlines the testing conducted on the traffic estimator simulator and its implemented model. Then, Section 0 contains a detailed description of the testing conducted on the multi-mode agent software and its associated model.

1.3 Individual testing criteria

The testing of the individual “core” components closely follows the evaluation criteria established in D3.2, where these apply to individual component functions rather than the integrated whole. In particular, testing per component, follows the first level of evaluation as stated in D3.2 in the areas of debugging, model verification, and evaluation. Each area is outlined below in turn. The following diagram also highlights their locations in the software development process:

(9)

D3.3.1 Report on analysis and evaluation of the final integrated simulation model page 9 checking the reproducibility of results when encountering specific traffic phenomena using a

pre-set dataset and confirming that the outcome of the model is as expected. Finally, model validation ensures that the model, given an observed dataset, is a reasonable representation of reality.

It should be noted that testing and evaluating the models and code, does not conclude with the delivery of this report. The developers understand that testing is a continuous process and are committed to ensuring that testing practices as described here are applied throughout the project's lifetime.

1.4 Scope of this deliverable

The scope of the working prototype is set to ensure that individual models and the accompanying software demonstrably function as intended and may be used with confidence. However, the evaluation of these discrete components as described here does not include testing them as an integrating whole; such an evaluation is described in detail in D5.3.

(10)

2 Traffic Estimator Evaluation

In this section, we present the evaluation conducted on the traffic estimator module, following the criteria established in D3.2 and in particular, the first evaluation level concerning the software development phase. Throughout, it is assumed that the reader is familiar with the architecture of the traffic estimator software deliverable as described in detail in D3.1.

2.1 Debugging

The development of the traffic estimator component was performed using an iterative prototype approach wherein core functionality was first implemented in a stable composite before any further development. Each iterative prototype was subject to extensive testing by both the NTU developers and the partners, which made use of the deliverables for early integration tests and proof-of-concept interactions.

We used two distinct paradigms to ensure correct software operations during development while implementing additional features, namely unit testing and employing best code practices. These are described in detail below.

2.1.1 Unit testing

In the context of the traffic estimator software deliverable unit testing refers to the automated checking of modules wherein a particular set of inputs are provided and particular outputs are expected. If the resulting outputs are not as expected then the developer inspects the relevant code segment under test and either alters the test case, in the rare case where the expected output was incorrect, or, more often, changes the code to produce the expected output. In time, a compendium of test cases exists, which when successfully run, increases confidence in the software deliverable, without, however, absolutely ensuring correct operation unless the test coverage is complete; the latter is an long-term goal, which is explicitly not pursued for prototype development due to time constraints.

We did not employ a test-driven development paradigm for the traffic estimator, to allow for fast-prototyping, however, every major individual component contains at a basic level test cases which verify its most basic aspect of operation. In particular, there are specific test cases for the Comms, Request Manager, Request Handler, Simulator Exec, and TrafficInfo modules. Notably, there are no significant unit tests that consider functions as the basic tested unit; instead we have focused on testing major functions of the significantly sized modules mentioned.

The test cases were run and their results inspected at the completion of each functional requirement. Further, each functionality implemented was scheduled to be accompanied by a test-case verifying its correct implementation – an aspiration which was largely met and still subject to ongoing work.

(11)

D3.3.1 Report on analysis and evaluation of the final integrated simulation model page 11 Most of the traffic estimator is implemented in Python 2.7 and its accompanying standard

library. As such, we opted to use the nose1 unit testing framework, which is mature, extensible and capable of producing results in an xUnit XML format [1]. The latter is of particular note as it allows the tests to be run under a continuous integration tool such as Jenkins2 to aid the software development process.

2.1.2 Best code practices

During development, the development team at NTU aims to employ best coding practices to produce maintainable and extensible code, suitable for future development and, potentially, transformable from prototype to production-ready status. The employment of such practices manifest in four areas, namely designing for modular code, adhering to specific coding guidelines, employing revision control, and including extensive logging facilities. Each area is now outlined in turn.

The traffic estimator's software architecture as outlined in D3.1 is a modular and extensible design, which may encompass expected future requirements, to some degree, without significant re-factoring. A more thorough discussion of this aspect of the code is included in D3.1 and, for brevity, is omitted here.

The code in the traffic estimator mostly adheres to the PEP 8 Style Guide for Python, which is a recommendation of coding conventions. The authors recognise that consistency and readability are indispensable traits of a maintainable code and, as such, have opted to follow the PEP 8 guide to aid future developers in understanding and expanding the software whilst avoiding unwanted side-effects. Notably, during development the code is regularly and automatically checked with the PEP 83 Python style guide checker to ensure adherence.

Importantly, all software development was performed using the Git revision control system. Although the distributed features of Git were seldomly employed, its extensive and well-documented branching and merging functions were often used as they fit the extensible core, iterative prototype development method. Specifically, each distinct feature was developed in a separate branch and then merged into the main development trunk when the work was sufficiently complete.

Finally, the traffic estimator includes extensive logging facilities using Python’s built-in logger. Such a practice allows for detailed, meaningful inspection after failure. There are different logging levels in the traffic estimator, as is standard practice, which may be customised per module through the “settings/logging.conf” configuration file; all logs are stored under the logs directory in the main traffic estimator directory.

1

The nose testing framework for Python may be accessed at: https://github.com/nose-devs/nose/ 2

The Jenking continuous integration server may be found at: http://jenkins-ci.org/ 3

(12)

2.2 Model verification

Model verification in the context of the traffic estimator refers to evaluating whether the simulation provides expected results given certain initial conditions. As succinctly stated in D3.2 the question to answer is whether the model was built right.

Following, we address the evaluation criteria as set in D3.2. However, it should be noted that as simulations in the traffic estimator are based on the widely used SUMO microscopic traffic simulator, several of the evaluation criteria have been previously addressed and we have only conducted limited verification in view of using such an established and trusted only conducted limited verification in view of using such an established and trusted component in the system. Below, we present the elements of evaluation pertinent to our input; i.e. we have evaluated features where we have introduced a new mechanism or integration with the existing well-tested SUMO simulator.

2.2.1 Vehicle generation and fleet composition

The traffic injection paradigm used in the traffic estimator model is as follows. Vehicles are generated at particular road segments and injected in traffic following a predetermined route, which is in turn dynamically generated based on turn probabilities. To test whether the traffic injection mechanism functions as intended we conduct experiments using a cross topology as shown in Figure 1.

(13)

D3.3.1 Report on analysis and evaluation of the final integrated simulation model page 13 Specifically, the edge tagged as “source” in Figure 1 is where the traffic is generated and

moves towards an intersection which forks to edges d2 and d1. Edges d2 and d1 are set with turn probabilities of 90% and 10% respectively. The injected traffic is set to 40 vehicles and the simulation time to 300 seconds. The turn and flow definitions file are included in Annex 4.1 and 4.2, where source, d1 and d2 correspond to edges 19061664, 17100630#1 and -17100630 respectively (the IDs are simplified in the Figure for clarity). The traffic is injected at a rate of one vehicle per second; note, however, that it takes more than a second for a vehicle to move sufficiently for another to spawn at its place of origin and in fact it takes approximately 120 seconds for all vehicles to spawn in the simulation.

We repeat the simulation of traffic injection 100 times and the cumulative results are shown in Table 1; both in absolute number of vehicles and percentage, shown in parenthesis. The results are very close to the configured behaviour (within a 0.5% margin), and the individual trial results deviate up to a 5% error margin from the expected value.

Turn to d2 Turn to d1 Total No. Vehicles

2690 (88.5%) 310 (11.5%) 4000

Table 1 Number of vehicles choosing edge at intersection

The above experiment is indicative of the accurate behaviour of the widely used SUMO simulator; we have in fact conducted several experimental trials with different turn ratios and more complicated intersections with success.

SUMO supports the inclusion of different vehicle types with custom or preset characteristics, such as buses, trucks, cars, and so on. However, fleet composition is not considered in our simulations and every vehicle is assumed to be a 4 m car, that obeys the legal speed limit. The previous statement stands without objection when one considers that in the traffic information sources available, it is not possible to identify different vehicle types. Nonetheless, if future evaluation indicates that considering a mix of traffic according to some preset ratio improves the traffic estimation results then such provisions may be enabled in simulations.

2.2.2 Travel demand

Travel (or traffic) demand is modelled in simulations using the turn probabilities model outlined above. Through interviews of knowledgeable experts at each of the test sites we have assigned turn probabilities on most busy roads, which allows the traffic to flow in a reasonably representative manner. Utilising local knowledge is particularly important as the great majority of the road segments in Nottingham (and all in Sofia) are not covered by road traffic detectors and, thus, it is particularly difficult to estimate how the flow of traffic evolves between subsequent road segments. Wherever detectors do exist in sequential or (nearby) road segments it is easier to estimate turn probabilities by directly examining correlations in traffic levels.

(14)

We are continuously experimenting with automatic assignment of road turn probabilities by considering street length and number of lanes; the intuition lies in that a long street with many lanes is a main corridor of traffic and vehicles are more likely to stay on it than divert to a side street. We have developed a utility which sets turn probabilities at junctions according to the exact ratio of the length of the streets; i.e. considering streets a, b and c, the turn probability would be:

l

turnProb(a) = length(a)/(length(a)+length(b)+length(c))

and similarly for streets b and c. Note that the term street refers to the collection of segments belonging to a particular street. Also note that the number of lanes contribute to the length of the street segment. This solution, however is not perfectly representative as it is biased against small street segments that lead to bigger streets and ignores the total number of intersections present along a big street segment. Work is ongoing to refine the model in this respect.

Finally note that second-order effects, such as considering traffic load on main links, which lead to drivers choosing alternate routes rather than the shortest one may be taken into account if reasonable Origin/Destination matrices become available. SUMO contains provisions to perform dynamic user assignment taking into account different demand definitions by calling iteratively the duarouter4 utility and distributing the traffic so as to divert it from the shortest path and distribute it via alternate routes.

2.3 Model validation

Model validation refers to applying the traffic model to an actual situation and estimating whether the model can predict the reality on the ground. As summarily stated in D3.2 the question to answer is whether the right model was built.

To calibrate the model parameters, i.e. the turn probabilities we have collected historical data and assigned turn probabilities according the recorded observations. In the case of main corridors with several detectors, probability assignment may be simply calculated by observing the traffic recorded at the starting point of the flow and calculating how much traffic “escapes” on side streets by observing subsequent points on the road network along the main corridor. To illustrate the point, consider the detector location in the city centre of Nottingham as depicted in Figure 2. Vehicles observed at detector N10231D propagate through and are recorded by detectors N20141A and N20141C. Observing less traffic on subsequent detectors along the traffic corridor indicates escaping traffic on side streets, whilst greater traffic indicates traffic accumulating from side streets. The calibration of the model is currently ongoing and it is expected that deployment of the CollabWifi road-side devices will significantly aid the process as these may be deployed to cover information gaps on the road network.

(15)

Figure 2 Detector location on the road network. Traffic flow is indicated by arrows

(16)

To demonstrate the validation process on the corridor as depicted in Figure 2, we present below the results of experiments where an observation at a particular time point is considered by eliminating one detector input traffic point, namely N20141C. Then, the traffic estimator returns a traffic estimate on that point based on traffic information provided by the other detectors. Since the ground truth is known, we note how close the estimate is to the actual measurements. Note that only the other two detectors, namely N20141A and N10231D contribute to the traffic observed at the road segment where N20141C is deployed. The results of the evaluation are presented in Table 2 for two time measurements on the same day and for two subsequent traffic detector updates.

Observed traffic at other sensors N10231D – 50 vehicles, N20141A – 39 vehicles (11:48am)

N10231D – 50 vehicles, N201141A – 21 vehicles (11:53am)

Observed traffic at N20141C 55 vehicles (11:48am) 51 vehicles (11:53am) Estimated traffic at N20141C 42 vehicles (14% error)

38 vehicles (26% error)

Table 2 Traffic estimator model validation at a particular corridor for two time points Although the turn probability details are omitted, consider that at the beginning of the corridor depicted in this example there is very high probability for a vehicle to stay on the corridor (at approximately that 95% level), which lowers as the traffic flow proceeds through the corridor, reaching a 20% level near the final junction. It should also be stated that there is static traffic at the end of the junction along the corridor, based on detector N19121D1, which is not depicted in Figure 2.

Finally, consider that validation of the traffic estimator is ongoing as the traffic model parameters are continuously calibrated based both on the increasing volume of historical data as well as the imminent deployment of new traffic information sources using the CollabWifi devices. Generally, we have observed that the error rate is lower near the traffic detector points and significantly higher elsewhere.

(17)

3 References

[1] Meszaros Gerard (2007), xUnit Test Patterns: Refactoring Test Code, London: Addison-Wesley

(18)

4 Annex

4.1 Turn definition xml file

</interval> </turnsType>

4.2 Flow xml file

WP3: Communication model integration and evaluation/validation framework

D3.3.1: Report on analysis and evaluation of the final integrated simulation model

Document Information

Table of Contents

List of Figures

List of Tables

1 Introduction

1.1

Purpose of this document

1.2

Structure of the document

1.3

Individual testing criteria

1.4

Scope of this deliverable

2 Traffic Estimator Evaluation

2.1

Debugging

2.1.1 Unit testing

2.1.2 Best code practices

2.2

Model verification

2.2.1 Vehicle generation and fleet composition

2.2.2 Travel demand

2.3

Model validation

3 References

4 Annex

4.1

Turn definition xml file

4.2

Flow xml file