Simulation Tuning And Calibration

A secondary output of the simulation is a set of holding area moves which can be performed to achieve the take-off sequence. These moves give the same information as the triplets in the triplet model, described in section 4.2.

The numerical delay and CTOT compliance results of the analysis of the final sequences obtained will indicate situations where the path allocation system or feasibility check are not flexible enough to obtain schedules with low delay and a low number of CTOTs missed. The numbers will not show whether the allocated paths would be acceptable to a runway controller, so methods were needed to verify this.

It is important to calibrate and validate any simulation and this can be complex [8]. Validation of the decision support system and simulation described in this thesis was performed by providing a visual display of the operation. The visualisation is not intended to be any kind of final user interface for the system, but instead was designed for obtaining information from the experts, such as air traffic controllers, and for illustrating to them the acceptability of the solutions from the final tuned system. It is not uncommon to provide a visualisation for such a system, for example the visualisation of the ATOS system [143, 96, 94] was discussed in section 3.8.2.

The path allocation algorithm suggested in this thesis is deterministic. Once tuned to controller preferences, the algorithm will continue to give acceptable results, but the difficult task is tuning it to controller preferences in the first place. It is not possible to automate this sort of tuning so holding area movement replay software was developed for this purpose.

The holding area movement described by the triplets can be replayed using the holding area movement viewer, described in section 7.6.2. The holding area viewer is ideal for obtaining feedback from problem domain experts and for comparing a replay of the sequencing the system achieves against the real sequencing that was performed. Even a count of how many times each path had been allocated would not show whether the allocation of a slower path was valid at the time the allocation was made. The viewer described in section 7.6.2 provides a fast time playback of the sequencing, so that the flow of aircraft through the holding area can be viewed and the path allocation can be considered and tuned.

The simulation works by presenting a series of problems for the decision support system to solve. The realism of these problems needs to be verified if the results from a simulation are to be accepted as viable. An optional user interface was developed for the simulation in order to simplify the examination of the problems created at each step, in addition to the playback facilities already discussed. The user interface allows step by step movement forwards or backwards through the simulation, so that the state at any stage in the simulation can be examined.

The replay program was mainly used to verify the acceptability of the path allocation rules. The user interface on the simulation was mainly used for verifying the realism of the

problems created by the simulation.

7.6.1 Running With A User Interface

A user interface is helpful to illustrate what the search is doing in real time. The empirical results in chapter 8 are presented for experiments which were executed in the absence of a user interface (although sample experiments were re-executed with a user interface just to ensure that performance was as expected). This ensured that experiments could be performed as batch processes and the results analysed later.

Components of the user interface

The user interface has three components. The first is a simple control window, allowing a user to step forward or backwards through the stored sequences, or to display a real-time update showing each new sequence as it is received. The sequencing information includes both the positional information for aircraft and the chosen take-off sequences. This information is displayed in the other two windows. Sequencing information can also be saved to a file and loaded later for examination and playback. This allows a detailed examination of the sequencing that the system performed and of the holding area positions the simulation predicted for aircraft.

The second window, shown in figure 7.1, displays information about the aircraft in the take-off sequence. The suggested take-off sequence and predicted take-off times can be seen. All times are shown relative to the start time of the dataset.

The first column shows an identifier for the aircraft and the position of the aircraft in the first-come-first-served sequence. The second column shows the predicted take-off time for the aircraft. Any separation which is more than the minimum one-minute is highlighted in red. An earliest take-off time can be estimated for an aircraft based upon the holding area arrival time, traversal time and the results of the feasibility check. This time is shown in the third column and is important in understanding why aircraft were not sequenced to take-off earlier than they did.

The departure route, weight class and speed group determine the separation rules. The allocated departure route (SID) is shown in the fourth column. The rough direction of departure (for example, North, South or West) is shown in parentheses to aid in understanding the sequencing.

Most aircraft are medium weight class and are speed group three. Where the charac- teristics of an aircraft differs from these norms, the information is noted in the fifth column. The final column in the usual display shows the allocated CTOT time for the aircraft. Aircraft must take off no more than five minutes before the CTOT time and should take off no more than ten minutes after the CTOT time. A number of extensions are available, allowing aircraft a five minute extension to the end of the CTOT window. Any take-off after that time is likely to require a new CTOT to be determined for the aircraft.

Optional extra columns provide more detailed information including the allocated traver- sal paths through the holding area and can show both the actual timings and the times which include the prediction errors to account for any simulated uncertainty.

Figure 7.1: An example take-off sequence

The third window, shown in figure 7.2, is actually the same as used by the holding area playback tool described in section 7.6.2 and is used to show the positions of aircraft within the holding area. This can aid in understanding why the decision support system makes the decisions that it makes.

This real-time feedback is especially useful for investigating the validity of the simulation system as well as the decision support system itself. For example, the display will show whether the predicted positions of aircraft are reasonable. It was precisely this feedback, and the replay of the holding area movement, that revealed the initial problems with positional predictions that are discussed in section 7.7. These problems were subsequently rectified.

Once the path allocation system had been tuned in this way and appropriate measures implemented to ensure the sequencing, movement and predicted positions were sensible, these areas were not assessed any further, beyond sampling to ensure that the measures continued to work as expected and that bugs had not been inadvertently introduced.

Figure 7.2: The holding area contents during a simulation

7.6.2 The Holding Area Viewer

The path allocation system was tested using the holding area viewer. This consisted of a graphical display similar to that 7.2 and a control window with controls to load new files, pause, play, step forwards or backwards through the playback and alter the displayed details or playback speed.

Due to the way in which the decision support system works, guarantees can be given that unacceptable traversal paths will never be used and less desirable paths will only be used when doing so is justified by the sequencing requirements. This can only be achieved once the controller preferences have actually been identified. The main purpose of the holding area viewer was to obtain feedback from the problem domain experts about whether the path allocation was acceptable. This is important since a consideration of the historic data is not always sufficient. One path in particular was observed to be used in the playback of the historic data, but playback to a controller of a sequence generated using this path allocation prompted the feedback that he would not usually use it as it was too difficult. The secondary purpose of the holding area viewer

was to demonstrate and verify that the path allocation actually performed as expected and that the holding area movement was simple enough to be acceptable.

The holding area movement viewer is given three pieces of information. The first is the set of triplets (similar to those described in section 4.2) which define the movement which took place in the holding area during a simulation run of the decision support system. The next is a background image to use, which shows the structure of the holding area. The third consists of the coordinates on the holding area picture of each of the nodes of the holding area graph.

With this information the holding area viewer can replay the movement of aircraft from the triplets specifying the old and new positions of each aircraft. Each aircraft is represented by a labelled circle on a background image for the holding area structure. Processing of a triplet involves moving the representation of the aircraft from the coordinates for the source node to the end node. By interpolating the coordinates for multiple points between the two nodes, the illusion can be given of a relatively smooth movement for the aircraft through the holding area structure. The number of interpolation points can be modified to increase or decrease the speed of replay and the tool can step backwards, forwards or run in a continuous play mode. An additional coordinate is available, if necessary, for any pair of nodes, specifying the mid-point on the path between the nodes. Using this, the path can appear to bend, so that aircraft will appear to more accurately follow the layout of the holding area.

Another feature of the playback tool is the ability to re-sequence triplets, or provide simultaneous movement. Where the sequence in which triplets are processed does not affect the feasibility of re-sequencing, the tool can swap the order in which the triplets are played back, if the reversed order appears more realistic. Similarly, some triplets can actually take place simultaneously, where the nodes and aircraft involved are disjoint, and the playback tool is capable of processing these at the same time, giving the more realistic appearance of multiple aircraft moving through the holding area at the same time.

In summary, the playback tool provides an easy facility to view the holding area move- ment, for comparison against real movement and for demonstration to runway controllers. It was invaluable in collecting information about controller preferences.

7.7

Holding Area Position Prediction

The simulation predicts positions for aircraft in the holding area by requesting the decision support system to perform a modified feasibility check and taking a snapshot of the positions during this. The snapshot is taken at the point at which the first aircraft enters the runway node or when the simulation time is advanced, whichever is earlier. These positions are then used as the predicted initial positions for the next problem presented to the decision support system. When the user interface was used to examine the positions that were generated, two related problems were seen.

The first problem was due to the point at which the snapshot is taken. At this point, it is possible that some aircraft have not had the opportunity to be moved forward through the holding area. A real controller would usually only advise aircraft to hold further back in the holding area if it was necessary in order for overtaking to take place, as doing so increases congestion at the holding area entrances. In practice, aircraft would probably be moved as far forward as possible but this was not always being done by the simulation. It would be useful to move aircraft forward before the snapshot is taken if they can do so without causing any sequencing problems.

The second problem is linked to taxiway congestion. Holding aircraft further back in the holding area, in order to provide possibilities for other aircraft to overtake them, can result in a queue of aircraft behind them, possibly blocking the taxiways and causing problems for the controllers. Unnecessary holding area or taxiway congestion may then occur behind these aircraft. This situation would not be permitted to occur in practice so any simulation should also not permit it. Controllers would not leave aircraft congesting the taxiways if it is at all possible to fit them into the holding area.

7.7.1 Additional modifications to enhance realism

To simulate the behaviour of a real system, some further checks are performed immediately prior to taking the snapshot of positions. Firstly, each aircraft is tested in turn to see if it can move forwards without either leaving the holding area or affecting the feasibility of the desired take-off sequence. This is performed by repeatedly checking each aircraft in turn and attempting to move it to the next node in the holding area, using the same rules about blocking that are normally used so that the feasibility of re-sequencing will not be affected. The only extra restriction is that aircraft cannot enter the runway. The effect of this is to move aircraft as far forward as possible in the holding areas, making room for aircraft on the taxiways to enter the holding area.

If the taxiways are not clear after this has been performed, the process is repeated with relaxed movement restrictions. The normal blocking system uses a simple rule: ‘Do not enter a node before a higher priority aircraft that is entering from a different node, unless there is room to immediately move out of the way.’ In this second stage, a slightly modified rule is used: ‘Do not enter a node before a higher priority aircraft that is entering from a different node, unless it will be possible to move out of the way by the time that aircraft needs to take off.’ This modified rule is more time consuming to apply but, of course, is only being applied to the final sequence rather than to every solution that is evaluated during the search, so the extra time is not a problem.

The normal movement rules require empty nodes further in the holding area to move a potentially blocking aircraft into. The modified check considers the priorities of aircraft in these nodes and treats the node as empty if the priority is such that the aircraft will have taken off by the time the potentially blocking aircraft must move. This is not desirable from the point of

view of providing schedule flexibility as it possibly leads to aircraft temporarily blocking other aircraft which take off before them and so is normally avoided. However, it does mean that aircraft can move further into the holding area, possibly freeing nodes so other aircraft can move off the taxiways.

In the experiments performed for this thesis, the first method (giving aircraft the op- portunity to move forward before taking the snapshot) usually cleared taxiway congestion. In the few cases where this did not clear it, the second method, relaxing the movement restrictions, always did so. The selected recovery method if these fail was to accept the taxiway congestion but this was never needed in experimentation.

With these enhancements, the predicted positions for aircraft within the holding areas were observed to be much more realistic. If the system did actually leave taxiway congestion at any point, the fact was flagged up in the output files. It was not observed in the experiments. These changes are only used by the simulation and this kind of holding area prediction would not be needed in a live situation.

7.8

Sequence Stability

Ideally, a decision support system should give advice and never have to change it. That ideal is not practical, however, as the changing situation and increasing knowledge over time may mean that there are advantages to be gained from re-visiting earlier suggestions. Assuming that the simulation steps are such that only a few aircraft are added to the system at each step, any sequence other than the first-come-first-served sequence would require new aircraft to be fitted into an existing take-off sequence. This would alter the positions of aircraft already in the take-off sequence, so some positional deviation over time is usually expected.

7.8.1 Observations of the strict insertion sequence

The sequence created by inserting each aircraft, in first-come-first-served order, into the existing take-off sequence will here be called the strict insertion sequence. In fact, the strict insertion sequence is also highly unlikely to be a good take-off sequence. To see why, consider the fact that the departure route separations mean that sequences that represent alternate departure routes are often good, thus partial sequences that are found are likely to have this form. As described in section 5.4, the insertion of a new aircraft into such a sequence is likely to introduce a larger separation into the take-off sequence.

There are reasons, such as a tight CTOT or specific aircraft characteristics, that may mean an aircraft must overtake a number of aircraft in the current take-off sequence. With a strict insertion sequence, this would introduce a large separation. Only the insertion of further new aircraft into the sequence would rectify the problem. Even if a later aircraft is available to be inserted into the gap, advancing such an aircraft unnecessarily is unlikely to be the most

equitable take-off sequence.

In summary, therefore, some re-sequencing is almost certain to be required when new aircraft enter the system in order to find good take-off sequences. Due to this requirement, it is not advisable to merely measure the amount of re-sequencing that is performed and use this as a measure of the stability of the sequencing. What is needed is a method to determine whether unnecessary re-sequencing is being performed; in other words, to identify whether the re-sequencing that is performed is actually necessary or not.

Not all change has the same cost. For example, the early part of the sequence is important as these are usually the aircraft that are already within the holding area and under