Generate a policy: having derived an ideal set of protocol configurations a neural network must be trained and embedded within an adaptation policy

The result of these three stages is an adaptation policy that may be loaded into the DRoPS runtime framework and used to control the configuration of a RAP based system.

12.3.1 Evaluating Protocol Performance

The evaluation of protocol performance is performed by brute force experimentation.

During protocol specification, a configuration file is used to identify microprotocol resources and default protocol configurations. Using this file it is possible for the APSL parser to automatically generate client and server applications that evaluate the performance characteristics of all valid protocol configurations.

Evaluating every protocol configuration in a static environment, where connection characteristics remain fixed, does not account for the protocols performance over real world connections in which connection characteristics are potentially variable. To function correctly in such circumstances an adaptation policy requires knowledge of how different configurations perform under different conditions. To simulate precisely defined network characteristics, a traffic shaper is introduced. This intercepts packets traversing a host’s

network interface, incurs bit and block errors, introduces variations in delay, loses packets and restricts throughput. The shaper is contained within a microprotocol module that may be dynamically loaded into the runtime framework. In concept this module lies on the connection between the client and server evaluation applications. In fact it is located in the runtime framework at the server side node. A basic overhead of 0.0438 microseconds is incurred for each invocation (measured on a Pentium II 300MHz based machine) with additional undefined overheads for the shaping code. The use of a modular structure permits protocol designers to add their own code for the shaping of network characteristics. Each microprotocol posses a control interface allowing instructions to be passed from the runtime framework. This permits an application to simulate numerous network conditions by scaling parameters, such as residual error rate, used by the shaper mechanisms.

Figure 12.4 presents a a pseudo-code algorithm that forms the core functionality of the evaluation application. To evaluate the performance of a configuration, two individual tests are performed; a ping test and a raw throughput test. The former is used to judge latency and jitter whilst the latter determines throughput and error detection capabilities. Rather than burden an application with detecting loss and corruption, the error checking functions themselves are required to post notification of any observed losses or errors to a shaper structure. If the function is not included in the protocol configuration the error will not be detected and will go unreported. In addition, mechanisms within the shaper keep track of the number of losses and errors that are caused. After the ping and throughput tests these statistics are combined to determine the configurations performance within the simulated environment. To obtain accurate statistics, each configuration must be evaluated several hundred times for each condition. Depending upon the actual condition, each evaluation can take several seconds to perform. The minimal number of evaluations that must be performed for each condition are:

Eval F

K

k k⋅

∏

where Eval is the number of evaluations performed for each configuration and the rest of the notation is consistent with equation 12.1. Even in the simple case the evaluation of all configurations can take tens of hours to complete.

The result of evaluation is a performance profile for every protocol configuration in each operating environment. This takes the form of a simple report for each combination of configuration and connection characteristic. Figure 12.5 illustrates such a report for a RAP configuration operating in a connection with a latency of 300 microseconds and a bit error occurring for approximately every 8 megabytes of data. The performance of this configuration occupies the remainder of the report. For the example protocol, 1943 reports were generated. These were generated by considering the protocol functions introduced in Table 12.2 with only four QoS characteristics (each assuming three states).

12.3.3 Evaluating Fitness of Configuration

The second stage attempts to determine the most appropriate configuration for each combination of requirements and conditions. This relies on the report file generated as

output from configuration performance evaluation. Unlike the previous phase, where the evaluation applications are automatically generated by the APSL parser, the responsibility for creating the fitness evaluation application falls to the protocol designer. For each entry in the report file, the fitness evaluator has to determine how well every combination of application requirements is served. Figure 12.6 presents a pseudo code example of the core fitness evaluator function for requirements of loss, error, delay and jitter. The fitness evaluation function reads individual records from the report file described in the previous section. For each report, the configurations performance is evaluated for every combination of application requirements. Evaluation is performed by a fitness function that must be defined by the protocol developer. When considering how well a particular configuration satisfies an applications requirements in a particular operating environment each combination is assigned a fitness value. The mechanism used to generate this value depends upon the desired objectives of the adaptation policy. The fitness function used by the example adaptation policy is based on a weighted sum with two objectives:

/* initialise protocol configuration */

initialise_environmental_conditions();

do {

/* initialise environmental conditions */

initialise_protocol_configuration();

/* evaluate every protocol configuration in this

* environment */

do {

/* PERFORMANCE TEST THE CURRENT PROTOCOL

* CONFIGURATION */

evaluate_configuration();

/* next possible configuration */

cycled_through_all = increment_configuration();

} while( !cycled_through_all );

/* increment environmental conditions */

cycled_through_all_conditions = increment_conditions();

} while( !cycled_through_all_conditions );

Figure 12.4 Pseudocode for the generation of protocol performance characteristics.

Figure 12.5 Performance report generated by performance evaluation. weights used to trade off between satisfaction of requirements and maximum throughput.

The primary objective of this function is to provide a protocol configuration that satisfies the specified QoS requirements and the secondary objective is to maximize throughput. The components of this function determine the runtime objectives of the adaptation policy.

It should be noted that the significance of certain microprotocols can not be determined solely through performance evaluation. These atomic functions are either required or not.

Functions such as message and stream based fragmentation, encryption and compression can only be evaluated by the above fitness function in terms of their effect on observable characteristics. A fitness function developer may wish to place additional clauses in the fitness function to favour message-based fragmentation if it is explicitly required by the application. Perhaps these should be placed after the result of the neural controllers suggestion. In this way, suggestions by the controller may be overridden by application specifications for particular functions. Unlike the protocol performance evaluator, there is currently no mechanism for generating the fitness evaluation application. Fitness evaluation results in a data set containing the most appropriate protocol configurations for each requirement in each operating environment. The configurations suggested by this set are 264

conditions:

latency 300 jitter 0 bit errors 8000000 loss 0

configuration:

buffer 1 frag 1 addr 1 seq ctrl 0 flow ctrl 0 csum 0 ack 3

results:

throughput: ave 2.361896 MB/s

delay : min 2000.885010, max 2009.724976, ave 2008.575073 jitter : ave 4.419983 microseconds

loss : caused 0, observed 0 error : caused 539, observed 264

determined by the objectives of the fitness function and may be used to train the neural network adaptation policy.

for every report in the report file while( read_item_from_report_file ) { foreach (loss_requirement) { foreach (error_requirement) { foreach (delay requirement) { foreach (jitter requirement) { /*

* how well does current configuration * suit these requirements in current * environment

*/

fitness = evaluate_fitness_of_configuration();

/*

* update most appropriate configuration */

if ( fitness > best_fitness ) { best_fitness = fitness;

} } } } } }

write_training_set();

Figure 12.6 Pseudocode for the evaluation of configuration fitness.

12.3.4 The Neural Network

Training data for a neural network is generated from the fitness evaluation of each protocol configuration. The remainder of this subsection describes the training process using an example policy for the optimization of RAP.

A simple feed forward MultiLayer Perceptron (MLP) was created using the Stuttgart Neural Network Simulator (SNNS). The SNNS is a software simulator for neural networks developed at the Institute for Parallel and Distributed High Performance Systems (IPVR) at the University of Stuttgart. The projects goal is to create an efficient and flexible simulation environment for research on and application of neural networks. The SNNS consists of two main components; a simulator kernel and a graphical user interface. The kernel operates on the internal network data structures and performs all operations of learning and recall. The

user interface provides graphical representations of a neural network and controls the kernel during the simulation run. In addition, the user interface has an integrated network editor which can be used to directly create, manipulate and visualise neural nets in various ways.

Choosing the optimal number of hidden layer nodes is an ad hoc process best determined through experimentation. An excess number of nodes can lead to large runtime execution overheads and too few nodes can lead to to poor classification and generalisation. The example MLP has nine input nodes, six hidden layer nodes (in a 2×3 arrangement) and nine output nodes. Four of the input nodes represent conditions whilst the remaining five represent user requirements. The topology of the example network, including logical partitioning of input and output nodes, is shown in Figure 12.7. A network is trained using the SNNS backpropogation algorithm with the patterns generated by the fitness function. The x-axis denotes the number of epochs, where each epoch represents the presentation of all training patterns to the network. The y-axis represents the sum of the squared differences at each output neuron between actual and required, referred to as the Sum Squared Error (SSE). The eagerness with which the network learns highlights how the mappings generated by the evaluation application are consistent enough to be successfully learnt by a neural network. Figure 12.8 visualizes the error development observed during the training of the example network. Various network topologies were implemented but no benefit was noticed in increasing the number of hidden layer nodes above 6, which is surprisingly small.

Figure 12.7 Mapping and logical grouping in example neural network.