Neural Network Structure

As is well-know, all neural networks are composed of a structure of interconnected primitive Processing Elements (PE). In most neural networks, processing elements are arranged into layers with each PE in one layer having connections to each PE in the next layer (sometimes PE’s in the same layer are connected). Associated with each interconnection is a weight and with each processing element is a state (normally on or off but can have a value). The combination of weights and states represents the knowledge of the network. Figure 2.3 shows a processing element operating as a simple threshold device. ♦ s (a) Sigmoidal Function f(net) 1 0.5 -10 0 net 10 (b) Figure 2.3 — A Neural Network Processing Element.

The output of the PE is dependent on the sum of the Inputs (5/) and the input connection weights (W/). A neural network “leams” information by modifying the connections between processing elements. This is analogous to synaptic weight changing in the brain. Neural network models have received a vast amount of research, which has spawned various classes of models, each with their own strengths and weaknesses [79]. The fundamental characteristics of various neural network models are shown in Table 2.4. These models differ on several aspects: network topology, recall phase, and learning phase.

Network topology ranges from single layer, feed-forward models, such as the Perceptron [73], through single layer with feedback connections, as the Hopfield model [49], to more complex interconnection patterns, found in the multi-layer networks with back propagation of errors [92], and also two-dimensional grid of neurons of the Self- Organising Map [59].

Chapter 2 Symbolic Processing versus Adaptive Processing 32

The recall phase / j , formed by the propagation rule and activation function,

presents some variations among these neural models. The propagation rule basically calculates the weighted sum of the input states. The activation function is usually restricted to either a threshold function, pseudo-linear function, or sigmoid families function.

The learning phase (J2* fs ) differs considerably among the models. In algorithms such as Hopfield/Kohonen’s associative memories [49, 59], the learning phase does not involve any error calculation (shown in Table 2.4 as x ), but generally most models depend upon an error calculation. They can be as simple as the difference between a target value and the neuron’s output state (Perceptron [73] and Delta Rule [54]), or can involve more complex computations, such as the Back Propagation model [92] in which the error calculation requires the derivative of the activation function (T(net)).

Neural Network Model Network Topology Range of input values Recall/Learning Phase

/ i - Recall Phase Learning Phase Propagation Rule Activation Function f i - Weight Updating /3 - Error Calculation Hopfield/ Kohonen single-layer with feedback binary n e t = 'Z s . w hard limiter Aw^. = s^.S j

_X

Perceptron single-layer feed-forward binary or continuous n e t = 'Z s . w hard limiter A w ,j - T \.s,.ej e j = t j - s , Widrow- Hoff (Delta Rule) single-layer

feed-forward continuous n e t = ' £ s . w linear AWfj - T\.Si.Ej e , = t j - s i Back

Propagation

multi-layer bidirectional

links

continuous n e t = Y .s .w sigmoid Awy =

e„ = r ( n e t ) . U i - S i ) = T\net)!^E.W Boltzman Machine multi-layer or randomly connected binary n e t = S -s-w sigmoid * J = T l ( ( p y ) - ( p i } > ) Counter Propagation multi-layer feed-forward binary n e t = Y ,s .w hard limiter *72 — —^2 Self Organising Map 2-dimensional grid of output PEs continuous n e t = sigmoid e j = S i - w . j

Table 2 .4 — Characteristics of Some Popular Neural Network Models.

2.3.3. Development Cycle

“Programming” a neural network consists of specifying the Activation and Transfer functions [117, 79] for the processing elements and their interconnections, followed by the training of the network with input patterns. The full neural network development cycle can be seen in Figure 2.4.

Chapter 2 Symbolic Processing versus Adaptive Processing 33

Design Problem

Collect train and test data

Design pre- and post­ processing Design Network Train Network Test Network Finished

Figure 2 .4 — Neural network development cycle.

Learning is the most important attribute of a neural network. During learning, input information is presented repeatedly and then the neural network produces an output and adjusts the pattern of weights to capture the underlying pattern of the input set. Learning can be classified into supervised and unsupervised learning. In supervised learning, input patterns plus the expected output patterns are repeatedly presented and an error is produced for the difference between the desired and actual pattern. This error is used to adjust the corresponding connection weights as in the formula above. In unsupervised

learning there is no need for input and expected output pairs, all that is required is enough sample inputs that the network will self-organise to reflect higher-order regularities within the input set. There also is a middle ground of graded learning in which the network is not given desired output for a input pattern but instead a performance score is given for the closeness of actual and desired values.

2.3.4. Applications

Neural Networks at present offer numerous basic-research possibilities and have found commercial and industrial applications within narrowly constrained tasks (mainly pattern recognition and generalisation systems [51]). Current neural network packages [112] exist that use either all software (neural network simulation packages run on conventional hardware), all hardware (special hardware [4, 82, 81]) or a combination of both. Some of the current applications include :

• Military/Aerospace: Aircraft pilot/navigation systems, radar/sonar/image processing, expert air traffic control, and autonomous vehicles.

• Communications: Voice and image recognition/compression, real-time language translation, continuous speech recognition, and network management.

• Manufacturing: Robot vision, adaptive robot movement [42], VLSI chip design, and processing control.

• Banking and Finance: Loan evaluation, consumer credit scoring, cheque and signature scanning, and predictive analysis of stocks and bonds [24].

• Medicine: Expert diagnosis systems, DNA code prediction, and medical image interpretation.

Chapter 2_________________Symbolic Processing versus Adaptive Processing________________________ ^

• Law Enforcement: Finger print identification, and face recognition.

All the above applications illustrate that neural networks have proven their strengths at prediction, classification and pattern recognition tasks.

2.3.5. Programming Environments

Programming environments range from commercial products from both established and start-up companies, to public-domain software, available free from university research groups. However all of these programming environments share many common features such as:

• an algorithm library of common, parameterised neural network algorithms, such as Back Propagation, Hopfield, Boltzmann etc.;

• a high-level language, often object-oriented, for programming or customising an algorithm or application;

• a network specification language, a low-level, machine-independent, language (often based on C) defining the neural network simulation;

• a graphic interface with menus and a command language: for configuring a neural network, then controlling and monitoring its execution;

• translators for mapping the network specification language to various target machines.

As shown in Figure 2.5, neural programming environments can be categorised with respect to the target user group, the algorithm and the application requirements [113].

1

User m. Proprietary i Architecture Busine^ • • N e s to r D L S Professional Novice User Computing Professional Neural Net • B e h av H eu rlstlc s AMT B ralnm aker • # N euroS hell • AI W are P y g m alio n C -llbrary # Owl * • A nne # SAIC A N Spec • HNC ANZA/Axon R o c h e s te r S im u la to r#

Open Architecture P y g m a lio n #

Expert % ^ Special-purpose General purpose

Applications Algorithm Programming

Oriented Oriented Systems Scope

C hapter 2________________ Symbolic Processing versus Adaptive Processing________________________ 35

Appendix B summaries some the most popular neural programming environments within these categories.

In document A Hybrid Object-Oriented Environment Integrating Neural Networks and Expert Systems (Page 32-36)