We will now examine the process that JOONE uses to implement a feed forward neural network. As we step through the various classes and methods in JOONE it is important to keep one of JOONE’s design principles in mind. JOONE was designed to run as a distributed system.
By structuring JOONE to optimize for running as a distributed system two clear advantages were achieved. First, JOONE will run considerably faster on a multi-processor machine. Secondly, components of a JOONE neural network can be ran on different machines. This allows JOONE to distribute its workload across a "farm" of computers.
Central to JOONE’s distributed design is the fact that each of the neural network layers is designed to operate as autonomously as possible. When running on a single computer each of the neural network layers is run in a different thread. These neuron layers begin running in the background and wait for data to be transmitted to them. Data is transmitted to JOONE objects using the synapse objects. You will now be shown how JOONE recalls a pattern. Listing 5.1 shows the run method of the Layer class.
Listing 5.1: The Layer.run Method
public void run() { while ( running ) { int dimI = getRows(); int dimO = getDimension(); // Recall
inps = new double[dimI]; this.fireFwdGet(); if ( m_pattern != null ) { forward(inps); m_pattern.setArray(outs); fireFwdPut(m_pattern); } if ( step != -1 )
// Checks if the next step is a learning step m_learning = monitor.isLearningCicle(step); else
// Stops the net running = false;
//if ((m_learning) && (m_batch != 1))
gradientInps = new double[dimO]; fireRevGet();
backward(gradientInps);
m_pattern = new Pattern(gradientOuts); m_pattern.setCount(step);
fireRevPut(m_pattern); }
} // END while (running = false) myThread = null;
}
As you can see the run method begins a loop that is constantly waiting for new input pattern and then transmitting this output pattern. As soon as a layer is started this run method begins its loop. For the case of the XOR neural network that we examined in the previous section, there are actually three such layers that are running as threads. These layers are the input, hidden and output layer. We will now examine how the input pattern is processed by the input layer. The process by which the pattern moves through
subsequent layers is the same.
Examining the run method we see that the process is begain by calling the fireFwdGet() method is called.
this.fireFwdGet();
This method will block if there is no pattern waiting on its input synapse. When a pattern is retrieved it is stored in the m_pattern variable that is associated with the layer class. Next the code makes sure that a valid pattern has been retrieved.
if ( m_pattern != null ) {
If a valid pattern has been retrieved we will call our forward method. The forward method is responable for applying the weights of the output synapse to the input pattern. Finally, the fireFwdPut method is called to pass the pattern into the output synapse.
forward(inps);
m_pattern.setArray(outs); fireFwdPut(m_pattern); }
Just by examining the run method of the Layer class you are able to see the high level process that JOONE goes through as a neural network recalls a pattern. We will now examine in closer detail how these methods work. As the forward and fireFwdPut methods are called, three important actions occur.
• The connection weights are applied • The bias weight is applied
• The threshold function is applied
The first method that is called by the Layer class’s run method is the fireFwdGet method. This method is more specific to JOONE than it is to the "feed forward" neural network algorithm. The primary task of this method is to accept the pattern from the input synapse and wait if no pattern is present. To understand the fireFwdGet method you must
understand how Java thread synchronization works. Java thread synchronization will be covered as we discuss this method. The fireFwdGet method of the Layer class is shown in Listing 5.2.
protected synchronized void fireFwdGet() { while (aInputPatternListener == null) { try {
wait();
} catch (InterruptedException ie) { ie.printStackTrace();
return; }
}
double[] patt;
int currentSize = aInputPatternListener.size(); InputPatternListener tempListener = null;
for (int index = 0; index < currentSize; index++){ tempListener = (InputPatternListener)aInputPatternListener.elementAt(index); if (tempListener != null) { m_pattern = tempListener.fwdGet(); if (m_pattern != null) { patt = m_pattern.getArray(); if (patt.length != inps.length) inps = new double[patt.length]; sumInput(patt); step = m_pattern.getCount(); } }; }; }
The fireFwdGet method is used to consulate the input from several layers. It could be that the current layer that is being processed receives input from several layers. It is the fireFwdGet method that consolidates these inputs. You will notice from the above code that the variable aInputPatternListener hold multiple pattern sources. The code loops through each of these input sources and sums the inputput. This summation is accomplished by calling the sumInput method inside of the loop.
Usually this is not the case, and you will have only layer to input from. If there is only one layer to input from then the fireFwdGet method does little more than return the pattern that is being presented to the current level. When the fireFwdGet method is done receiving the input it returns back to the Layer.run() method.
For the case of the XOR example, the input layer is simply receiving a pattern to recall, such as {0,1}. The input layer will take this value, and then apply the threshold function, and pass it onto the next layer. For the XOR example we used a sigmoid type input layer. JOONE implements the sigmoid through the class SigmoidLayer. When the Layer.run() method calls the forward method, it is actually the SigmoidLayer.forward() method that is being called, since we are using a SigmoidLayer class. Listing 5.3 shows the
SigmoidLayer.forward() method.
Listing 5.3: The SigmoidLayer.forward Method
public void forward(double[] pattern) { int x; double in; int n = getRows(); try { for ( x = 0; x < n; ++x ) { in = pattern[x] + bias.value[x][0]; outs[x] = 1 / (1 + Math.exp(-in)); }
} catch ( Exception aioobe ) { aioobe.printStackTrace(); }
}
As you can see from Listing 5.3 the SigmoidLayer.layer() method applies the sigmoid function to each of the neurons in this layer. The sigmoid function was discussed in Chapter 2.
Now that the Layer.run method has processed its input using the sigmoid threshold function, the layer is ready to pass the pattern to the next layer. As the pattern is passed to the next layer, the appropriate weights must be applied. As you can see from Figure 5.5 the pattern presented to a neuron is made up of an input from each neuron in the previous layer.
In the case of the XOR problem, examined in the previous section, the pattern presented to the input layer consists of simply our two binary numbers that we are inputting into the XOR problem, such as {0,1}. The three hidden layers of the neural network will next receive the pattern. Each of these three neurons will each receive two numbers, one from each input neuron in the previous layer. The hidden layer, therefore will receive a total of six numbers from the previous layer, two input neurons for each of the three hidden neurons. The input pattern will not be simply passed unmodified to the hidden layer, though. First the connection weights, threshold method and bias must all be applied. We will first examine how the connection weight between layers is applied. In the case of a single hidden layer neural network there will be two groups of connection weights. The first is from the input layer to the hidden layer. The second is from the hidden layer to the output layer.
To process the connection weights JOONE uses synapses. Now that the Layer.run method has applied the threshold function to each of the neuron values the layer must pass the pattern onto the synapse. It will be the synapse that applies the connection weights and actually sends the pattern to the next layer. Do not forget that in JOONE each layer is running as a separate thread. So passing the pattern from one layer to the next does not involve a simple function call. As I show you how the pattern is passed from the layer to the next layer you will see how the thread synchronization works. The Layer method has a method called fireFwdPut() that helps with this process. You will notice from Listing 5.1 that the Layer.fireFwdPut() method is called once the input has been obtained, and processed by the Layer.forward methods. The Layer.fireFwdPut method is shown in Listing 5.2.
Listing 5.4: The Layer.fireFwdPut Method
protected void fireFwdPut(Pattern pattern) { if ( aOutputPatternListener == null ) { return;
};
int currentSize = aOutputPatternListener.size(); OutputPatternListener tempListener = null;
for ( int index = 0; index < currentSize; index++ ){ tempListener = (OutputPatternListener)aOutputPatternListener.elementAt(index); if ( tempListener != null ) { tempListener.fwdPut((Pattern)pattern.clone()); }; }; }
The Layer.fireFwdPut method has two responsibilities. First it must apply the connection weights between each the neurons of the current level, and those of the next level. Second, it must transmit this pattern to the synapse.
There may be zero or more synapses that will receive input from the current layer. Most likely there will be only one. The for loop seen in Listing 5.4 loops through the collection of synapses and calls the fireFwdPut() method for each. This method will carry the pattern to the synapse by calling the Synapse.fireFwdPut method. The Synapse.fireFwdPut method is shown in Listing 5.3.
Listing 5.3: The Synapse.fireFwdPut Method
public synchronized void fwdPut(Pattern pattern) { if ( isEnabled() ) { count = pattern.getCount(); while ( items > 0 ) { try { wait(); } catch ( InterruptedException e ) { //e.printStackTrace(); return; } } m_pattern = pattern; inps = (double[])pattern.getArray(); forward(inps); ++items; notifyAll(); } }
One of the first things that you should notice about the fireFwdPut method is that it is designed to support thread synchronization. You can see this by the fact that the keyword synchronized in the method signature. The fireFwdPut method will call the wait method if there is a pattern already being processed at the destination layer. This will cause the fireFwdPut method to block, and therefore not consume processor resources, if the receiving layer is not ready to handle the input yet.
Once the wait loop completes the synapse will process the input and then pass the pattern to the next layer. The only processing that the synapse will perform on the pattern is to apply the bias. As the Synapse.fwdPut method processes it first copies the pattern that ti was passed to the m_pattern class variable.
m_pattern = pattern;
The pattern array is then copied to an array of double values for processing.
inps = (double[])pattern.getArray();
The pattern, as an array of doubles, is then passed to the forward method. In all of the JOONE classes, the forward method is always used to perform whatever processing the current object is to perform. In the case of the synapse, the processing that is to be applied is the application of the bias.
forward(inps);
Once the bias has been applied, the pattern is ready to be processed by the next layer. The next layer is in a separate thread and is likely already waiting to process this pattern. To cause the next layer to accept the pattern the next layer's thread must be notified. This is done by calling the Java notifyAll method that will release all threads that are waiting on this method.
We will now examine the FullSynapse.forward method that applies the bias. You will notice that this method is inside of the FullSynapse class rather than the Synapse class. This is because the Synapse class is an abstract class that does not actually implement the forward method. This is allows multiple synapse types to be created in the JOONE library that can handle the bias in different ways. The FullSynapse.forward method is shown in Listing 5.4.
Listing 5.4: The FullSynapse.forward Method
public void forward(double[] pattern) { int x; double in; int n = getRows(); try { for ( x = 0; x < n; ++x ) { in = pattern[x] + bias.value[x][0]; outs[x] = 1 / (1 + Math.exp(-in)); }
} catch ( Exception aioobe ) { aioobe.printStackTrace(); }
}
As you can see the bias is applied to each element of the pattern. The bias is not simply multiplied against the pattern like a connection weight. Rather the pattern is summed with the bias. The sign of this value is inverted and the reciprocal is taken. This value will be returned to the calling method to be passed onto the next level.
You have observed the process that JOONE goes thorough as an input pattern is passed from one layer to the next layer. This process will be repeated for each layer of the neural network. In the case of the single hidden layer neural network being used for the XOR problem, this process will be repeated as the pattern then goes from the hidden layer to the output layer.
Chapter 5: Understanding Back Propagation
Article Title: Chapter 5: Understanding Back Propagation Category: Artificial Intelligence Most Popular
From Series: Programming Neural Networks in Java
Posted: Wednesday, November 16, 2005 05:15 PM Author: JeffHeaton
Page: 5/6