SOFT COMPUTING Subject Code : 18MIT24C Prepared by Dr. N.Thenmozhi
UNIT-I: Fundamentals of Neural Networks: Basic concepts of Neural Networks –Human Brain – Model of an Artificial Neuron – Neural Network Architectures – Characteristics of Neural Networks – Learning methods - Easy Neural Network Architectures – Some Application domains.
UNIT-II: Back propagation Networks: Architecture of a Back-Propagation Network – Back propagation Learning- Effect of Tuning parameters of the Back Propagation Neural Network – Selection of various parameters in BPN.
UNIT-III: Adaptive Resonance Theory: Introduction: Cluster Structure, Vector Quantization, Classical ART Networks, Simplified ART Architecture. ART1: Architecture of ART1–Special features of ART1 Models-ART1 Algorithms. ART2: Architecture of ART2–
ART2 Algorithms.
UNIT-IV: Fuzzy Set Theory: Fuzzy versus crisp, Crisp sets: Operation on Crisp sets- Properties of Crisp Sets-Partition and Covering. Fuzzy sets: Membership Function – Basic fuzzy set Operations-properties of fuzzy sets. Crisp relations: Cartesian product-Other Crisp Relations-Operations on Relations. Fuzzy relations: Fuzzy Cartesian product- Operations on Fuzzy Relations.
UNIT-V: Fuzzy Systems: Crisp logic: Laws of Propositional Logic-Inference in propositional Logic. Predicate logic: Interpretations of Predicate Logic Formula – Inference in Predicate Logic. Fuzzy logic: Fuzzy Quantifiers – Fuzzy Inference, Fuzzy rule based system – Defuzzification.
TEXT BOOK
1. S.Rajasekaran & G.A.Vijayalakshmi Pai, “Neural Networks, Fuzzy logic, and Genetic Algorithms
Synthesis and Applications, PHI, 2005.
REFERENCE BOOKS
1. James A. Freeman, David M.Skapura, “Neural Networks-Algorithms, Applications, and Programming Techniques”, Pearson Education.
2. Fredric M. Ham, Ivica Kostanic, “Principles of Neuro computing for science of Engineering”, TMCH.
UNIT-I: Fundamentals of Neural Networks: Basic concepts of Neural Networks –Human Brain – Model of an Artificial Neuron – Neural Network Architectures – Characteristics of Neural Networks – Learning methods - Easy Neural Network Architectures – Some Application domains.
1. Fundamentals of Neural Networks What is soft computing?
An approach to computing which parallels the remarkable ability of the human mind to reason and learn in an environment of uncertainty and imprecision.
It is characterized by the use of inexact solutions to computationally hard tasks such as the solution of nonparametric complex problems for which an exact solution can‟t be derived in polynomial of time.
Why soft computing approach?
Mathematical model & analysis can be done for relatively simple systems. More complex systems arising in biology, medicine and management systems remain intractable to conventional mathematical and analytical methods. Soft computing deals with imprecision, uncertainty, partial truth and approximation to achieve tractability, robustness and low solution cost. It extends its application to various disciplines of Engg. and science. Typically human can:
1. Take decisions
2. Inference from previous situations experienced 3. Expertise in an area
4. Adapt to changing environment 5. Learn to do better
6. Social behaviour of collective intelligence
Intelligent control strategies have emerged from the above mentioned characteristics of human/ animals. The first two characteristics have given rise to Fuzzy logic;2nd , 3rd and 4th have led to Neural Networks; 4th , 5th and 6th have been used in evolutionary algorithms.
Characteristics of Neuro-Fuzzy & Soft Computing:
1. Human Expertise
2. Biologically inspired computing models 3. New Optimization Techniques
4. Numerical Computation 5. New Application domains 6. Model-free learning 7. Intensive computation 8. Fault tolerance
9. Goal driven characteristics 10. Real world applications
Intelligent Control Strategies (Components of Soft Computing): The popular soft computing components in designing intelligent control theory are:
1. Fuzzy Logic 2. Neural Networks
3. Evolutionary Algorithms
Fuzzy logic:
Most of the time, people are fascinated about fuzzy logic controller. At some point of time in Japan, the scientists designed fuzzy logic controller even for household appliances like a room heater or a washing machine. Its popularity is such that it has been applied to various engineering products.
Neural networks:
Neural networks are basically inspired by various way of observing the biological organism. Most of the time, it is motivated from human way of learning. It is a learning theory. This is an artificial network that learns from example and because it is distributed in nature, fault tolerant, parallel processing of data and distributed structure.
The basic elements of artificial Neural Network are: input nodes, weights, activation function and output node. Inputs are associated with synaptic weights. They are all summed and passed through an activation function giving output y. In a way, output is summation of the signal multiplied with synaptic weight over many input channels.
Evolutionary algorithms:
These are mostly derivative free optimization algorithms that perform random search in a systematic manner to optimize the solution to a hard problem. In this course Genetic Algorithm being the first such algorithm developed in 1970‟s will be discussed in detail. The other algorithms are swarm based that mimic behaviour of organisms, or any systematic process.
1.1. Basic concepts of Neural Networks
Neurons are not only enormously complex but also vary considerably in the details of their structure and function. We will therefore describe typical properties enjoyed by a majority of neurons and make the usual working assumption of connectionism that these provide for the bulk of their computational ability. Readers interested in finding out more may consult one of the many texts in neurophysiology; Thompson (1993) provides a good introductory text, while more comprehensive accounts are given by Kandel et al. (1991) and Kuffler et al. (1984).
All the methods discussed so far makes a strong assumption about the space around;
that is, when we use whether a neural network or fuzzy logic or/and any method that may have been adopted in intelligent control framework, they all make always very strong assumptions and normally they cannot work in a generalized condition. The question is that can they hypothesize a theory? When I design all these controllers, I always take the data; the engineer takes the data. He always builds these models that are updated. They update their own weights based on the feedback from the plant. But the structure of the controller, the model by which we assume the physical plant, all these are done by the engineer and also the structure of the intelligent controller is also decided by the engineer. We do not have a machine that can hypothesize everything; the model it should select, the controller it should select, looking at simply data. As it encounters a specific kind of data from a plant can it come up with specific controller architecture and can it come up with specific type of system model? That is the question we are asking now.
You will see that in the entire course we will be discussing various tools. They will only be dealing with these two things; behaviour. These tools are actually developed by mimicking the human behavior, but not the human way of working. An intelligent machine is one which learns, thinks and behaves in line with the thought process. That we would like but we are very far from it. At least, at the moment, we are very far from this target of achieving real intelligence.
We perceive the environment in a very unique way, in a coherent manner. This is called unity of perception and intelligence has also something to do with this unity of perception, awareness and certain things are not very clear to us until now. So an intelligent machine is one which learns, thinks & behaves in line with thought process.
Neural networks are analogous to adaptive control concepts that we have in control theory and one of the most important aspects of intelligent control is to learn the control parameters, to learn the system model. Some of the learning methodologies we will be learning here is the error-back propagation algorithm, real-time learning algorithm for recurrent network, Kohonen‟s self organizing feature map & Hopfield network.
Features of Artificial Neural Network (ANN) models:
1. Parallel Distributed information processing 2. High degree of connectivity between basic units 3. Connections are modifiable based on experience 4. Learning is a continuous unsupervised process 5. Learns based on local information
6. Performance degrades with less units
Definition: the ability to learn, memorize and still generalize, prompted research in algorithmic modeling of biological neural systems.
Do you think that computer smarter than human brain?
“While successes have been achieved in modeling biological neural systems, there are still no solutions to the complex problem of modeling intuition, consciousness and emotion – which form integral parts of human intelligence” Alan Turing, 1950
---Human brain has the ability to perform tasks such as pattern recognition, perception and motor control much faster than any computer---
1.2. Human Brain
What is a neuron? A neuron is the basic processing unit in a neural network sitting on our brain. It consists of
1. Nucleus-
2. Axon- Output node 3. Dendrites-Input node 4. Synaptic junction
The dynamics of this synaptic junction is complex. We can see the signal inputs from the action of a neuron and through synaptic junction an output is actuated which is carried over through dendrites to another neuron. Here, these are the neurotransmitters. We learned
from our experience that these synaptic junctions are either reinforced or in the sense they behave in such a way that the output of synaptic junction may excite a neuron or inhibit the neuron. This reinforcement of the synaptic weight is a concept that has been taken to artificial neural model.
The objective is to create artificial machine and this artificial neural networks are motivated by certain features that are observed in human brain, like as we said earlier, parallel distributed information processing.
Biological Neuron
A nerve cell neuron is a special biological cell that processes information. According to an estimation, there are huge number of neurons, approximately 1011 with numerous interconnections, approximately 1015.
Figure 1 : Structure of a neuron Working of a Biological Neuron
As shown in the above diagram, a typical neuron consists of the following four parts with the help of which we can explain its working −
• Soma: Nucleus of neuron (the cell body) - process the input
• Dendrites: long irregularly shaped filaments attached to the soma – input channels
• Axon: another type link attached to the soma – output channels
• Output of the axon: voltage pulse (spike) that lasts for a ms
• Firing of neuron – membrane potential • Axon terminates in a specialized contact called the synaptic junction – the electrochemical contact between neurons
• The size of synapses are believed to be linked with learning
• Larger area: excitatory—smaller area: inhibitory ANN versus BNN
Before taking a look at the differences between Artificial Neural Network ANN and Biological Neural Network BNN, let us take a look at the similarities based on the terminology between these two.
Biological Neural Network Artificial Neural Network
Soma Node
Dendrites Input
Synapse Weights or Interconnections
Axon Output
The following table shows the comparison between ANN and BNN based on some criteria mentioned.
Criteria BNN ANN
Processing Massively parallel, slow but superior than ANN
Massively parallel, fast but inferior than BNN
Size 1011 neurons and 1015 interconnections
102 to
104 nodes mainlydependsonthetypeofapplicatio nandnetworkdesignermainlydependsonthetypeo fapplicationandnetworkdesigner
Learning They can tolerate ambiguity
Very precise, structured and formatted data is required to tolerate ambiguity
Fault tolerance
Performance degrades with even partial damage
It is capable of robust performance, hence has the potential to be fault tolerant
Storage capacity
Stores the information in the synapse
Stores the information in continuous memory locations
1.3. Model of Artificial Neural Network
The following diagram represents the general model of ANN followed by its processing.
Figure 2 : Simple model of an artificial neuron
Our basic computational element(model neuron) is often called a node or unit.it receives input from some other units, or perhaps from a external source. Each input has an associated weight w, which can be modified so as to model synaptic learning. The unit computes some function f of the weighted sum of its inputs.
Its output, in turn, can serve as input to other units. The Bias is
Activation functions : Also called the squashing function as it limits the amplitude of the output of the neuron.
1.4 Artificial Neural Network architectures
• Neural Networks are known to be universal function approximators
• Various architectures are available to approximate any nonlinear function
• Different architectures allow for generation of functions of different complexity and power A network topology is the arrangement of a network along with its nodes and connecting lines. According to the topology, ANN can be classified as the following kinds − Feedforward Network
It is a non-recurrent network having processing units/nodes in layers and all the nodes in a layer are connected with the nodes of the previous layers. The connection has different
weights upon them. There is no feedback loop means the signal can only flow in one direction, from input to output. It may be divided into the following two types −
• Single layer feedforward network − The concept is of feedforward ANN having only one weighted layer. In other words, we can say the input layer is fully connected to the output layer.
• Multilayer feedforward network − The concept is of feedforward ANN having more than one weighted layer. As this network has one or more layers between the input and the output layer, it is called hidden layers.
Feedback Network
As the name suggests, a feedback network has feedback paths, which means the signal can flow in both directions using loops. This makes it a non-linear dynamic system, which changes continuously until it reaches a state of equilibrium. It may be divided into the following types −
• Recurrent networks − They are feedback networks with closed loops. Following are the two types of recurrent networks.
• Fully recurrent network − It is the simplest neural network architecture because all nodes are connected to all other nodes and each node works as both input and output.
• Jordan network − It is a closed loop network in which the output will go to the input again as feedback as shown in the following diagram.
1.5 Characteristics of Artificial Neural Network
Any Artificial Neural Network, irrespective of the style and logic of implementation, has a few basic characteristics. These are mentioned below.
• An Artificial Neural Network consists of large number of “neuron” like processing elements.
• All these processing elements have a large number of weighted connections between them.
• The connections between the elements provide a distributed representation of data.
• A Learning Process is implemented to acquire knowledge.
1.6 Learning Methods
• Artificial neural networks work through the optimized weight values.
• The method by which the optimized weight values are attained is called learning.
• In the learning process -> try to teach the network how to produce the output when the corresponding input is presented
• When learning is complete: the trained neural network, with the updated optimal weights, should be able to produce the output within desired accuracy corresponding to an input pattern.
The Learning methods are classified into
• Supervised learning
• Unsupervised learning
• Reinforced learning Supervised learning
Unsupervised learning
The objective of unsupervised learning is to discover patterns or features in the input data with no help from a teacher, basically performing a clustering of input space.
The system learns about the pattern from the data itself without a priori knowledge.
This is similar to our learning experience in adulthood “For example, often in our working environment we are thrown into a project or situation which we know very little about
However, we try to familiarize with the situation as quickly as possible using our previous experiences, education, willingness and similar other factors”
Hebbian Learning
Hebb’s rule: It helps the neural network or neuron assemblies to remember specific patterns much like the memory. From that stored knowledge, similar sort of incomplete or spatial patterns could be recognized. This is even faster than the delta rule or the backpropagation algorithm because there is no repetitive presentation and training of input–
output pairs.
Reinforced learning
Hebbian Learning
Hebbian Learning Rule, also known as Hebb Learning Rule, was proposed by Donald O Hebb. Hebbian rule works by updating the weights between neurons in the neural network for each training sample. Hebbian Learning Rule Algorithm : Set all weights to zero, wi = 0 for i=1 to n, and bias to zero.
Competitive Learning
Stochastic Learning
Stochastic refers to a variable process where the outcome involves some randomness and has some uncertainty. It is a mathematical term and is closely related to “randomness”
and “probabilistic” and can be contrasted to the idea of “deterministic.”
In this method, weights are adjusted in a probabilistic fashion. An example is evident in simulated annealing – the learning mechanism employed by Boltzmann and Cauchy machines, which are a kind of NN system.
Figure 2 : Classification of learning algorithms
1.7. Early Neural Network Architectures
1.8 Some Application Domain
• Pattern Recognition and Image processing
• Optimization/constraint satisfaction
• Forecasting and risk assessment
• Control Systems
