Artificial Neural Networks for NIU session 2016 17
Download as PPTX, PDF•1 like•682 views
The document provides an overview of artificial neural networks (ANNs), outlining key concepts such as neuron models, types of networks, learning methods, and applications across various fields. It describes how ANNs mimic human brain functions, emphasizing their ability to learn from data through supervised, unsupervised, and reinforcement learning techniques. Additionally, it covers technical aspects such as activation functions, weights, and the architecture of different network models, including feedforward and recurrent networks.
![Supervised Learning
• Child learns from a teacher
• Each input vector requires a corresponding
target vector.
• Training pair=[input vector, target vector]
Neural
Network
W
Error
Signal
Generator
X
(Input)
Y
(Actual output)
(Desired Output)
Error
(D-Y)
signals](https://image.slidesharecdn.com/artificialneuralnetworksforniusession2016-17-170418045744/85/Artificial-Neural-Networks-for-NIU-session-2016-17-79-320.jpg)
Artificial Neural Networks for NIU session 2016 17
- 1. Artificial Neural Networks 14.2.2017 Prof. (Dr.) Neeta Awasthy Director, School of Engineering and Technology NOIDA INTERNATIONAL UNIVERSITY, Greater Noida
- 2. Course Objective To understand, successfully apply and evaluate Neural Network structures and paradigms for problems in Science, Engineering and Business.
- 3. PreRequisites It is expected that, the audience has a flair to understand algorithms and basic knowledge of Mathematics, Logic gates and Programming
- 4. Outline Introduction How the human brain learns Neuron Models Different types of Neural Networks Network Layers and Structure Training a Neural Network Application of ANN
- 5. Introduction: Soft Computing techniques such as Neural networks, genetic algorithms and fuzzy logic are among the most powerful tools available for detecting and describing subtle relationships in massive amounts of seemingly unrelated data. Neural networks can learn and are actually taught instead of being programmed. Teaching mode can be supervised or unsupervised Neural Networks learn in the presence of noise
- 7. Neuron and a sample of pulse train
- 8. How does the brain work • Each neuron receives inputs from other neurons – Use spikes to communicate • The effect of each input line on the neuron is controlled by a synaptic weight – Positive or negative • Synaptic weight adapts so that the whole network learns to perform useful computations – Recognizing objects, understanding languages, making plans, controlling the body • There are 1011 neurons with 104 weights.
- 9. How the Human Brain learns In the human brain, a typical neuron collects signals from others through a host of fine structures called dendrites. The neuron sends out spikes of electrical activity through a long, thin stand known as an axon, which splits into thousands of branches. At the end of each branch, a structure called a synapse converts the activity from the axon into electrical effects that inhibit or excite activity in the connected neurons.
- 10. Modularity and brain • Different bits of the cortex do different things • Local damage to the brain has specific effects • Early brain damage makes function relocate • Cortex gives rapid parallel computation plus flexibility • Conventional computers requires very fast central processors for long sequential computations
- 11. Information flow in nervous system
- 12. Fundamental concept • NN are constructed and implemented to model the human brain. • Performs various tasks such as pattern- matching, classification, optimization function, approximation, vector quantization and data clustering. • These tasks are difficult for traditional computers
- 13. ANN • ANN posess a large number of processing elements called nodes/neurons which operate in parallel. • Neurons are connected with others by connection link. • Each link is associated with weights which contain information about the input signal. • Each neuron has an internal state of its own which is a function of the inputs that neuron receives- Activation level
- 14. Comparison between brain verses computer Brain ANN Speed Few ms. Few nano sec. massive ||el processing Size and complexity 1011 neurons & 1015 interconnections Depends on designer Storage capacity Stores information in its interconnection or in synapse. No Loss of memory Contiguous memory locations loss of memory may happen sometimes. Tolerance Has fault tolerance No fault tolerance Inf gets disrupted when interconnections are disconnected Control mechanism Complicated involves chemicals in biological neuron Simpler in ANN
- 15. Types of Problems ANN can handle Mathematical Modeling (Function Approximation) Classification Clustering Forecasting Vector Quantization Pattern Association Control Optimization
- 16. A Neuron Model When a neuron receives excitatory input that is sufficiently large compared with its inhibitory input, it sends a spike of electrical activity down its axon. Learning occurs by changing the effectiveness of the synapses so that the influence of one neuron on another changes. We conduct these neural networks by first trying to deduce the essential features of neurons and their interconnections. We then typically program a computer to simulate these features.
- 17. A Simple Neuron An artificial neuron is a device with many inputs and one output. The neuron has two modes of operation; the training mode and the using mode.
- 18. Important terminologies of ANNs • Weights • Bias • Threshold • Learning rate • Momentum factor • Vigilance parameter • Notations used in ANN
- 19. Weights • Each neuron is connected to every other neuron by means of directed links • Links are associated with weights • Weights contain information about the input signal and is represented as a matrix • Weight matrix also called connection matrix
- 21. Weights contd… • wij –is the weight from processing element ”i” (source node) to processing element “j” (destination node) X1 1 Xi Yj Xn w1j wij wnj bj
- 22. Activation Functions • Used to calculate the output response of a neuron. • Sum of the weighted input signal is applied with an activation to obtain the response. • Activation functions can be linear or non linear • Already dealt – Identity function – Single/binary step function – Discrete/continuous sigmoidal function.
- 23. Bias • Bias is like another weight. Its included by adding a component x0=1 to the input vector X. • X=(1,X1,X2…Xi,…Xn) • Bias is of two types – Positive bias: increase the net input – Negative bias: decrease the net input
- 24. Why Bias is required? • The relationship between input and output given by the equation of straight line y=mx+c X YInput C(bias) y=mx+C
- 25. Threshold • Set value based upon which the final output of the network may be calculated • Used in activation function • The activation function using threshold can be defined as
- 26. Learning rate • Denoted by α. • Used to control the amount of weight adjustment at each step of training • Learning rate ranging from 0 to 1 determines the rate of learning in each time step
- 27. Other terminologies • Momentum factor: – used for convergence when momentum factor is added to weight updation process. • Vigilance parameter: – Denoted by ρ – Used to control the degree of similarity required for patterns to be assigned to the same cluster
- 28. The McCulloch-Pitts model Neurons work by processing information. They receive and provide information in form of spikes. Inputs Output w2 w1 w3 wn . . . x1 x2 x3 … xn-1 xn y
- 29. McCulloch Pitts for NOT Model
- 30. McCulloch Pits for And and or model
- 32. Features of McCulloch-Pitts model • Allows binary 0,1 states only • Operates under a discrete-time assumption • Weights and the neurons’ thresholds are fixed in the model and no interaction among network neurons • Just a primitive model
- 33. Properties for Mc Culloch and Pitts Model Input is 0 or 1 Weights are -1, 0 or +1 Threshold is an integer Output is 0 or 1 Output is 1 if multiplication of weight and input is more than the threshold else Outputs 0 Represent the NOT gate with the help of this model, using signal flow graph and flow Truth Table L=0 -1 x y x y 0 1 1 0 Input x w= -1 Start L=0
- 34. Mc Culloch and Pitts Model……… OR Gate and AND Gate 35 OR Gatex y z 0 0 0 0 1 1 1 0 1 1 1 1 z y z x y L>=1 Start Stop wx,wy Input x,y
- 35. OR gate v/s XOR gate
- 37. Advantages and Disadvantages of McCulloch Pitt model • Advantages • Simplistic • Substantial computing power • Disadvantages – Weights and thresholds are fixed – Not very flexible
- 38. Quiz • Which of the following tasks are neural networks good at? – Recognizing fragments of words in a pre- processed sound wave. – Recognizing badly written characters. – Storing lists of names and birth dates. – logical reasoning Neural networks are good at finding statistical regularities that allow them to recognize patterns. They are not good at flawlessly applying symbolic rules or storing exact numbers.
- 40. Perceptron Learning rule • Learning signal is the difference between the desired and actual neuron’s response • Learning is supervised
- 41. General symbol of neuron consisting of processing node and synaptic connections
- 42. Neuron Modeling for ANN Is referred to activation function. Domain is set of activation values net. Scalar product of weight and input vector Neuron as a processing node performs the operation of summation of its weighted input.
- 43. Sigmoid neurons • These give a real-valued output that is a smooth and bounded function of their total input. – Typically they use the logistic function – They have nice derivatives which make learning easy 0.5 0 0 1
- 44. Activation function • Bipolar binary and unipolar binary are called as hard limiting activation functions used in discrete neuron model • Unipolar continuous and bipolar continuous are called soft limiting activation functions are called sigmoidal characteristics.
- 45. Activation functions Bipolar continuous Bipolar binary functions
- 47. Common models of neurons Binary perceptrons Continuous perceptrons
- 49. Example
- 50. Quiz • Suppose we have 3D input x=(0.5,-0.5) connected to a neuron with weights w=(2,-1) and bias b=0.5. furthermore the target for x is t=0. in this case we use a binary threshold neuron for the output so that y=1 if xTw+b>=0 and 0 otherwise What will be the weights and bias after 1 iteration of perceptron learning algorithm? w= (1.5,-0.5) b=-1.5 w=(1.5,-0.5) b=-0.5 w=(2.5,-1.5) b=0.5 w=(-1.5,0.5) b=1.5
- 51. Basic models of ANN Basic Models of ANN Interconnections Learning rules Activation function
- 54. Summary of the simple networks Single layer nets have limited representation power (linear separability problem) Error drive seems a good way to train a net Multi-layer nets (or nets with non-linear hidden units) may overcome linear inseparability problem, learning methods for such nets are needed Threshold/step output functions hinders the effort to develop learning methods for multi- layered nets
- 55. Training/ Learning Learning can be of one of the following forms: Supervised Learning Unsupervised Learning Reinforced Learning The patterns given to classifier may be on: Parametric Estimation Non- Parametric Estimation
- 56. Machine Learning in ANNs Supervised Learning − It involves a teacher that is scholar than the ANN itself. For example, the teacher feeds some example data about which the teacher already knows the answers.
- 57. Machine Learning in ANNs Unsupervised Learning − It is required when there is no example data set with known answers. For example, searching for a hidden pattern. In this case, clustering i.e. dividing a set of elements into groups according to some unknown pattern is carried out based on the existing data sets present.
- 58. Machine Learning in ANNs Reinforcement Learning − This strategy built on observation. The ANN makes a decision by observing its environment. If the observation is negative, the network adjusts its weights to be able to make a different required decision the next time.
- 59. Unsupervised Learning: why? Collecting and labeling a large set of sample patterns can be costly. Train with large amounts of unlabeled data, and only then use supervision to label the groupings found. In dynamic systems, the samples can change slowly. To find features that will then be useful for categorization. To provide a form of data dependent smart processing or smart feature extraction. To Perform exploratory data analysis, to find structure of data, to form proper classes for supervised analysis.
- 60. Measure of Dissimilarity: Define a metric or distance function d on the vector space λ as a real-valued function on the Cartesian product λX λ such that: Positive Definiteness: 0 < d(x,y) < ∞ for x,y ελ and d(x,y)=0 if and only if x=y Symmetry: d(x,y) = d(y,x) for x,y ελ Triangular Inequality: d(x,y) = d(x,z) + d(y,z) for x,y,z ελ Invariance or distance function: d(x+z,y+z) = d(x,y)
- 61. Error Computation Minkowski Matrix or Lk norm Manhattan Distance or L1 norm Euclidian Distance or L2 norm Ln norm
- 62. Neural networks have performed successfully where other methods have not, predicting system behavior, recognizing and matching complicated, vague, or incomplete data patterns. Apply ANNs to pattern recognition, interpretation, prediction, diagnosis, planning, monitoring, debugging, repair, instruction, control Biomedical Signal Processing Biometric Identification Pattern Recognition System Reliability Business Target Tracking Neural Network Applications
- 63. Pattern Recognition System Sensing Segmentation Classification (missing features & context) Post-processing (costs/ errors) Feature Extraction Input Output (decision)
- 65. Feed-forward neural networks • These are the commonest type of neural network in practical applications. – The first layer is the input and the last layer is the output. – If there is more than one hidden layer, we call them “deep” neural networks. • They compute a series of transformations that change the similarities between cases. – The activities of the neurons in each layer are a non-linear function of the activities in the layer below. hidden units output units input units
- 66. Feedforward Network • Its output and input vectors are respectively • Weight wij connects the i’th neuron with j’th input. Activation rule of ith neuron is where EXAMPLE
- 67. Multilayer feed forward network Can be used to solve complicated problems
- 68. Feedback network When outputs are directed back as inputs to same or preceding layer nodes it results in the formation of feedback networks
- 69. Lateral feedback If the feedback of the output of the processing elements is directed back as input to the processing elements in the same layer then it is called lateral feedback
- 70. Recurrent networks • These have directed cycles in their connection graph. – That means you can sometimes get back to where you started by following the arrows. • They can have complicated dynamics and this can make them very difficult to train. – There is a lot of interest at present in finding efficient ways of training recurrent nets. • They are more biologically realistic. Recurrent nets with multiple hidden layers are just a special case that has some of the hiddenhidden connections missing.
- 71. Recurrent neural networks for modeling sequences • Recurrent neural networks are a very natural way to model sequential data: – They are equivalent to very deep nets with one hidden layer per time slice. – Except that they use the same weights at every time slice and they get input at every time slice. • They have the ability to remember information in their hidden state for a long time. – But its very hard to train them to use this potential. input input input hidden hidden hidden output output output time
- 72. An example of what recurrent neural nets can now do (to whet your interest!) • Ilya Sutskever (2011) trained a special type of recurrent neural net to predict the next character in a sequence. • After training for a long time on a string of half a billion characters from English Wikipedia, he got it to generate new text. – It generates by predicting the probability distribution for the next character and then sampling a character from this distribution.
- 73. Symmetrically connected networks • These are like recurrent networks, but the connections between units are symmetrical (they have the same weight in both directions). – John Hopfield (and others) realized that symmetric networks are much easier to analyze than recurrent networks. – They are also more restricted in what they can do. because they obey an energy function. • For example, they cannot model cycles. • Symmetrically connected nets without hidden units are called “Hopfield nets”.
- 74. Symmetrically connected networks with hidden units • These are called “Boltzmann machines”. – They are much more powerful models than Hopfield nets. – They are less powerful than recurrent neural networks. – They have a beautifully simple learning algorithm.
- 75. Basic models of ANN Basic Models of ANN Interconnections Learning rules Activation function
- 76. Learning • It’s a process by which a NN adapts itself to a stimulus by making proper parameter adjustments, resulting in the production of desired response • Two kinds of learning – Parameter learning:- connection weights are updated – Structure Learning:- change in network structure
- 77. Training • The process of modifying the weights in the connections between network layers with the objective of achieving the expected output is called training a network. • This is achieved through – Supervised learning – Unsupervised learning – Reinforcement learning
- 78. Classification of learning • Supervised learning:- – Learn to predict an output when given an input vector. • Unsupervised learning – Discover a good internal representation of the input. • Reinforcement learning – Learn to select an action to maximize payoff.
- 79. Supervised Learning • Child learns from a teacher • Each input vector requires a corresponding target vector. • Training pair=[input vector, target vector] Neural Network W Error Signal Generator X (Input) Y (Actual output) (Desired Output) Error (D-Y) signals
- 80. • Each training case consists of an input vector x and a target output t. • Regression: The target output is a real number or a whole vector of real numbers. – The price of a stock in 6 months time. – The temperature at noon tomorrow. • Classification: The target output is a class label. – The simplest case is a choice between 1 and 0. – We can also have multiple alternative labels. Two types of supervised learning
- 81. Unsupervised Learning • How a fish or tadpole learns • All similar input patterns are grouped together as clusters. • If a matching input pattern is not found a new cluster is formed • One major aim is to create an internal representation of the input that is useful for subsequent supervised or reinforcement learning. • It provides a compact, low-dimensional representation of the input.
- 82. Self-organizing • In unsupervised learning there is no feedback • Network must discover patterns, regularities, features for the input data over the output • While doing so the network might change in parameters • This process is called self-organizing
- 83. Reinforcement Learning NN W Error Signal Generator X (Input) Y (Actual output) Error signals R Reinforcement signal
- 84. When Reinforcement learning is used? • If less information is available about the target output values (critic information) • Learning based on this critic information is called reinforcement learning and the feedback sent is called reinforcement signal • Feedback in this case is only evaluative and not instructive
- 85. Basic models of ANN Basic Models of ANN Interconnections Learning rules Activation function
- 86. 1. Identity Function f(x)=x for all x 2. Binary Step function 3. Bipolar Step function 4. Sigmoidal Functions:- Continuous functions 5. Ramp functions:- Activation Function
- 87. Some learning algorithms we will learn are • Supervised: • Adaline, Madaline • Perceptron • Back Propagation • multilayer perceptrons • Radial Basis Function Networks • Unsupervised • Competitive Learning • Kohenen self organizing map • Learning vector quantization • Hebbian learning
- 88. Neural processing • Recall:- processing phase for a NN and its objective is to retrieve the information. The process of computing o for a given x • Basic forms of neural information processing – Auto association – Hetero association – Classification
- 89. Neural processing-Autoassociation • Set of patterns can be stored in the network • If a pattern similar to a member of the stored set is presented, an association with the input of closest stored pattern is made
- 90. Neural Processing- Heteroassociation • Associations between pairs of patterns are stored • Distorted input pattern may cause correct heteroassociation at the output
- 91. Neural processing-Classification • Set of input patterns is divided into a number of classes or categories • In response to an input pattern from the set, the classifier is supposed to recall the information regarding class membership of the input pattern.
- 92. Neural Network Learning rules c – learning constant
- 93. Hebbian Learning Rule • The learning signal is equal to the neuron’s output FEED FORWARD UNSUPERVISED LEARNING
- 94. Features of Hebbian Learning • Feedforward unsupervised learning • “When an axon of a cell A is near enough to exicite a cell B and repeatedly and persistently takes place in firing it, some growth process or change takes place in one or both cells increasing the efficiency” • If oixj is positive the results is increase in weight else vice versa
- 96. Delta Learning Rule • Only valid for continuous activation function • Used in supervised training mode • Learning signal for this rule is called delta • The aim of the delta rule is to minimize the error over all training patterns
- 97. Delta Learning Rule Contd. Learning rule is derived from the condition of least squared error. Calculating the gradient vector with respect to wi Minimization of error requires the weight changes to be in the negative gradient direction
- 98. Widrow-Hoff learning Rule • Also called as least mean square learning rule • Introduced by Widrow(1962), used in supervised learning • Independent of the activation function • Special case of delta learning rule wherein activation function is an identity function ie f(net)=net • Minimizes the squared error between the desired output value di and neti
- 100. Winner-Take-All Learning rule Contd… • Can be explained for a layer of neurons • Example of competitive learning and used for unsupervised network training • Learning is based on the premise that one of the neurons in the layer has a maximum response due to the input x • This neuron is declared the winner with a weight
- 102. Summary of learning rules
- 103. Linear Separability • Separation of the input space into regions is based on whether the network response is positive or negative • Line of separation is called linear- separable line. • Example:- – AND function & OR function are linear separable Example – EXOR function Linearly inseparable. Example
- 104. Hebb Network • Hebb learning rule is the simpliest one • The learning in the brain is performed by the change in the synaptic gap • When an axon of cell A is near enough to excite cell B and repeatedly keep firing it, some growth process takes place in one or both cells • According to Hebb rule, weight vector is found to increase proportionately to the product of the input and learning signal.
- 105. Flow chart of Hebb training algorithm Start Initialize Weights For Each s:t Activate input xi=si 1 1 Activate output y=t Weight update Bias update b(new)=b(old) + y Stop y n