unit 5 Neural Networks and Deep Learning.pdf
- 1. CCS355 NEURAL NETWORKS AND DEEP LEARNING UNIT 5 : RECURRENT NEURAL NETWORKS Recurrent Neural Networks: Introduction – Recursive Neural Networks – Bidirectional RNNs – Deep Recurrent Networks – Applications: Image Generation, Image Compression, Natural Language Processing. Complete Auto encoder, Regularized Auto encoder, Stochastic Encoders and Decoders, Contractive Encoders.
- 2. Recurrent Neural Networks(RNN) • Comparison: • Machine Learning→Feed Forward Networks →CNN →RNN→ LSTM CCS355 NEURAL NETWORKS AND DEEP LEARNING
- 3. Machine Learning Small Dataset • Feature Extraction • Feature selection Huge Dataset: • Its not easy to extract the features • Time complexity is high CCS355 NEURAL NETWORKS AND DEEP LEARNING
- 4. So solution is Feed Forward Network(ANN): • Auto Feature Engineering (No need to consider the feature extraction and selection) • Activation Function-Bias, Weight • Capable of Learning any non- linear function, hence called Universal Function Approximation CCS355 NEURAL NETWORKS AND DEEP LEARNING
- 5. Problems in ANN Problem1 • Image classification • Speech Recognition (not able to do that in ANN) • Huge input Permutation (eg.,248x248) • Complexity is also high → leads to poor performance Solution is CNN(convolutional Neural Network) • Huge inputs → image classification CCS355 NEURAL NETWORKS AND DEEP LEARNING
- 6. Problems in ANN Problem2 • In ANN, Cannot handle the sequential Data. So use Time series Use case • Example: Language Translation, • This problem can be solved by RNN • Multiple ANNs can be used to solve the sequential Data (one input→ one output, another input→ another output, no connection between the first input and last output, because it does not have any memory concept) CCS355 NEURAL NETWORKS AND DEEP LEARNING
- 7. Recurrent Neural Networks -Two parts- Fold and Unfold Representation CCS355 NEURAL NETWORKS AND DEEP LEARNING
- 8. Types of RNN 1. One to One • Vanilla Neural Network • Used to regulates the data CCS355 NEURAL NETWORKS AND DEEP LEARNING
- 9. Types of RNN 2. One to Many • Input: Image • Output: Text Description of Input Image Use case • Image Captioning • Music Generation CCS355 NEURAL NETWORKS AND DEEP LEARNING
- 10. Types of RNN • Many to One • Input: Sequence • Output: Fixed size vector Use case • Sequential Analysis • Example: Mice eats cheese→English CCS355 NEURAL NETWORKS AND DEEP LEARNING
- 11. Types of RNN • Many to Many • Use case: Language Translation (English→ French) • Encoder-Decoder CCS355 NEURAL NETWORKS AND DEEP LEARNING
- 12. Vanishing Gradient Problem • W= w+ change in weight At one point, there is no changes in the new weight (i.e change in weight=0) • That problem is called the vanishing Gradient Problem • Solution is LSTM • Cell rate: Two gates • One gate: Remove all the unnecessary memory • Other gate: Add the important memory CCS355 NEURAL NETWORKS AND DEEP LEARNING
- 13. Bidirectional RNN • BRNN connect two hidden layers of opposite directions to the same output • With this form of Generative Deep Learning, the output layer can get information from past(backwards) and future(forward) states simultaneously • BRNNs are especially useful when the context of the input is needed. For example, in Handwriting Recognition, the performance can be enhanced by knowledge of the letters located before and after the current letter • Bidirectional Recurrent Neural Networks are really just putting two independent RNNs together. CCS355 NEURAL NETWORKS AND DEEP LEARNING
- 14. CCS355 NEURAL NETWORKS AND DEEP LEARNING
- 15. Auto Encoders • An autoencoder neural network is an Unsupervised Machine learning algorithm that applies backpropagation, setting the target values to be equal to the inputs. • Autoencoders are used to reduce the size of our inputs into a smaller representation. • If anyone needs the original data, they can reconstruct it from the compressed data. CCS355 NEURAL NETWORKS AND DEEP LEARNING
- 16. Architecture of Auto Encoder CCS355 NEURAL NETWORKS AND DEEP LEARNING
- 17. Three components: • Encoder: This part of the network compresses the input into a latent space representation. The encoder layer encodes the input image as a compressed representation in a reduced dimension. • Code: This part of the network represents the compressed input which is fed to the decoder. • Decoder: This layer decodes the encoded image back to the original dimension. The decoded image is a lossy reconstruction of the original image and it is reconstructed from the latent space representation CCS355 NEURAL NETWORKS AND DEEP LEARNING
- 18. The layer between the encoder and decoder, ie. the code is also known as Bottleneck. CCS355 NEURAL NETWORKS AND DEEP LEARNING
- 19. Applications of Autoencoders Image Colouring • Autoencoders are used for converting any black and white picture into a colored image. CCS355 NEURAL NETWORKS AND DEEP LEARNING
- 20. Feature variation • It extracts only the required features of an image and generates the output by removing any noise or unnecessary interruption. CCS355 NEURAL NETWORKS AND DEEP LEARNING
- 21. Dimensionality Reduction • The reconstructed image is the same as our input but with reduced dimensions. It helps in providing the similar image with a reduced pixel value. CCS355 NEURAL NETWORKS AND DEEP LEARNING
- 22. Denoising Image • The input seen by the autoencoder is not the raw input but a stochastically corrupted version. A denoising autoencoder is thus trained to reconstruct the original input from the noisy version. CCS355 NEURAL NETWORKS AND DEEP LEARNING
- 23. Watermark Removal • It is also used for removing watermarks from images or to remove any object while filming a video or a movie. CCS355 NEURAL NETWORKS AND DEEP LEARNING
- 24. Different types of Auto Encoder • Complete Auto encoder, • Regularized Auto encoder, • Stochastic Encoders and Decoders, • Contractive Encoders CCS355 NEURAL NETWORKS AND DEEP LEARNING
- 25. Complete Auto encoder • It is an unsupervised Learning Network, used to generate a compressed version of the input data. • An Auto encoder whose code dimension is less than the input dimension is called complete auto encoder. CCS355 NEURAL NETWORKS AND DEEP LEARNING
- 26. • The learning process is described as minimizing a loss function L(x, g(f(x)), where L is a loss function, g(f(x)) –being dissimilar from x, such as the mean squared error • The loss function used to train a complete autoencoder is called Reconstruction loss. CCS355 NEURAL NETWORKS AND DEEP LEARNING
- 27. Regularized Auto encoder • A Regularized Auto encoder is non-linear and over complete, but still learn something useful about the data distribution. • Sparse Autoencoders and Denoising Autoencoders CCS355 NEURAL NETWORKS AND DEEP LEARNING
- 28. Sparse Autoencoders • Sparse autoencoder includes more hidden nodes than inputs, but only a small amount are allowed to be active at once CCS355 NEURAL NETWORKS AND DEEP LEARNING
- 29. Denoising Auto Encoders (DAE) • DAE are standard encoders that reduces the risk of learning the identity function by introducing noise • Unsupervised Learning • Adding Noise to the raw input before providing it to the network, CCS355 NEURAL NETWORKS AND DEEP LEARNING
- 30. Contractive Encoders • The objective of a Contractive Autoencoder is to have a robust learned representation which is less sensitive to small variation in the data. • Robustness representation for the data is done by applying a penalty term to the loss function. • However, this Regularizer corresponds to the Frobenius norm of the Jacobian matrix of the encoder activations with respect to the input. • Frobenius norm of the Jacobian matrix for the hidden layer is calculated with respect to input and it is basically the sum of square of all elements. CCS355 NEURAL NETWORKS AND DEEP LEARNING
- 31. CCS355 NEURAL NETWORKS AND DEEP LEARNING
- 32. Stochastic Encoders & Decoders • Both the encoder and the decoder are not simple functions, instead it involves some noise injection. • Any latent variable model Pmodel (h,x) defines a • stochastic encoder Pencoder(h|x)=Pmodel(h|x) and • stochastic decoder Pdecoder(x|h)=Pmodel(x|h) CCS355 NEURAL NETWORKS AND DEEP LEARNING
- 33. Deep Recurrent Networks • Three blocks of parameters: • From the input to the hidden state • From the previous hidden state to the next hidden state • From the hidden state to the output CCS355 NEURAL NETWORKS AND DEEP LEARNING
- 34. Different ways of making RNN Deep CCS355 NEURAL NETWORKS AND DEEP LEARNING
- 35. Applications • Image Generation • Image compression • Natural Language Processing CCS355 NEURAL NETWORKS AND DEEP LEARNING