UNIT 2: Artificial Neural Networks (ANN)
- 1. 1 DEEP LEARNING & SOFT COMPUTING UNIT 2: Artificial Neural Networks (ANN) ➢ Feedforward Neural Networks 1. Basic Structure: - FNNs consist of three main types of layers: Input, Hidden, and Output. - Information flows in one direction: from the input layer, through the hidden layers, to the output layer. 2. Neurons and Layers: - Each layer contains neurons (nodes). - Input layer: Neurons correspond to input features. - Hidden layers: Perform computations, and extract features. - Output layer: Provides the network's final prediction. 3. Weights and Connections: - Connections between neurons have associated weights. - Weights determine the strength of connections. - Learning involves adjusting these weights. 4. Activation Functions: - Each neuron uses an activation function. - Common activation functions include ReLU, sigmoid, and tanh. - Activation functions introduce non-linearity, allowing the network to learn complex patterns. 5. Forward Pass: - To make a prediction, input data is passed forward through the network. - Each neuron calculates a weighted sum of its inputs and applies the activation function. - This process continues through the hidden layers to the output layer. 6. Training: - Training involves adjusting weights to minimize prediction errors. - Common algorithms like backpropagation are used. - The network learns by comparing its predictions to actual target values and updating weights accordingly. 7. Example - Binary Classification: - Let's say we want to classify images as "cat" or "not cat." - Input layer: Each neuron represents a pixel's value. - Hidden layers: Extract features like edges, and textures. - Output layer: Single neuron with sigmoid activation (0 for "not cat," 1 for "cat"). - During training, weights are adjusted based on correctly or incorrectly classified images. 8. Applications: - FNNs are used in various fields, including image and speech recognition, sentiment analysis, and financial prediction.
- 2. 2 9. Limitations: - May struggle with complex, high-dimensional data. 10. Summary: - Feedforward Neural Networks are the foundational building blocks of deep learning. - They process data layer by layer, learning to represent and classify information. FNNs are like a stack of filters that transform raw data into meaningful predictions, making them a fundamental tool in machine learning and deep learning. ➢ Deep Neural Networks vs. Shallow Neural Networks Aspect Deep Neural Networks Shallow Neural Networks Network Depth Consist of many hidden layers, often > 3 Typically have few hidden layers, usually ≤ 3 Feature Learning Effective at automatically learning hierarchical features from data May struggle to learn complex, hierarchical features Representation Can capture intricate patterns and relationships in data Limited in capturing intricate patterns Training Difficulty May require more data and computational resources May converge faster with less data and resources Overfitting More prone to overfitting, necessitating regularization Less prone to overfitting, simpler architectures Interpretability Often harder to interpret due to increased complexity Relatively easier to interpret Use Cases Common in complex tasks like image and speech recognition Suitable for simpler tasks and quick prototyping Computational Cost Tends to be computationally expensive Requires less computational power Hyperparameter Tuning More hyperparameters to tune Fewer hyperparameters to tune Example Applications Deep learning models like CNNs and LSTMs Simple perceptrons and logistic regression
- 3. 3 ➢ Generative Adversarial Networks (GAN) 1. Introduction: - GANs are a type of artificial neural network architecture introduced by Ian Goodfellow in 2014. - They consist of two neural networks, a generator and a discriminator, trained competitively. 2. Generator: - The generator network inputs random noise and generates data samples (e.g., images). - It aims to produce data that is indistinguishable from real data. 3. Discriminator: - The discriminator network evaluates whether a given data sample is real or fake. - It provides feedback to the generator on how to improve its generated samples. 4. Training Process: - The generator and discriminator are trained together in an adversarial manner. - The generator tries to produce better fake data to fool the discriminator. - The discriminator tries to become better at distinguishing real from fake data. 5. Loss Function: - GANs use a minimax game framework, where the generator minimizes the loss, and the discriminator maximizes it. - The loss function guides the competition between the two networks. 6. Example - Image Generation: - Suppose we want to generate realistic images of cats. - Generator: Takes random noise as input and generates images. - Discriminator: Evaluates if an image is real (from a dataset) or fake (generated by the generator). - During training, the generator learns to create cat images that become increasingly convincing, while the discriminator becomes better at telling real from fake. 7. Convergence: - GAN training continues until the generator creates nearly indistinguishable data from real data, and the discriminator is effectively guessing. 8. Applications: - GANs are used for image-to-image translation, super-resolution, style transfer, and generating photorealistic images. - They have applications in creating deepfakes, improving medical image resolution, and more. 9. Challenges: - GANs can be challenging to train and may suffer from mode collapse (producing limited types of samples). - Hyperparameter tuning and careful design are often required for stable training. In summary, Generative Adversarial Networks (GANs) are a powerful deep- learning framework for generating realistic data. They consist of a generator and a discriminator trained competitively. GANs have a wide range of applications, from creating art to solving complex image-generation problems.
- 4. 4 ➢ Semi-supervised learning Regularization in Autoencoders 1. Introduction: - Semi-supervised learning combines labeled and unlabeled data for training machine learning models. - Autoencoders are a type of neural network used for feature learning and data reconstruction. 2. Autoencoders Overview: - Autoencoders consist of an encoder and a decoder. - The encoder compresses input data into a lower-dimensional representation (encoding). - The decoder reconstructs the input data from the encoding. 3. Semi-Supervised Learning: - In semi-supervised learning, we have a limited amount of labeled data and a larger amount of unlabeled data. 4. Regularization in Autoencoders: - To incorporate semi-supervised learning, we can use regularization techniques in autoencoders. - Regularization helps prevent overfitting and encourages the model to generalize better. 5. Example - Image Classification: - Suppose we have a dataset of cat images, with some labeled as "cats" and others unlabeled. - We want to use semi-supervised learning to train an autoencoder to classify cat images. 6. Semi-Supervised Regularization Techniques: - Denoising Autoencoder: Add noise to input images and train the autoencoder to reconstruct the original, clean images. This helps the model learn robust features. - Variational Autoencoder (VAE): VAEs model the latent space probabilistically, enabling better handling of uncertainty and regularization. - Consistency Regularization: Encourage the model to produce similar encodings for similar data points, making use of unlabeled data. - Adversarial Training: Introduce an adversarial network to distinguish between real and fake encodings, forcing the encoder to generate meaningful representations. 7. Training Process: - During training, the autoencoder minimizes a combination of reconstruction loss (for labeled data) and regularization loss (for both labeled and unlabeled data). - The regularization loss enforces constraints on the encoding space to ensure better generalization. 8. Benefits: - Semi-supervised regularization in autoencoders allows us to leverage unlabeled data effectively. - It can lead to improved model performance, especially when labeled data is scarce.
- 5. 5 9. Applications: - Besides image classification, semi-supervised autoencoders are used in various domains, such as natural language processing, anomaly detection, and recommendation systems. In summary, semi-supervised learning regularization techniques in autoencoders enable us to leverage both labeled and unlabeled data for better model generalization. By incorporating regularization methods like denoising, VAEs, and consistency constraints, we can enhance the performance of autoencoders in various tasks, such as image classification and feature learning. ➢ Denoising autoencoders, Sparse autoencoders, Contractive autoencoders A. Denoising Autoencoders 1. Introduction: - Denoising Autoencoders are a type of autoencoder designed to learn robust representations of data by reconstructing clean inputs from noisy or corrupted inputs. - They help in feature learning and data denoising simultaneously. 2. Working Principle: - Given a noisy input, the denoising autoencoder aims to produce a clean output. - During training, the network is exposed to noisy data and learns to remove the noise, forcing it to capture meaningful features. 3. Example - Image Denoising: - Suppose you have a dataset of handwritten digits with random noise added. - Denoising autoencoders can learn to remove the noise and reconstruct the clean digit images. 4. Benefits: - They learn robust representations that are less sensitive to input noise. - Useful for data pre-processing and feature extraction. B. Sparse Autoencoders 1. Introduction: - Sparse Autoencoders are designed to learn sparse representations of data. - They encourage most neurons in the hidden layer to be inactive, focusing on the most informative features. 2. Working Principle: - A penalty term is added to the loss function to encourage sparsity in the activation of neurons. - This encourages the network to use a small subset of neurons for each input. 3. Example - Image Feature Extraction: - Imagine you want to extract key features from images of faces. - Sparse autoencoders can learn to activate only a few neurons corresponding to distinctive facial features.
- 6. 6 4. Benefits: - They result in more compact and informative representations. - Useful for dimensionality reduction and feature selection. C. Contractive Autoencoders 1. Introduction: - Contractive Autoencoders are designed to learn representations that are robust to small input perturbations. - They capture the local structure of data and are used for applications where preserving nearby points is essential. 2. Working Principle: - A penalty term is added to the loss function based on the gradient of the encoder concerning the input. - This penalty enforces small changes in the input resulting in small changes in the encoding. 3. Example - Text Data: - Suppose you want to represent sentences in a way that similar sentences have similar encodings. - Contractive autoencoders can learn to capture the semantic similarity of sentences. 4. Benefits: - They are effective in applications where local structure preservation is crucial, like recommendation systems. - They can be used for clustering and similarity-based tasks. In summary, Denoising Autoencoders focus on data denoising, Sparse Autoencoders emphasize sparsity in learned representations, and Contractive Autoencoders aim to capture the local structure of data. These variants of autoencoders have specific use cases and can be applied to various domains for feature learning and data representation.