DEEP LEARNING (UNIT 2 ) by surbhi saroha

Deep Learning

1. DEEP LEARNING (UNIT 2) BY: SURBHI SAROHA

2. SYLLABUS DEEP NETWORKS : History of Deep Learning- A Probabilistic Theory of Deep Learning Backpropagation and regularization, batch normalization- VC Dimension and Neural Nets-Deep Vs Shallow Networks-Convolutional Networks- Generative Adversarial Networks (GAN), Semi supervised Learning

3. History of Deep Learning- A Probabilistic Theory of Deep Learning Deep learning, a subset of machine learning, has a rich history that dates back several decades. Here are some key milestones: 1940s-1950s: Early Neural Networks 1943 : Warren McCulloch and Walter Pitts introduced the concept of artificial neurons, the basic units of neural networks. 1950s : Frank Rosenblatt developed the Perceptron, an early neural network model capable of learning from data. 1980s: Backpropagation and Multi-Layer Perceptrons 1986 : Geoffrey Hinton, David Rumelhart , and Ronald Williams popularized the backpropagation algorithm, which allowed multi-layer perceptrons to learn from data more effectively.

4. Cont ….. The backpropagation algorithm enabled the training of neural networks with multiple hidden layers, marking a significant breakthrough . 1990s-2000s: Convolutional and Recurrent Neural Networks 1998 : Yann LeCun developed LeNet , a convolutional neural network (CNN) for digit recognition, demonstrating the potential of CNNs in image processing tasks . 1997 : Sepp Hochreiter and Jürgen Schmidhuber introduced Long Short-Term Memory (LSTM) networks, a type of recurrent neural network (RNN) capable of learning long-term dependencies, crucial for tasks like speech recognition and language modeling . 2010s : Deep Learning Renaissance

5. Cont ….. 2012 : AlexNet , a deep CNN developed by Alex Krizhevsky , Ilya Sutskever , and Geoffrey Hinton, won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC), significantly outperforming traditional methods . This success sparked widespread interest in deep learning and led to the development of many more advanced architectures like VGG, GoogLeNet , ResNet , and more . Late 2010s-Present : Advances and Applications Transformers and attention mechanisms, introduced in 2017 by Vaswani et al., revolutionized natural language processing (NLP) and led to state-of-the-art models like BERT, GPT, and T5.

6. A Probabilistic Theory of Deep Learning A probabilistic theory of deep learning aims to provide a theoretical framework to understand how deep learning models generalize from data. Key concepts include: Bayesian Inference Bayesian methods provide a probabilistic framework for learning from data by combining prior knowledge with observed data to update beliefs. In deep learning, Bayesian inference can be used to estimate model parameters and uncertainty, leading to more robust predictions . Variational Inference Variational inference is a technique for approximating complex probability distributions in a computationally efficient manner.

7. Cont …. In deep learning, variational autoencoders (VAEs) use variational inference to learn latent representations of data, facilitating tasks like generation and anomaly detection . Stochastic Gradient Descent (SGD ) SGD is a probabilistic optimization method widely used in deep learning to minimize the loss function by updating model parameters based on random subsets of data (mini-batches ). The stochastic nature of SGD introduces randomness into the training process, which can help escape local minima and improve generalization . Regularization Techniques Regularization methods, such as dropout and weight decay, introduce probabilistic elements into the training process to prevent overfitting.

8. Cont …. Deep Generative Models Deep generative models, such as GANs (Generative Adversarial Networks) and VAEs, provide probabilistic frameworks for generating new data samples from learned distributions . These models have applications in image synthesis, text generation, and more.

9. Backpropagation and regularization Backpropagation Backpropagation (backward propagation of errors) is the key algorithm used to train artificial neural networks. It involves a two-step process: forward pass and backward pass. Forward Pass Input Data : The input data is fed into the network. Propagation : The input data propagates through the network, layer by layer, to produce an output. Loss Calculation : The output is compared with the actual target output to calculate the loss (error).

10. Backward Pass Compute Gradients : The loss is propagated backward through the network to compute gradients of the loss with respect to each weight using the chain rule of calculus. Update Weights : The weights are updated using gradient descent (or one of its variants) to minimize the loss. This is done by subtracting the gradient multiplied by the learning rate from the current weights. The process is repeated for many iterations (epochs) until the network converges to a state where the loss is minimized.

11. Regularization Regularization techniques are used to prevent overfitting, where the model performs well on the training data but poorly on new, unseen data. Here are some common regularization methods : Dropout : Randomly sets a fraction of the input units to zero at each update during training time. This prevents units from co-adapting too much. Data Augmentation : Artificially enlarges the training dataset by creating modified versions of the existing data, helping the model generalize better . Early Stopping : Monitors the performance of the model on a validation set and stops training when the performance stops improving, thus preventing overfitting.

12. Batch normalization- VC Dimension Batch Normalization improves the training process of neural networks by normalizing layer inputs, leading to more stable and faster training . VC Dimension is a measure of a model's capacity to classify data, helping to understand the balance between model complexity and generalization ability . Benefits of Batch Normalization Stabilizes Training : Reduces internal covariate shift, making the training process more stable. Higher Learning Rates : Allows the use of higher learning rates, speeding up convergence. Regularization Effect : Acts as a regularizer , potentially reducing the need for dropout.

13. Cont ….. VC Dimension VC ( Vapnik-Chervonenkis ) dimension is a measure of the capacity (complexity) of a statistical classification algorithm. It is used to understand the learning capability and overfitting tendency of a model . Definition The VC dimension of a model is defined as the largest number of points that can be shattered by the model. A set of points is said to be shattered by a model if the model can correctly classify all possible labelings of those points. Importance of VC Dimension Model Complexity : A higher VC dimension indicates a more complex model that can fit more intricate patterns. Generalization : Models with a very high VC dimension relative to the number of training samples are prone to overfitting, as they can fit the training data very well but may not generalize to unseen data.

14. Neural Nets-Deep Vs Shallow Networks Neural networks, a key technology in machine learning, come in various forms, with deep and shallow networks being two primary types. The distinction between them lies mainly in their architecture and complexity : Shallow Neural Networks Architecture : Consist of an input layer, one or two hidden layers, and an output layer. Complexity : Less complex due to fewer hidden layers. Training Time : Generally faster to train due to fewer parameters. Computational Resources : Require fewer computational resources. Performance : Suitable for simpler tasks where the relationship between input and output is not highly complex. Examples of Use Cases : Basic classification tasks, simple pattern recognition, linear or mildly nonlinear relationships.

15. Deep Neural Networks Architecture : Consist of an input layer, multiple hidden layers (often more than three), and an output layer . Complexity : More complex due to the increased number of layers . Training Time : Longer training times due to the greater number of parameters . Computational Resources : Require significantly more computational resources, including advanced hardware like GPUs . Performance : Capable of capturing highly complex relationships and patterns in data, making them suitable for more complex tasks . Examples of Use Cases : Image and speech recognition, natural language processing, autonomous driving, and other tasks involving large datasets and complex feature relationships.

16. Shallow Networks Advantages : Faster training times. Simpler to implement and understand. Require less computational power. Disadvantages : Limited ability to model complex data. May under fit complex datasets.

17. Deep Networks Advantages : Can model complex patterns and relationships. More powerful in handling large and complex datasets. Disadvantages : Longer training times and higher computational cost. Risk of overfitting if not properly regularized.

18. Networks-Convolutional Networks- Generative Adversarial Networks (GAN ) Convolutional Neural Networks (CNNs) are specialized deep neural networks primarily used for processing structured grid data such as images. They are particularly effective for tasks involving visual data due to their unique architecture . Key Features of CNNs Convolutional Layers : These layers apply a set of filters to the input data, producing feature maps. Filters help in detecting patterns such as edges, textures, and shapes. Pooling Layers : These layers reduce the spatial dimensions of the feature maps, lowering computational cost and helping in abstraction. Fully Connected Layers : Typically placed at the end, these layers combine the extracted features to make the final prediction. Activation Functions : Commonly used activation functions like ReLU (Rectified Linear Unit) add non-linearity, enabling the network to learn complex patterns. Dropout and Batch Normalization : Techniques to prevent overfitting and ensure faster convergence during training.

19. Applications of CNNs Image Classification : Assigning a label to an image (e.g., recognizing objects in photos). Object Detection : Identifying and localizing objects within an image. Image Segmentation : Classifying each pixel in an image into predefined categories. Face Recognition : Identifying or verifying a person from an image. Medical Image Analysis : Detecting anomalies or diseases from medical scans.

20. Generative Adversarial Networks (GANs) Generative Adversarial Networks (GANs) are a class of machine learning frameworks designed for generative modeling . They consist of two neural networks, the generator and the discriminator, that are trained simultaneously in a competitive manner . Key Components of GANs Generator : This network generates new data instances. It tries to produce data that is indistinguishable from real data. Discriminator : This network evaluates the authenticity of the data, distinguishing between real data from the training set and fake data produced by the generator. Adversarial Process : The generator and discriminator are trained together in a zero-sum game. The generator aims to fool the discriminator, while the discriminator aims to correctly identify real versus fake data.

21. Training GANs Initialization : Both networks are initialized. Adversarial Training : The generator creates fake data. The discriminator evaluates the data. The discriminator is trained on real and fake data. The generator is trained to produce data that the discriminator classifies as real. Iterative Improvement : Through this iterative process, both networks improve, resulting in the generator producing highly realistic data over time.

22. Applications of GANs Image Generation : Creating realistic images from noise (e.g., generating faces, artwork). Data Augmentation : Generating additional training data to improve machine learning models. Super-Resolution : Enhancing the resolution of images. Style Transfer : Applying the artistic style of one image to another. Text-to-Image Synthesis : Generating images from textual descriptions.

23. Semi supervised Learning Semi-supervised learning is a type of machine learning that combines a small amount of labeled data with a large amount of unlabeled data during training . This approach leverages the labeled data to improve learning accuracy and can significantly reduce the cost and effort of data labeling . It lies between supervised learning (using only labeled data) and unsupervised learning (using only unlabeled data ).

24. K ey Concepts in Semi-Supervised Learning Labeled Data : A small dataset where each example is paired with an associated label. Unlabeled Data : A larger dataset where examples do not have associated labels. Label Propagation : The process of spreading label information from labeled to unlabeled data based on similarity. Self-Training : An iterative method where the model is initially trained on labeled data, predicts labels for the unlabeled data, and then retrained on the newly labeled data. Co-Training : Uses multiple classifiers trained on different views of the data to label the unlabeled data. Generative Models : Models that assume a distribution for the data and try to estimate it using both labeled and unlabeled data.

25. Techniques in Semi-Supervised Learning Self-Training : The model is trained on labeled data. It predicts labels for the unlabeled data. High-confidence predictions are added to the training set as pseudo-labeled data. The process is repeated iteratively. Co-Training : Uses two or more different classifiers. Each classifier is trained on different subsets of features. Classifiers label the unlabeled data, and their predictions are added to the training set for the other classifiers.

26. Cont ….. Semi-Supervised SVM : Extends Support Vector Machines (SVM) to use unlabeled data. Finds the decision boundary that maximizes the margin on both labeled and unlabeled data. Graph-Based Methods : Construct a graph where nodes represent data points, and edges represent similarities. Use label propagation to spread label information through the graph. Generative Models : Assume a probabilistic model for the data distribution. Use labeled data to learn parameters for the distribution and then use these parameters to assign labels to the unlabeled data.

27. Applications of Semi-Supervised Learning Natural Language Processing (NLP) : Tasks like text classification, named entity recognition, and sentiment analysis often use semi-supervised learning to leverage large amounts of unlabeled text data. Computer Vision : Object detection, image classification, and segmentation can benefit from semi-supervised learning due to the high cost of labeling images. Speech Recognition : Large volumes of audio data can be used to improve speech recognition models with minimal labeled data. Bioinformatics : Protein function prediction, gene expression analysis, and other tasks where labeled data is scarce but unlabeled data is abundant. Medical Imaging : Diagnosis and treatment planning can use semi-supervised learning to analyze medical images with limited labeled examples.

28. Advantages and Challenges Advantages Reduced Labeling Cost : Significantly reduces the need for labeled data, which can be expensive and time-consuming to obtain. Improved Performance : Can lead to better performance than using only labeled data, especially when the amount of labeled data is small. Utilization of Unlabeled Data : Makes effective use of the large amounts of unlabeled data that are often available.

29. Challenges Complexity : Implementing semi-supervised learning techniques can be more complex than supervised or unsupervised learning. Quality of Pseudo-Labels : Incorrect pseudo-labels can degrade model performance. Model Assumptions : Many semi-supervised learning methods rely on specific assumptions (e.g., the cluster assumption), which may not hold true for all datasets . In summary, semi-supervised learning is a powerful technique that bridges the gap between supervised and unsupervised learning, leveraging the strengths of both to create more accurate and cost-effective models. It is particularly useful in scenarios where labeled data is scarce but unlabeled data is abundant.

30. THANK YOU 