Ai & Machine learning - 31140523010 - BDS302.pptx

Good

1. Decision Tree Algorithm NAME – BOBBY KUMAR SINGH STREAM – BSC DATA SCIENCE (3 rd SEM) SUBJECT – MACHINE LEARNING & ARTIFICIAL INTELIGENCE (BDS 302) ROLL NO – 31140523010

2. What is a Decision Tree? A Decision Tree is a popular supervised machine learning algorithm used for both classification and regression tasks. It is a flowchart-like tree structure where: Nodes represent features (attributes) or decisions. Branches represent the outcome of a decision or test. Leaves represent the final outcome or decision. In essence, it models decisions and their possible consequences as a tree of decisions. Each branch of the tree represents a possible decision, outcome, or path to the final classification or prediction. How Decision Trees Work 1. Structure of a Decision Tree Root Node: The topmost node that represents the entire dataset or the initial feature. Decision Nodes: Nodes that represent tests or decisions based on features. Leaf Nodes: Nodes at the end of branches that provide the final decision or prediction. Branches: Connect nodes and represent the outcome of tests or decisions.

3. 2. Building a Decision Tree Select the Best Feature: Classification Trees: The goal is to choose the feature that best separates the data into different classes. Regression Trees: The goal is to choose the feature that best reduces the variance of the output. Common Metrics for Selection: Gini Impurity: Measures the impurity of a node. Lower Gini impurity indicates a better split. Information Gain: Measures how much information is gained by splitting the data based on a feature. Higher information gain indicates a better split. Variance Reduction: Measures how much the variance in the target variable is reduced by splitting the data.

4. Split the Data: Classification Trees: The data is split based on the chosen feature, aiming to maximize the purity of the resulting subsets. Regression Trees: The data is split to minimize the variance within each subset. Repeat: Apply the same process recursively to each resulting subset (i.e., create new nodes and splits) until one of the stopping criteria is met. Stopping Criteria: Maximum Depth: Limit the depth of the tree. Minimum Samples per Leaf: Require a minimum number of samples in each leaf node. Minimum Information Gain: Stop splitting if the gain in information is below a threshold.

5. 3. Making Predictions Classification Trees: The class label of a new data point is determined by traversing the tree from the root to a leaf node, following the decisions based on feature values. The class label of the leaf node is assigned to the data point. Regression Trees: The prediction for a new data point is the average value of the target variable in the leaf node where the data point falls. 4. Example Imagine a decision tree for classifying whether to play tennis based on weather conditions: Root Node: Weather (Sunny, Overcast, Rainy) Decision Nodes: If weather is Sunny, check Humidity (High, Normal) If weather is Rainy, check Wind (Strong, Weak) Leaf Nodes: "Play" or "Don't Play" based on the outcomes of the decisions.

6. Visual Representation of a Simple Decision Tree: [Weather] / | \ Sunny Overcast Rainy / \ / \ Humidity - Wind - / \ / \ High Normal Strong Weak / \ / \ Don't Play Play Don't Play Play

7. TYPES OF DESCISION TREE Classification Trees: Purpose: Used for categorical target variables where the goal is to assign labels or classes to data points. Example Use Cases: Email spam detection, medical diagnosis, customer segmentation. Metrics for Splitting: Gini Impurity: Measures the likelihood of incorrectly classifying a randomly chosen element from the node. Information Gain: Measures how much knowing the value of a feature improves classification. Regression Trees: Purpose: Used for continuous target variables where the goal is to predict a numerical value. Example Use Cases: Predicting house prices, forecasting sales, estimating risk. Metrics for Splitting: Variance Reduction: Measures how much the variance in the target variable is reduced by the split. Mean Squared Error (MSE): Measures the average of the squared differences between actual and predicted values. Other Variants and Techniques: CART (Classification and Regression Trees): A framework that supports both classification and regression tasks. Uses Gini impurity for classification and variance reduction for regression. ID3 (Iterative Dichotomiseer 3): An earlier algorithm for classification that uses information gain to select the best feature. C4.5: An improvement over ID3, handling both continuous and categorical data and using gain ratio for feature selection. CHAID (Chi-square Automatic Interaction Detector): Uses chi-square tests for selecting features and is suitable for categorical target variables. Example of a Decision Tree For a classification task like predicting whether a customer will buy a product based on age and income: Root Node: Age (Young, Middle-aged, Old) Decision Nodes: If Age is Young, check Income (Low, High) If Age is Middle-aged or Old, check other features like Employment Status Leaf Nodes: "Buy" or "Don't Buy" based on the decisions at the end of each branch.

8. Easy to Understand and Interpret: Visual Representation: Decision trees are intuitive and easy to visualize, making them straightforward to interpret and explain to non-technical stakeholders. Feature Importance: The structure of the tree can highlight the most important features for decision-making. No Need for Feature Scaling: Decision trees do not require normalization or standardization of features, which simplifies preprocessing. Handles Non-Linear Relationships: Decision trees can capture non-linear relationships between features and the target variable without needing transformation or complex models. Versatile: Classification and Regression: Decision trees can be used for both classification (categorical target) and regression (continuous target) tasks. Robust to Outliers: Decision trees are relatively robust to outliers compared to some other algorithms because they make decisions based on splitting criteria rather than absolute values. Feature Selection: Implicitly performs feature selection, as only the most important features are used to split the nodes. Non-parametric: No assumptions about the underlying data distribution, making them suitable for a wide range of problems. Handles Mixed Data Types: Can handle both numerical and categorical data, making them flexible in various contexts. ADVANTAGES OF DECISION TREE

9. LIMITATION OF DESCISION TREE Overfitting: Complex Trees: Decision trees can create overly complex models that fit the training data very well but perform poorly on unseen data (overfitting). Large Trees: Deep trees with many branches can be complex and difficult to interpret. Instability: Small Changes in Data: Decision trees can be sensitive to small changes in the data, leading to different tree structures. This can affect model stability and generalization. Bias Towards Features with More Levels: Features with more possible values can dominate the decision-making process, leading to biased splits. Poor Performance on Imbalanced Datasets: Decision trees may perform poorly if the dataset has imbalanced classes, as they might favor the majority class. Greedy Algorithms: Local Optima: Decision trees use a greedy approach to make splits that optimize for the current node but do not guarantee the globally optimal tree structure. Difficulty in Capturing Complex Patterns: Simple Models: While effective for many problems, decision trees might struggle with capturing very complex patterns or interactions in the data. Lack of Smooth Decision Boundaries: Decision trees create axis-aligned decision boundaries, which can be limiting for some types of data. This can result in less smooth or less accurate predictions.

10. Real-World Applications of Decision Trees Decision trees are versatile and can be applied across various industries and domains. Here are some prominent real-world applications: 1. Healthcare Medical Diagnosis: Decision trees can help in diagnosing diseases by evaluating symptoms and medical history. For example, they can be used to classify whether a patient has a specific condition based on test results and symptoms. Treatment Planning: Helps in determining the best treatment options by analyzing patient data and outcomes from similar cases. 2. Finance Credit Scoring: Financial institutions use decision trees to evaluate the creditworthiness of loan applicants. They assess factors like credit history, income, and loan amount to predict the likelihood of default. Fraud Detection: Decision trees can identify fraudulent transactions by analyzing patterns and anomalies in transaction data.