Ensemble methods

A brief presentation given on the basics of Ensemble Methods. Given as a 'Lightning Talk' during the 7th Cohort of General Assembly's Data Science Immersive Course

1. Ensemble Methods Chris Marker

2. Evaluation and recommendation Treatment

3. Evaluation and recommendation Evaluation and recommendation Evaluation and recommendation Evaluation and recommendation Treatment

4. Ensemble Methods In contrast to traditional modeling methods which train a single model on a set of data, ensemble methods seek to train multiple models and aggregate the results to achieve superior results In most cases, a single base learning algorithm is used in ensemble methods which is called a homogenous ensemble In some cases it is useful to combine multiple algorithms in a heterogeneous ensemble

5. The Statistical Issue It is often the case that the hypothesis space is too large to explore for limited training data, and that there may be several different hypotheses giving the same accuracy on the training data. By "averaging" predictions from multiple models, we can often cancel out errors and get closer to the true function we are seeking to learn.

6. The Computational Issue It might be impossible to develop one model that globally optimizes our objective function. For instance, classification and regression trees reach locally-optimal solutions and that all generalized linear models iterate toward a solution that isn't guaranteed to be the globally-optimal. Starting "local searches" at different points and aggregating our predictions might produce a better result.

7. The Representational Issue Even with vast data and computer power, it might still be impossible for one model to exactly model our objective function. For example, a linear regression model can never model a relationship where a one-unit change in X effects some different change in y based on the value of X. All models have their shortcomings but by creating multiple model and aggregating their predictions it is often possible to get closer to the objective function.

8. Two Ensemble Paradigms - Parallel The base models are generated in parallel and the results are aggregated Bagging Random Forests

9. Bootstrap Aggregated Decision Trees (Bagging) From the original data of size N, bootstrap K samples each of size N (with replacement!). Build a classification or regression decision tree on each bootstrapped sample. Make predictions by passing a test observation through all K trees and developing one aggregate prediction for that observation. Discrete: In ensemble methods, we will most commonly predict a discrete y by "plurality vote," where the most common class is the predicted value for a given observation. Continuous: In ensemble methods, we will most commonly predict a continuous y by averaging the predicted values into one final prediction.

10. Random Forests Bagging reduces variance in individual decision trees but they are still highly correlated with each other. By "decorrelating" our trees from one another, we can drastically reduce the variance of our model. Random forests differ from bagging decision trees in only one way: they use a modified tree learning algorithm that selects, at each split in the learning process, a random subset of the features . This process is sometimes called the random subspace method .

11. Two Ensemble Paradigms - Sequential The base models are generated sequentially and the results are updated based on the previous results AdaBoosting Gradient Boosting

12. AdaBoosting Instead of using deep/full decision trees like in bagging, boosting uses shallow/high-bias base estimators. Iterative fitting is used to explain error/misclassification unexplained by the previous base models and reduces bias without increasing variance. The core principle of AdaBoost is to fit a sequence of weak learners (i.e., models that are only slightly better than random guessing, such as a single-split tree) on repeatedly modified versions of the data . After each fit, the importance weights on each observation need to be updated. The predictions are then combined through a weighted majority vote (or sum) to produce the final prediction. AdaBoost, like all boosting ensemble methods, focuses the next model's fit on the misclassifications/weaknesses of the prior models.

13. Gradient Boosting Gradient boosting is similar to AdaBoosting but functions by fitting the next model to the residuals of the previous model , for instance: Suppose you start with a model F that gives you a less than satisfactory result. F(x1) = 0.8, while y1 = 0.9 Each successive model is correcting for that distance between the true value and predicted This process can be repeated until you have fit an effective model. In other words, residuals are interpreted as negative gradients.