Build Deep Learning model to identify santader bank's dissatisfied customers

1. Santander Bank Challenge Duy Tran, Indranil Dey, Sriram RV, Sushir Simkhada, Dane Arnesen

2. Agenda Santander Bank customer satisfaction dataset overview (Sushir) Data preprocessing (Sushir) Algorithms / Tools Random Forest using Python (Dane Arnesen) SVM using Matlab (Indranil Dey) Gradient Tree Boosting / XGBoost using R (Duy Tran) Neural Network using Spark with H2O (Sriram RV) Conclusions & Lessons Learned (Sushir) Q&A

3. Santander Bank Challenge The competition was listed in www.kaggle.com . Santander Bank wants to identify the dissatisfied customers. This will help them to take actions to improve the customers happiness. Which customers are unhappy? Happy = 0, Unhappy = 1 371 features including CustomerID & TargetAttr 76,020 rows in training data, only 3,008 rows where TargetAttr =1

4. Preprocessing Issues: More happy customer than unhappy customer. Variables were provided in Spanish so we don ’ t understand the meaning of these variables. Data processing How to remove highly correlated variables and zero frequency variables Solution Removal of zero variance attributes Removal of highly correlated attributes using correlation matrix

5. Random Forest Python

6. Python RandomForestClassifier Python DS library called Scikit -Learn Classification, Regression, Clustering, Dimensionality Reduction, Visualization, etc. Open Source Recommend Anacanda download: https://www.continuum.io/downloads RandomForestClassifier part of the Ensemble family of classifiers Using random subset of features + bagging techniques Lots of parameters…

7. Model Prediction Probability

8. Number of Random Trees

9. Model Feature Importance Of 371 total features… Only 13 features with measurable impact to the Random Forest classifier

10. AUC Curve & Confusion Matrix Class 1 1 1 1603 ( TP ) 405 (FN) 586 (FP) 1408 (TN) Using 55% probability cutoff: Accuracy: 75% TPR : 80% FPR : 29% Precision: 73% F1: 76%

11. Support Vector Machine Matlab

12. Support Vector Machine 12 A Support Vector Machine (SVM) is a discriminative classifier formally defined by a separating hyperplane. Given labeled training data (supervised learning), the algorithm outputs an optimal hyperplane which categorizes new examples. Advantages: SVMs produces large margin separating hyperplane, and efficient in higher dimension It maximizes the margin between points closest to the boundary SVMs only consider points near the margin (support vectors) – more robust Disadvantages : Due to complexity of the algorithm it requires high amount of memory and takes long time to train the model and predict the test data The model is sensitive to optimal choice of kernel and regularization parameters

13. Support Vector Machine 13 MODEL INFO: Status: Trained Training Time: 04:48:27 Classifier Options  Type: SVM  Kernel function: Linear  kernel scale: 1.0  Kernel scale mode: Auto  Box constraint level: 1.0  Multiclass method: One-vs-One  Standardize data: true Cross Validation: 10 Folds Feature Selection Options   Features Included: 369 Validation Results  Validation accuracy: 96% Model 1 : SVM using Linear Kernel – complete dataset with 369 predictors Class Precision Recall F1 100% 96.04% 97.98% 1 0% 0% -- Class 0 AUC: 58.01% Class 1 AUC: 58.01%

14. Reducing the Number of Predictors 14 By using MATLAB we created a correlation matrix for 369 predictors From the correlation matrix we identified predictors which are highly positively or negatively correlated Highly positively correlated: Correlation greater than 0.75 Highly negatively correlated: Correlation less than -0.75 After removing the highly correlated predictors the total number of predictors gor reduced to 115 from 369 Correlation Matrix with 369 Predictors

15. Balancing the Dataset & Applying PCA 15 After removal of correlated predictors the SVM models became more trained in predicting class 0, which was not a desired outcome To overcome this issue we had to balance the training dataset, i.e. keeping equal number of records of both the classes in the training data Using MATLAB randomly selected 3008 records of class 0 and combined 3008 records of class 1 Also to improve the SVM models further, we used PCA with 50 components Principal component analysis (PCA) is a statistical procedure that uses an orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called principal components # .

16. Support Vector Machine 16 MODEL INFO: Status: Trained Training Time: 00:06:42 Classifier Options  Type: SVM  Kernel function: Linear  kernel scale: 1  Kernel scale mode: Auto  Box constraint level: 1.0  Multiclass method: One-vs-One  Standardize data: true Cross Validation: 10 Folds Feature Selection Options   Features Included: 115 PCA Options  Enable PCA: true  Maximum number of components: 50 Validation Results  Validation accuracy: 72.6% Model 6 : SVM using Linear kernel – PCA (50 components) Class Precision Recall F1 72.47% 72.67% 72.57% 1 72.74% 72.55% 72.64% Class 0 AUC: 77.54% Class 1 AUC: 77.54% PCA explained variances: 61.5%, 28.6%, 10.0%, …….

17. Comparing the SVM Models 17 The model 6 has best prediction accuracy for both the classes Model No. Description Accuracy Class Precision Recall F1 AUC Model 1 SVM Linear Kernel – Complete dataset with 369 predictors 96% 100% 96.04% 97.98% 58.01% 1 0% 0% -- Model 2 SVM Linear Kernel – Complete dataset with 115 predictors 96% 99.99% 96.04% 97.98% 59.68% 1 0% 0% -- Model 3 SVM Gaussian Kernel – Complete dataset with 115 predictors 96% 99.99% 96.04% 97.98% 51.07% 1 0% 0% -- Model 4 SVM Linear Kernel – Balanced dataset with 115 predictors 70.8% 67.75% 72.14% 69.88% 78.64% 1 73.84% 69.6% 71.66% Model 5 SVM Gaussian Kernel – Balanced dataset with 115 predictors 70.2% 84.48% 65.71% 73.92% 77.58% 1 55.92% 78.27% 65.23% Model 6 SVM Linear Kernel (PCA) – Balanced dataset with 115 predictors 72.6% 72.47% 72.67% 72.57% 77.54% 1 72.74% 72.55% 72.64% * All models built with 10 folds cross-validation

18. Learnings from building SVM Model 18 Removing highly correlated predictors simplifies models PCA is also a good way to deal with correlated attributes in a dataset Unbalanced training dataset will impact the model’s prediction, and skew it towards the class with higher number of instances in the dataset There is no single way for increasing the prediction accuracy of a model, we should take multiple approaches to iteratively improve the prediction accuracy of the predictive models

19. Gradient Tree Boosting R

20. Performance Metrics - GBM Class 1 Class 0 Class 1 256 316 Class 0 1104 13569 Accuracy : 0.9069 Precision : 0.44755 TPR : 0.18824 TNR: 0.97724 F1 : 0.51751

21. Training Process - GBM Number of Trees Use all observations? Use all predictors? Maximum depth of each tree Learning rate Balance response classes? Increase true positive rate but also increase false positive rate!

22. Hyperparameter optimization – Grid vs Random http://h2o-release.s3.amazonaws.com/h2o/rel-turchin/3/docs-website/h2o-docs/booklets/GBM_Vignette.pdf Grid search – exhaustive, curse of dimensionality. Random search – found to be more effective: http://jmlr.csail.mit.edu/papers/volume13/bergstra12a/bergstra12a.pdf Easy parallelization

23. Neural Network Spark with H2O

24. What is Deep Learning? Deep Learning learns a hierarchy of non linear transformations. Neurons transform their input in non linear way. Three types of neurons Input, Output and Hidden neurons Input neurons get activated by numbers in your dataset and output neurons is the output you want to see.

25. Why did I choose this model? Prediction speed is fast and also the results are very significant with less misclassification errors compared to any other algorithms. Handles lots of irrelevant features well (separates signal from noise). Automatically learns feature interactions. H 2 O is a Java Virtual Machine that brings database-like interactiveness to Hadoop that is optimized for doing “in memory” processing of distributed, parallel machine learning algorithms on clusters. It can be installed as a standalone or on top of existing Hadoop installation.

26. Performance Metrics – Deep Learning Class 0 Class 1 Class 0 64856 8156 Class 1 1673 1335 Error Rate:0.12925 Accuracy: 0.70785 F1 : 0.31751 0.129295

27. Performance Metrics

28. Training the Deep Learning Model

29. Spark Integration RStudio

30. Drawbacks Needs a large data set. The training time is long. Needs a lot of parameter tuning (feature selection). Features need to be on the same scale.

31. Conclusions & Lessons Learned

32. Conclusions & Lessons Learned Understanding the concept of data mining using Classification Python/R/Scala/Matlab are useful tool for data mining Data processing and removal of highly correlated variables helps to identify the main variables. Random Forest classifier/Confusion matrix /PCA/SVM/Neural Network/ Gradient Tree Boosting Combination of various technique helps to identify the factors related to unsatisfied customers. ROC curve was helpful to detect the accuracy of the model. Gradient Tree Boosting gave us the best model.

33. Q&A