Fault Detection and Classification for Robotic Test-bench

Abstract: Maintenance of industrial systems often cost as much as their initial investment. Implementing predictive maintenance via system health analysis is one of the strategies to reduce maintenance costs. Health status and life estimation of the machinery are the most researched topics in this context. In this paper, we present our analysis for Sixth European Conference of the Prognostics and Health Management Society 2021 Data
Challenge, which introduces a fuse test bench for quality control system, and asks fault detection and classification for the test bench. We proposed classification workflows, which deploy gradient boosting, linear discriminant analysis, and Gaussian process classifiers, and report their performance for different window sizes. Our gradient boosting based solution has been ranked 4th in the data challenge.

https://doi.org/10.36001/phme.2021.v6i1.3040

1. 1/19
Fault Detection and Classification for Robotic
Test-bench: A Data Challenge
Team GTU
Kürşat İnce, Uğur Ceylan,
NazifeNur Erdoğmuş, Engin Sirkeci,
Yakup Genc (Advisor)

2. 2/22
Outline
•Data Challenge and Dataset
•Our approach to Data Challenge
•Results & Discussion

3. 3/22
PHME21 Data Challenge
•System Under Investigation: A robotic fuse test bench.
•Objectives are:
•Identify and classify faults in unlabeled test data (Task #1),
•Rank the features that help us to classify faults correctly (Task #2),
•Predict the correct fault in the earliest time stamp (Task #3),
•Develop unsupervised solutions that identify the experiments’
system parameter configurations (Bonus Task).

4. 4/22
Our Approach –General Framework

5. 5/22
Our Approach –General Framework

6. 6/22
Our Approach –General Framework

7. 7/22
Our Approach –General Framework

8. 8/22
Our Approach –General Framework

9. 9/22
Data Modeling and Optimization
•We used the following Machine Learning methods:
•Gradient Boosting
•Linear Discriminant Analysis
•Gaussian Process
•Genetic Algorithms

10. 10/22
Genetic Algorithms (GA)
•Evolution based metaheuristic search algorithm.
•Commonly used for optimization of the hyper-parameters of
the machine learning models.
•GA requires us to define
•Gene sequence: the genetic representation of the hyper-
parameters,
•Fitness function: evaluate the gene's adaptation to the
"environment."

11. 11/22
Genetic Algorithms –continued
•The (simplified) GA process:
•Starts with a randomly generated initial population of individuals.
•For each generation/iteration
•Evaluate fitness of the individuals.
•Select the most fit individuals.
•Create new individuals randomly for mating, and for mutation.
•Continue until enough number of generations are produced
•We have used DEAP package for GA optimizations.
https://github.com/DEAP/deap

12. 12/22
Model Training
•3-fold cross validation
•Single step classification pipeline (GB & LDA)
•Scaler (RobustScaler)
•Dimension reducer (LDA)
•Classifier (XGB, LGBM, LDA)
•Two step classification (GP)
•Step 1: For each fault class train a GP classifier pipeline
•Scaler (RobustScaler)
•Dimension reducer (LDA)
•Intermediate Classifier (GP)
•Transform the training dataset w/ model.predict_proba()
•Step 2: Final Classifier (LDA)

13. 13/22
Model Evaluations

14. 14/22
Sample Confusion Matrices

15. 15/22
Evaluation Results
•Classification results for each model is given in the following
slides.

16. 16/22
Evaluation Results – continued
Classifier
ws=5 ws=10 ws=15 ws=20 ws=30 ws=40 ws=50
F1MCCF1MCCF1MCCF1MCCF1MCCF1MCCF1MCC
LGBM 0.800.670.870.790.900.840.900.860.900.860.910.870.900.85
XGB 0.800.690.890.820.900.860.910.870.910.880.910.890.920.88
LDA 0.830.700.880.810.900.830.900.830.090.840.920.860.910.85
GP+LDA0.890.820.900.840.870.780.870.770.490.430.530.430.540.41

17. 17/22
Training Results (GA Optimized)
Classifier Parameters F1 MCC
LGBM ws=49, learningrate=0.01,
boostingtype=’gbdt’, noofleaves=10,
maxdepth=-1, noofestimators=300,
minsplitgain=0, subsample=0.5,
subsamplefreq=0, colsample=0.8,
objective=’multiclass’
0.91 0.88
XGB ws=20, learningrate=0.001,
booster=’dart’, maxdepth=40,
noofestimators=750,
minchildweight=0.5, gamma=0
maxleaves=10, subsample=0.8,
colsample=0.8,
objective=’multi:softmax’
0.93 0.90

18. 18/22
Task #2: Identification and ranking of the sensors
•LOFO based approach on features.
•Basic process:
•Calculate base score using all features.
•For each feature in the dataset
•Remove the feature from the dataset
•Calculate a new score with remaining features
•Calculate feature’s contribution to the model and add to list
•Sort the list by score contributions

19. 19/22
Task #3: Shortest Time Prediction
•Most occurring prediction label → The final classification label
•Shortest time prediction should maintain this feature.
•We search for a cut time that satisfies the above
requirements,
•i.e. the two most occurring labels for the full experiment data are
also the two most occurring labels for the trimmed experiment data.

20. 20/22
Bonus Task: Identifying System Configurations
•Most important features: Humidify and Temperature
•Using Humidity and Temperature features in Class 0
experiments, build a clustering model using K-Means
algorithm with k=2.

21. 21/22
Conclusion
•Challenge: Fault detection and classification of a fuse test bench for quality-
control system.
•Built a data pipeline with gradient boosting (XGB, and LGBM), linear
discriminant analysis (LDA) and Gaussian process (GP) classification
algorithms.
•For small window sizes, two-phase Gaussian process classifier predicted better than
other algorithms.
•Performance of XGB, LGBM, and LDA classifications were better with the increased
window sizes.
•We also performed GA for hyper-parameter optimization.
•Our XGB based model scored 0.3902 using the challenge's evaluation score.
•Team GTU ranked 4
th
in the data challenge.
https://github.com/zakkum42/phme21-public

22. 22/19
Thank You