Towards understanding the impacts of textual dissimilarity on duplicate bug report detection

Conference Presentation

1. Towards Understanding the Impacts of Textual Dissimilarity on Duplicate Bug Report Detection Sigma Jahan & Mohammad Masudur Rahman Faculty of Computer Science, Dalhousie University, Canada International Conference on Software Analysis, Evolution and Reengineering ( SANER 2023 ), Macao, China RAISE Lab Intelligent Automation in Software EngineeRing

2. 2018 Survey on Bug Report Management 327 practitioners – Google, MSFT, Amazon, Facebook , Apache, Mozilla, GitHub 1

3. Textually Similar and Dissimilar Duplicate Bug reports 2

4. E XISTING W ORKS Information Retrieval (IR) Nguyen et al, ASE 2012 Yang et al, ISSRE 2016 Chaparro et al, Saner 2017 IR + Topic Modeling Nguyen et al, ASE 2012 Alipour et al, MSR 2013 Akilan et al, SMC 2020 Machine Learning/Deep Learning Sun et al, ICSE 2010 Deshmukh et al, ICSME 2017 Cooper et al, ICSE 2017 4 Most studies expect the duplicate bug reports to be textually similar We replicate three existing techniques (IR, IR+Topic-Modeling , and Deep Learning based) First attempt to comprehensively understand the impacts of textual dissimilarity on duplicate bug detection 3

5. RQ_1: Does the performance of existing techniques differ significantly in duplicate bug report detection between textually similar and textually dissimilar duplicate bug reports? 4 RQ_ 2 : How do textually similar and textually dissimilar duplicate bug reports differ in their semantics and structures? RQ_ 3 : Does domain-specific embedding help improve the detection of textually dissimilar duplicate bug reports?

6. Schematic Diagram of the Research Methodology 5 5

7. Research Methodology: Dataset Collection Data cleaning and preprocessing Construction of triplets . Dataset preprocessing for ML-based approach Dataset collection Constructing subsets based on textual similarity Firefox System Eclipse System Mobile System (Android & iOS) Data Preprocessing Eclipse 662 Firefox 1414 Mobile 122 6 19% - 26% of duplicate bug reports are textually dissimilar bug report

8. IR: BM25 Topic Modeling: LDA+GloVe Deep Learning: Siamese CNN 7 Implementation of Existing Methodology IR & Topic Modeling: Recall-Rate@k Performance evaluation Deep Learning: AUC, F1, Precision & Recall

9. Result from RQ1: Significant performance difference and large effect size Medium effect size Significant performance difference and large effect size Medium effect size 8 Figure: Result of BM25 & LDA+GloVe

10. P erformance difference between textually similar and textually dissimilar duplicates is less apparent than other techniques 9 Result from RQ1: Figure: Result of Siamese CNN deep learning approach

11. Descriptive analysis Embedding analysis 10 (a) Eclipse (b) Firefox (c) Mobile (a) Eclipse (b) Firefox (c) Mobile

12. Manual analysis Methodology 100 duplicate bug report pairs from each subject system 50 textually similar & 50 textually dissimilar from each subject system Total 300 duplicate bug report pair Used three components – 1. Expected Behaviour (EB) 2. Observed Behaviour (OB) 3. Steps to Reproduce (S2R) to understand differences between textually similar and textually dissimilar duplicate bug reports Prevalence Ratio of EB, OB, and S2R components using title and description T extual similarity ratio for each component S hared terms, keywords, technologies, and overall literary analogies Primary manual analysis conducted by the first author D ocumented using an Excel sheet ≈25 hours spent on the analysis. 11

13. 12 Figure: Similarity Ratio and Prevalence ratio of Textually Similar and Textually Dissimilar Duplicate Bug Reports 13% & 44.61% of textually dissimilar pairs do not contain EB and S2R Similar pairs of bug reports have higher similarity percentages for both OB and EB components, with 83% and 78%. Dissimilar pairs have lower similarity percentages with 38% and 57% for OB and EB components.

14. D omain-specific embedding with Siamese CNN D omain-specific embeddings improves the performance for textually dissimilar duplicate bug reports and reduces the performance gap. However, these embeddings have negligible or negative impacts on t extually similar duplicate bug reports. 13 Figure: Result of Domain-specific embedding with Siamese CNN approach

15. 20 14

16. T HANK Y OU !!! Q UESTIONS ? More details on the paper: https://arxiv.org/pdf/2212.09976.pdf Contact: [email protected] 15