Incivility in Open Source Projects: A Comprehensive Annotated Dataset of Locked GitHub Issue Threads

In the dynamic landscape of open source software (OSS) development, understanding and addressing incivility within issue discussions is crucial for fostering healthy and productive collaborations. This paper presents a curated dataset of 404 locked GitHub issue discussion threads and 5961 individual comments, collected from 213 OSS projects. We annotated the comments with various categories of incivility using Tone Bearing Discussion Features (TBDFs), and, for each issue thread, we annotated the triggers, targets, and consequences of incivility. We observed that Bitter frustration, Impatience, and Mocking are the most prevalent TBDFs exhibited inour dataset. The most common triggers, targets, and consequences of incivility include Failed use of tool/code or error messages, People, and Discontinued further discussion, respectively. This dataset can serve as a valuable resource for analyzing incivility in OSS and improving automated tools to detect and mitigate such behavior

1. 21st International Conference on Mining Software Repositories
Incivility in Open Source Projects:
A Comprehensive Annotated Dataset of Locked
GitHub Issue Threads
Ramtin Ehsani, Mia Mohammad Imran , Robert Zita, Kostadin Damevski, Preetha Chatterjee
Drexel University
1
Preprint: https://arxiv.org/abs/2402.04183
Virginia Commonwealth
University
Elmhurst University
[email protected]

2. Motivation and Research Objective
•Fostering healthy collaborations, one of the main challenges in OSS
•Understanding and addressing incivility within OSS discussions is crucial
•A lack of a comprehensive approach to address uncivil interactions
•Scarcity of large annotated SE datasets
2
Research Objective: Curating a dataset of locked GitHub issue discussion threads
Annotated dataset of locked GitHub issue threads with heated discussions

3. Dataset Annotation
•404 Locked issue threads, and 5,961 Individual comments
•Locked as "too heated" or demonstrated clear characteristics indicative of heated
discussions
•A total of 19 annotators
•To further improve the annotation quality, we used GPT-4
•Manually checked the instances of disagreements between GPT and annotators
3

4. Annotated Features
4
•Tone Bearing Discussion Features (TBDFs), uncivil features*
⚬Bitter frustration, Impatience, Mocking, Irony, Vulgarity, etc.
•Triggers*
⚬Failed use of code, Technical disagreements, Communication breakdown, etc.
•Targets*
⚬People, Code/Tool, Company/organization, Undirected
•Consequences*
⚬Discontinued further discussion, Escalating further, etc.
*
C. Miller, S. Cohen, D. Klug, B. Vasilescu and C. Kästner, "“Did You Miss My Comment or What?” Understanding Toxicity in Open Source Discussions," 2022
*
Isabella Ferreira, Jinghui Cheng, and Bram Adams, The "Shut the f**k up" Phenomenon: Characterizing Incivility in Open Source Code Review Discussions, 2021
*
Jaydeb Sarker, Asif Kamal Turzo, Ming Dong, and Amiangshu Bosu, Automated Identification of Toxic Code Reviews Using ToxiCR, 2023
*
Our open coding process

5. Dataset Description
5
•1,365 comments annotated with an uncivil feature
⚬Bitter frustration, Impatience, and Mocking the most recurrents (~68%)

6. Summary Research Opportunities
● A curated dataset of 404 locked GitHub issue threads from 213
OSS projects
[ Scan the QR code to access]
● Bitter frustration, Impatience, and Mocking are the most
prevalent TBDFs
● Failed use of tool/code or error messages the most common
trigger
● People are the most common target of incivility
● Discontinued further discussion is the most common
consequence
6
Preprint: https://arxiv.org/abs/2307.15631
[email protected]
Preprint: https://arxiv.org/abs/2402.04183
[email protected]
● Resource for conducting comprehensive analyses of incivility
● Automatic incivility detection tools
○ an SE-specific incivility detection tool
● More than just flags for incivility
○ Offer insights into the specific types
● Exploration of how incivility might impact projects’ health
● Investigate how incivility affects targets from underrepresented
communities