Annotation examples--Fribourg--2019-09-03

Annotation examples. This is an overview of some of the software I have used for annotation (and a few extra features some of this software has.) This was presented in the SwissUniversities Doctoral Programme, Language & Cognition, in the Module: Linguistic and corpus perspectives on argumentative discourse.

Screenshots are given of GATE, UAM Corpus Tool, Excel, BRAT, EPPI Reviewer, and a custom tool. In most cases there are references to one of my papers for further details.

I briefly describe a typical annotation process:
Find text of interest
Find phenomena of interest
Draft an annotation manual
Iteratively test annotation & revise manual
Find questionable annotations, check disagreements.
Revise the manual.
Iterate.
Annotate

1. Annotation Jodi Schneider Linguistic and corpus perspectives on argumentative discourse, SwissUniversities Doctoral Programme Language & Cognition University of Fribourg, Fribourg, Switzerland 2019-09-02

2. A typical annotation process Find text of interest Find phenomena of interest Draft an annotation manual Iteratively test annotation & revise manual Find questionable annotations, check disagreements. Revise the manual. Iterate. Annotate

3. Examples of annotation software GATE: https://gate.ac.uk Free & Open source NLP pipeline integration, robust developmnet community, ingests lots of formats, UAM CorpusTool : http://www.corpustool.com Free Comparative statistics, corpus search, annotation schemes easy to set up Excel Great for simple annotation BRAT : http://brat.nlplab.org Free & Open source Run your own instance, browser-based for collaboration EPPI Reviewer: https://eppi.ioe.ac.uk/cms/Default.aspx?alias=eppi.ioe.ac.uk/cms/er4 Data extraction for systematic review Custom tools

4. GATE Jodi Schneider, Alexandre Passant, and Stefan Decker “ Deletion Discussions in Wikipedia: Decision Factors and Outcomes. ” In  WikiSym2012 . Linz, Austria, August 27-29, 2012.

5. UAM CorpusTool (V 2.8.16) Jodi Schneider, Krystian Samp , Alexandre Passant, Stefan Decker. “ Arguments about Deletion: How Experience Improves the Acceptability of Arguments in Ad-hoc Online Task Groups ”. In CSCW 2013, San Antonio, TX, February 23-27, 2013.

6. Excel Dong, Xiaoru ; Xie , Jingyi ; Hoang, Linh (2019): Inclusion_Criteria_Annotation . University of Illinois at Urbana-Champaign. https://doi.org/10.13012/B2IDB-5958960_V2 for Text Mining Pipeline to Accelerate Systematic Reviews in Evidence-Based Medicine

7. Excel Hoang, Linh; Schneider, Jodi (2018): Citation context analysis of RobotReviewer core papers circa 2018-06. University of Illinois at Urbana-Champaign. https://doi.org/10.13012/B2IDB-1075526_V1 for Text Mining Pipeline to Accelerate Systematic Reviews in Evidence-Based Medicine

8. BRAT Halil Kilicoglu , Zeshan Peng Shabnam Tafreshi , Tung Tran, Graciela Rosemblat , Jodi Schneider. “Confirm or Refute?: A Comparative Study on Citation Sentiment Classification in Clinical Research Publications.” Journal of Biomedical Informatics, Vol 91, 103123. doi : 10.1016/j.jbi.2019.103123

9. EPPI-Reviewer Work in progress, Systematic Review of Empirical Research about Retracted Publications project team

10. Custom Tools Halil Kilicoglu , Graciela Rosemblat , Zeshan Peng, Mario Malicki , Tony Tse , Jodi Schneider, Gerben ter Riet. Annotating Clinical Trial Publications to Assess CONSORT Adherence: A Feasibility Study. 6th World Conference on Research Integrity, Hong Kong, 2019.

11. Some tools have additional features

12. GATE - sentiment ( gazeteer ) Jodi Schneider & Adam Wyner . “ Identifying Consumers' Arguments in Text ” In  SWAIE 2012: Semantic Web and Information Extraction at EKAW 2012.

13. GATE - Domain properties , positive sentiment , premises ( gazeteers ) Jodi Schneider & Adam Wyner . “ Identifying Consumers' Arguments in Text ” In  SWAIE 2012: Semantic Web and Information Extraction at EKAW 2012.

14. GATE – semantic search Jodi Schneider & Adam Wyner . “ Identifying Consumers' Arguments in Text ” In  SWAIE 2012: Semantic Web and Information Extraction at EKAW 2012.

15. UAM CorpusTool (V 2.8.16) Jodi Schneider, Krystian Samp , Alexandre Passant, Stefan Decker. “ Arguments about Deletion: How Experience Improves the Acceptability of Arguments in Ad-hoc Online Task Groups ”. In CSCW 2013, San Antonio, TX, February 23-27, 2013.

16. UAM CorpusTool (V 2.8.16) Jodi Schneider, Krystian Samp , Alexandre Passant, Stefan Decker. “ Arguments about Deletion: How Experience Improves the Acceptability of Arguments in Ad-hoc Online Task Groups ”. In CSCW 2013, San Antonio, TX, February 23-27, 2013.

17. EPPI-Reviewer Work in progress, Systematic Review of Empirical Research about Retracted Publications project team