Evaluating citation functions in CiTO: cognitive issues
2 likes682 views
The document discusses an evaluation of the CITO citation classification tool, focusing on human annotators' behaviors in using its properties. An experiment revealed low inter-rater agreement among annotators, and usability assessments indicated variability in property use. The authors propose improvements to enhance clarity and usability while suggesting future work to refine the Citalo tool for automatic citation classification.
![STLab University of Bologna#eswc2014Ciancarini
6
user
author
“It extends the research
outlined in earlier work [3]”
CiTO annotations and mental
models
CiTO](https://image.slidesharecdn.com/eswc2014-140530053010-phpapp02/85/Evaluating-citation-functions-in-CiTO-cognitive-issues-7-320.jpg)
![STLab University of Bologna#eswc2014Ciancarini
6
Interpretation
of author’s
text
Understanding
of CiTO
user
author
“It extends the research
outlined in earlier work [3]”
CiTO annotations and mental
models
CiTO](https://image.slidesharecdn.com/eswc2014-140530053010-phpapp02/85/Evaluating-citation-functions-in-CiTO-cognitive-issues-8-320.jpg)
![STLab University of Bologna#eswc2014Ciancarini
6
Interpretation
of author’s
text
Understanding
of CiTO
user
mental
model
author
“It extends the research
outlined in earlier work [3]”
CiTO annotations and mental
models
CiTO](https://image.slidesharecdn.com/eswc2014-140530053010-phpapp02/85/Evaluating-citation-functions-in-CiTO-cognitive-issues-9-320.jpg)
![STLab University of Bologna#eswc2014Ciancarini
6
Interpretation
of author’s
text
Understanding
of CiTO
user
mental
model
Annotation
cito:extends
author
“It extends the research
outlined in earlier work [3]”
CiTO annotations and mental
models
CiTO](https://image.slidesharecdn.com/eswc2014-140530053010-phpapp02/85/Evaluating-citation-functions-in-CiTO-cognitive-issues-10-320.jpg)
![STLab University of Bologna#eswc2014Ciancarini
mental
model
mental
model
6
Interpretation
of author’s
text
Understanding
of CiTO
user
mental
model
Annotation
cito:extends
author
“It extends the research
outlined in earlier work [3]”
mental
model
mental
model
CiTO annotations and mental
models
CiTO](https://image.slidesharecdn.com/eswc2014-140530053010-phpapp02/85/Evaluating-citation-functions-in-CiTO-cognitive-issues-11-320.jpg)
![STLab University of Bologna#eswc2014Ciancarini
mental
model
mental
model
6
Interpretation
of author’s
text
Understanding
of CiTO
user
mental
model
Annotation
cito:extends
author
“It extends the research
outlined in earlier work [3]”
mental
model
mental
model
CiTO annotations and mental
models
CiTO!
cito:citesForInformation
cito:givesSupportTo
…
The mental models of
different annotators
hardly ever converge to
a single shared opinion](https://image.slidesharecdn.com/eswc2014-140530053010-phpapp02/85/Evaluating-citation-functions-in-CiTO-cognitive-issues-12-320.jpg)
Evaluating citation functions in CiTO: cognitive issues
- 1. STLab University of Bologna#eswc2014Ciancarini ! Evaluating citation functions in CiTO: cognitive issues ! ! ! ! ! 28 May 2014 - Heraklion, Crete Paolo Ciancarini1,2,Angelo Di Iorio1, Andrea Giovanni Nuzzolese1,2, Silvio Peroni1 and FabioVitali1 1Department of Computer Science and Engineering, University of Bologna, Italy 2STLab, Institute of Cognitive Science and Technology, National Research Council, Rome, Italy
- 2. STLab University of Bologna#eswc2014Ciancarini Outline 2 • Motivations • CiTO • Experiment • Evaluation • Lessons learnt and conclusions
- 3. STLab University of Bologna#eswc2014Ciancarini • Bibliographic citations can be seen as tools for: • linking, disseminating, exploring, evaluating research • The task of characterising the functional role of citations in scientific literature is very difficult for agents and humans • Investigating how an existing reference model for classifying citations, i.e., CiTO, is interpreted and used by human annotators • We want to study human’s behaviour in order to simulate it within CiTalO, a tool that automatically classifies citations with CiTO Motivations and goals 3
- 4. STLab University of Bologna#eswc2014Ciancarini • An OWL ontology for describing factual as well as rhetorical functions of citations in scholarly articles • Defines • a top-level property cites (and its inverse isCitedBy) • 41 sub-properties of cites that allow users to characterise precisely the semantics of a citation act • Has been successfully used in large projects, like CiteULike, data.open.ac.uk and the Open Citation Corpus • Several tools have been developed to annotate citations with CiTO • e.g., Chrome and WordPress plug-ins CiTO 4
- 5. STLab University of Bologna#eswc2014Ciancarini • The richness of properties in CiTO (CiTO-Ps) is • a key feature of CiTO: this aspect has contributed to the adoption of the ontology by the Semantic Publishing Users’ adoption of CiTO 5
- 6. STLab University of Bologna#eswc2014Ciancarini • The richness of properties in CiTO (CiTO-Ps) is • a key feature of CiTO: this aspect has contributed to the adoption of the ontology by the Semantic Publishing • an hindrance: most tools actually employ a sub-set of the CiTO properties, • e.g., 6 CiTO-Ps enabled for user annotation by Pensoft Publishers and 9 in the Chrome plug-in Users’ adoption of CiTO 5
- 7. STLab University of Bologna#eswc2014Ciancarini 6 user author “It extends the research outlined in earlier work [3]” CiTO annotations and mental models CiTO
- 8. STLab University of Bologna#eswc2014Ciancarini 6 Interpretation of author’s text Understanding of CiTO user author “It extends the research outlined in earlier work [3]” CiTO annotations and mental models CiTO
- 9. STLab University of Bologna#eswc2014Ciancarini 6 Interpretation of author’s text Understanding of CiTO user mental model author “It extends the research outlined in earlier work [3]” CiTO annotations and mental models CiTO
- 10. STLab University of Bologna#eswc2014Ciancarini 6 Interpretation of author’s text Understanding of CiTO user mental model Annotation cito:extends author “It extends the research outlined in earlier work [3]” CiTO annotations and mental models CiTO
- 11. STLab University of Bologna#eswc2014Ciancarini mental model mental model 6 Interpretation of author’s text Understanding of CiTO user mental model Annotation cito:extends author “It extends the research outlined in earlier work [3]” mental model mental model CiTO annotations and mental models CiTO
- 12. STLab University of Bologna#eswc2014Ciancarini mental model mental model 6 Interpretation of author’s text Understanding of CiTO user mental model Annotation cito:extends author “It extends the research outlined in earlier work [3]” mental model mental model CiTO annotations and mental models CiTO! cito:citesForInformation cito:givesSupportTo … The mental models of different annotators hardly ever converge to a single shared opinion
- 13. STLab University of Bologna#eswc2014Ciancarini 7 What we did We performed an experiment to investigate how humans use CiTO to annotate citations with a type • 20 • 105 citations chosen among the seventh volume of the proceedings of the Balisage Conference Series subjects
- 14. STLab University of Bologna#eswc2014Ciancarini 7 What we did We performed an experiment to investigate how humans use CiTO to annotate citations with a type • 20 • 105 citations chosen among the seventh volume of the proceedings of the Balisage Conference Series (lucky) subjects
- 15. STLab University of Bologna#eswc2014Ciancarini • The experiment had one independent variable, i.e., the number of CiTO-Ps available to subjects for the annotation • Condition T41: 10 subjects used 41 CiTO-Ps • Condition T10: 10 subjects used a subset of 10 CiTO-Ps • i.e., citesAsDataSource, citesAsPotentialSolution, citesAsRecommendedReading, citesAsRelated, citesForInformation, credits, critiques, includesQuotationFrom, obtainsBackgroundFrom, usesMethodIn • T10 CiTO-Ps were chosen among those that had shown a moderate inter-rater agreement (Fleiss’k>0.33) in a preliminary experiment on the same data sample The experiment 8
- 16. STLab University of Bologna#eswc2014Ciancarini ! Evaluation framework 9 Citation context CITO-Ps Explanations and examples of the usage of the CiTO-Ps Available on line at http://www.cs.unibo.it/~nuzzoles/cito_1/?user=r
- 17. STLab University of Bologna#eswc2014Ciancarini 10 1. Which properties have been used by subjects during the experiment? 2. Which were the most used properties? 3. What was the global inter-rater agreement of the subjects? 4. Did the number of available choices bias the global inter-rater agreement? 5. Which properties showed an acceptable positive agreement among subjects? 6. Could properties be organized according to their similarity in subjects’ annotations? 7. What was the perceived usability of the CiTO-Ps? 8. Which were the features of CiTO-Ps that subjects perceived as most useful or problematic? Target questions
- 18. STLab University of Bologna#eswc2014Ciancarini 11 ! ! • Condition T41 • used 37 different CiTO-Ps over 41(avg: 21.7 CiTO-Ps per subject) • 4 properties not selected by any subject • i.e., parodies, plagiarizes, repliesTo and ridicules ! • Condition T10 • used all the 10 CiTO-Ps Results “Which properties have been used by subjects during the experiment?”
- 19. STLab University of Bologna#eswc2014Ciancarini 12 Results “Which were the most used properties?”
- 20. STLab University of Bologna#eswc2014Ciancarini “What was the global inter-rater agreement of the subjects?” “Did the number of available choices bias the global inter-rater agreement?” “Which properties showed an acceptable positive agreement among subjects?” 13 • Condition T41 • Global Fleiss’k = 0.13 • 5 CiTO-Ps with moderate local positive agreement (k > 0.5) • i.e., citesAsPotentialSolution (0.66), citesAsRecommendedReading (0.6), agreesWith (0.54), citesAsDataSource (0.52), usesMethodIn (0.54) • Condition T10 • Global Fleiss’k = 0.15 • 4 CiTO-Ps with moderate local positive agreement • i.e., citesAsPotentialSolution (0.71), citesAsDataSource (0.63), citesAsRecommendedReading (0.52), includesQuotationFrom (0.69) Data evaluation
- 21. STLab University of Bologna#eswc2014Ciancarini “What was the global inter-rater agreement of the subjects?” “Did the number of available choices bias the global inter-rater agreement?” “Which properties showed an acceptable positive agreement among subjects?” 13 • Condition T41 • Global Fleiss’k = 0.13 • 5 CiTO-Ps with moderate local positive agreement (k > 0.5) • i.e., citesAsPotentialSolution (0.66), citesAsRecommendedReading (0.6), agreesWith (0.54), citesAsDataSource (0.52), usesMethodIn (0.54) • Condition T10 • Global Fleiss’k = 0.15 • 4 CiTO-Ps with moderate local positive agreement • i.e., citesAsPotentialSolution (0.71), citesAsDataSource (0.63), citesAsRecommendedReading (0.52), includesQuotationFrom (0.69) Data evaluation The global agreement is very low
- 22. STLab University of Bologna#eswc2014Ciancarini “What was the global inter-rater agreement of the subjects?” “Did the number of available choices bias the global inter-rater agreement?” “Which properties showed an acceptable positive agreement among subjects?” 13 • Condition T41 • Global Fleiss’k = 0.13 • 5 CiTO-Ps with moderate local positive agreement (k > 0.5) • i.e., citesAsPotentialSolution (0.66), citesAsRecommendedReading (0.6), agreesWith (0.54), citesAsDataSource (0.52), usesMethodIn (0.54) • Condition T10 • Global Fleiss’k = 0.15 • 4 CiTO-Ps with moderate local positive agreement • i.e., citesAsPotentialSolution (0.71), citesAsDataSource (0.63), citesAsRecommendedReading (0.52), includesQuotationFrom (0.69) Data evaluation The global agreement is very low The # of CiTO-Ps does not affect the agreement
- 23. STLab University of Bologna#eswc2014Ciancarini “What was the global inter-rater agreement of the subjects?” “Did the number of available choices bias the global inter-rater agreement?” “Which properties showed an acceptable positive agreement among subjects?” 13 • Condition T41 • Global Fleiss’k = 0.13 • 5 CiTO-Ps with moderate local positive agreement (k > 0.5) • i.e., citesAsPotentialSolution (0.66), citesAsRecommendedReading (0.6), agreesWith (0.54), citesAsDataSource (0.52), usesMethodIn (0.54) • Condition T10 • Global Fleiss’k = 0.15 • 4 CiTO-Ps with moderate local positive agreement • i.e., citesAsPotentialSolution (0.71), citesAsDataSource (0.63), citesAsRecommendedReading (0.52), includesQuotationFrom (0.69) Data evaluation The global agreement is very low The # of CiTO-Ps does not affect the agreement The set of CiTO-PS with moderate local positive agreement is little affected by the # of CiTO-Ps
- 24. STLab University of Bologna#eswc2014Ciancarini • We applied the Chinese Whispers clustering algorithm • Input: 2 graphs built by combining all the pairs of different CiTO-Ps as annotated by subjects for each citation • Gr - it takes into account repetitions in annotations for a each CiTO property • e.g.,“extends”,“extends”, and “updates” on a citation generate (extends,updates) and (extends,updates) • Gn - it does not take into account repetitions • e.g.,“extends”,“extends”, and “updates” generate (extends,updates) 14 Clustering CiTO-Ps “Could properties be organized according to their similarity in subjects’ annotations?”
- 25. STLab University of Bologna#eswc2014Ciancarini 14 Clustering CiTO-Ps “Could properties be organized according to their similarity in subjects’ annotations?” Gr Gn disputes critiques derides refutes confirms credits obtainsSupportFrom
- 26. STLab University of Bologna#eswc2014Ciancarini 14 Clustering CiTO-Ps “Could properties be organized according to their similarity in subjects’ annotations?” Gr Gn disputes critiques derides refutes confirms credits obtainsSupportFrom There exist some sort of relations (e.g., taxonomical, equivalence) among the CiTO-Ps of each cluster
- 27. STLab University of Bologna#eswc2014Ciancarini 15 • We computed the System Usability Scale (SUS) Measuring the usability of CiTO-Ps 0.00! 10.00! 20.00! 30.00! 40.00! 50.00! 60.00! 70.00! 80.00! 90.00! 100.00! SUS mean! Usability mean! Learnability mean! T41! T10! “What was the perceived usability of the CiTO-Ps? ”
- 28. STLab University of Bologna#eswc2014Ciancarini 15 • We computed the System Usability Scale (SUS) Measuring the usability of CiTO-Ps 0.00! 10.00! 20.00! 30.00! 40.00! 50.00! 60.00! 70.00! 80.00! 90.00! 100.00! SUS mean! Usability mean! Learnability mean! T41! T10! “What was the perceived usability of the CiTO-Ps? ” Only the usability score approaches the statistical significance
- 29. STLab University of Bologna#eswc2014Ciancarini 16 Grounded theory analysis • The subjects filled a final text questionnaire aimed at capturing positive and negative aspects of CiTO-Ps • We used the text answers for performing a grounded theory analysis, used in Social Science to extract relevant concepts from unstructured text “Which were the features of CiTO-Ps that subjects perceived as most useful or problematic?”
- 30. STLab University of Bologna#eswc2014Ciancarini 16 Grounded theory analysis “Which were the features of CiTO-Ps that subjects perceived as most useful or problematic?”
- 31. STLab University of Bologna#eswc2014Ciancarini 17 Conclusions • Lessons learnt and suggestions to improve CiTO • Reduce the number of less-used properties • Identify the most-used neutral properties • Investigate motivations for low inter-rater agreement • Define explicit relations between CiTO properties • Add support for customised properties • Extend examples, labels and explanations • Future work • Improve CiTalO, a tool for identifying automatically the nature of citations • e.g., by investigating cognitive architectures in order to simulate humans’ behaviour
- 32. STLab University of Bologna#eswc2014Ciancarini 18 Thank you!