Reference Rot in Scholarly Communication: A Reliable Quantification and a Proposed Solution
Download as PPTX, PDF1 like899 views
The document discusses 'reference rot' in scholarly communication, which involves link rot and content drift that threaten the integrity of web-based scholarly records. A study is presented quantifying the extent of reference rot, revealing that a significant percentage of referenced URIs have drifted over time. The proposed solution includes creating robust links through public web archives to enhance the stability and longevity of referenced resources.
Reference Rot in Scholarly Communication: A Reliable Quantification and a Proposed Solution
- 1. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC Reference Rot in Scholarly Communication: A Reliable Quantification and a Proposed Solution Martin Klein @mart1nkle1n Research Library Los Alamos National Laboratory Acknowledgements: Herbert Van de Sompel, Shawn Jones, Harihar Shankar (LANL) Richard Tobin, Claire Grover (University of of Edinburgh) Andy Jackson (British Library)
- 2. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 2 Agenda 1. Definition: Reference Rot 2. Study: Quantification of Content Drift 3. Proposed Solution: Robust Links
- 3. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 3 Agenda 1. Definition: Reference Rot 2. Study: Quantification of Content Drift 3. Proposed Solution: Robust Links
- 4. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 4
- 5. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 5 http://dl00.org 2000
- 6. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 6 http://dl00.org 2004
- 7. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 7 http://dl00.org 2005
- 8. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 8 http://dl00.org 2008
- 9. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 9 http://hiberlink.org/ Definition: • Link Rot + Content Drift = Reference Rot Observation: • Links to these resources are subject to Reference Rot • Web at large resources referenced in scholarly articles Problem: • Thread to integrity of the web-based scholarly record • Resources do not have the same sense of fixity like e.g., journal articles • Resources’ custodianship is different, in terms of long- term archiving, integrity, and access
- 10. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 10
- 11. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 11 http://dx.doi.org/10.1371/journal.pone.0115253
- 12. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 12 Agenda 1. Definition: Reference Rot 2. Study: Quantification of Content Drift 3. Proposed Solution: Robust Links
- 13. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 13 http://dx.doi.org/10.1371/journal.pone.0167475
- 14. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 14 Study Dataset • 3.5 million articles from arXiv, Elsevier, PMC • Published between Jan 1997 – Dec 2012 • Converted from PDF to XML • Extraction of URIs to web at large resources (>1 million) • Keep track of articles’ publication date
- 15. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 15 Corpora
- 16. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 16 Novel Approach to Assess Content Drift
- 17. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 17 Step 1: Find Mementos • ~ 1 million URI references • ~ 650k Memento Pre/Post pairs discovered via Memento https://mementoweb.org https://tools.ietf.org/html/rfc7089 t t+1t-1
- 18. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 18 Step 2: Select Representative Mementos
- 19. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 19 Referenced in http://dx.doi.org/10.1016/j.nuclphysa.2009.05.110 published on August 15th 2009 May 8th 2009 August 27th 2009
- 20. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 20 Referenced in http://arxiv.org/abs/astro-ph/9707064 published on July 4th 1977 June 7th 1997 today
- 21. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 21 • Apply content similarity measures • How similar is representative? Step 2: Select Representative Mementos
- 22. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 22 Content Similarity Measures • Compute normalized scores (values between 0...100) for: • Simhash • Jaccard • Sørensen-Dice • Cosine
- 23. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 23 Representative Mementos • Idea • If perfect score in all 4 similarity measures Memento Pre and Post are the same Representative Mementos • Sanity check needed • Via HTTP headers: E-Tag and Last-Modified • If same for Pre and Post Memento HTTP-same • Sanity check passed! • 98.88% of Memento pairs that are HTTP-same have perfect score in all 4 similarity measures
- 24. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 24 • ~ 313k referenced URIs have representative Mementos Step 2: Select Representative Mementos
- 25. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 25 • 241k out of 313k URIs have a live web version Step 3: Dereference Live Web Version of URI
- 26. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 26 Step 4: Representative Memento vs. Live Version • Apply content similarity measures • Bin results into 6 clusters
- 27. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 27
- 28. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 28 Aggregate Similarity Score Good: 23.7% of URIs have *not* drifted! Bad: 3/4 URIs *have* drifted!
- 29. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 29 Content Drift & Link Rot Over Time - arXiv
- 30. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 30 arXiv Elsevier PMC
- 31. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 31 Agenda 1. Definition: Reference Rot 2. Study: Quantification of Content Drift 3. Proposed Solution: Robust Links
- 32. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 32 Why don’t DOIs solve this problem? • Designed to combat link rot of references to scholarly articles • Fully relies on the custodians of DOI-identified resources to maintain links • Strong incentives to invest in link stability, usability of their content is at stake • Custodians of web at large resources do not have such incentives (typically web admins, not academic publishers) • Not overly concerned about longevity of their website, the scholarly record • Reference rot is a problem largely rooted outside scholarly communication community, will have to be solved by it
- 33. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 33 Common Practice – original URI (& last accessed date) Original URI Accessed datetime
- 34. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 34 Robust Links 1. Create a snapshot of referenced resources in a public web archive
- 35. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 35 Common Practice – Archived URI URI of archived snapshot Capture datetime
- 36. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 36 Robust Links 1. Create a snapshot of referenced resources in a publically available web archive 2. Decorate links with: • URI of archived snapshot • datetime of archiving • resource’s original URI Benefits: • Original URI allows finding captures in all web archives • Capture datetime allows finding an appropriate capture in all web archives • Uniform, machine-actionable http://robustlinks.mementoweb.org/
- 37. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 37 Robust Links - Desired Practice URI of archived snapshot Capture datetime Original URI
- 38. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 38 Link Decoration <a href="https://www.cni.org/">CNI</a> http://robustlinks.mementoweb.org/spec
- 39. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 39 Link Decoration <a href="https://www.cni.org/" data-versionurl="http://archive.is/fItfw" data-versiondate="2016-12-11"> CNI</a> http://robustlinks.mementoweb.org/spec
- 40. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 40 Link Decoration in Action http://dx.doi.org/10.1045/november2015-vandesompel
- 41. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 41 Link Decoration in Action http://dx.doi.org/10.1045/november2015-vandesompel
- 42. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC 42 Take-Aways 1. Scholarly articles increasingly contain URI references to web at large resources. 2. Such resources are subject to reference rot (link rot + content drift). 3. Custodians of these resources are typically not overly concerned with archiving of their content and longevity of the scholarly record. 4. Authors, publishers, and other parties can help tackle this problem by making links more robust.
- 43. Reference Rot in Scholarly Communication @mart1nkle1n CNI Fall Meeting, 12/12/2016, Washington, DC Reference Rot in Scholarly Communication: A Reliable Quantification and a Proposed Solution Martin Klein @mart1nkle1n Research Library Los Alamos National Laboratory
Editor's Notes
- #17: Previously, archival status (14-day window) as proxy
- #18: Previously, archival status (14-day window) as proxy
- #20: IceCube Neutrino Observatory at the University of Wisconsin http://icecube.wisc.edu
- #21: Institute for Astronomy at the University of Hawaii http://www.ifa.hawaii.edu/~cowie/k_table.html