Download as PDF, PPTX
Uploaded bylljohnston
Leslie Johnston Keynote, Best Practices Exchange 2011
This document discusses how digital collections are now considered data rather than just records or content. It notes that researchers want to analyze entire collections as data sets rather than individual records. Large digital collections like web archives, historic newspapers, and Twitter archives contain billions of records that researchers want to query, analyze, and visualize as data. Institutions are collaborating through groups like the National Digital Stewardship Alliance and developing open source tools like ViewShare to support access to and preservation of these "big data" collections.
More Related Content
Similar to Leslie Johnston Keynote, Best Practices Exchange 2011
Leslie Johnston Keynote, Best Practices Exchange 2011
- 1. From Records toData: It’s Not Just About Collections Any More Leslie Johnston, Library of Congress Best Practices Exchange 2011
- 2. What are theBiggest Insights that we have Learned in Fifteen Years of Building Digital Collections?
- 3. Researchers do notuse digital collections the same way that they use analog collections
- 4. We Can NeverGuess Every Way that Our Collections Will Be Used
- 5. Stewardship organizations have, untilrecently, spoken of “collections” or “content” or “records” or even “files,” but not data.
- 6. We Have Datain our Libraries, Archives and Museums? Yes. Data is not just generated by satellites, identified during experiments, or collected during surveys.
- 7. Datasets are notjust scientific and business tables and spreadsheets: our collections are now considered data. They are the building blocks for interpretation and discovery that transform and combine them into entities that we may not recognize.
- 8. More and moreresearchers want to use collections as a whole, mining and organizing the information in novel ways. Researchers use algorithms to mine the rich information and tools to create pictures that translate that information into knowledge. Researchers may want to interact with a collection of artifacts, or they may want to work with a data corpus.
- 9. Consider the DiggingInto Data Challenge The repositories available for research include not only scientific information—astronomy, geology, physics, biology, social science surveys—but images, film, sound, newspapers, maps, art, archaeology, architecture and government records. http://www.diggingintodata.org/
- 10. What Constitutes “BigData?” The definition of Big Data is very fluid, as it is a moving target — what cannot be easily manipulated with common tools — and specific to the organization: what can be managed and stewarded by any one institution in its infrastructure. One researcher or organization’s concept of a large data set is small to another. Not too long ago, an organization would be surprised to need 10 TB of storage for a large digital collection. Now a collection can increase by 10 TB in a single week.
- 11. We still havecollections. But what we also have is Big Data, which requires us to rethink the infrastructure that is needed to support Big Data services. Our community used to expect researchers to come to us, ask us questions about our collections, and use our digital collections in our environment. Now our collections are, more often than not, self-serve.
- 12. Case Study: WebArchives • Web Archives, such as the one at the Library of Congress, may be comprised of billions of files. • When we began archiving election web sites, we imagined users browsing through the web pages, studying the graphics or use of phrases or links. But when our first researchers came to the Library, they wanted to know about all those topics, but they used scripts to query for them and sort them into categories. They were not very much interested in reading web pages. http://www.loc.gov/webarchiving/
- 13. Case Study: HistoricNewspapers • The Chronicling America collection has over 4 million page images from historic newspapers with OCR from organizations in 25 states. • The site gets approximately 4 million views per day. • Some researchers want to search for stories in historic newspapers. • Some researchers want to mine newspaper OCR for trends across time periods and geographic areas. • Requests have come in to analyze all 4 million page images. http://chroniclingamerica.loc.gov/
- 14. Case Study: Twitter • The Twitter archive has 10s of billions of tweets in it. • Research requests have included users looking for their own Twitter history, the study of the geographic spread of news, the study of the spread of epidemics, and the study of the transmission of new uses of language. social science visualization social media status events personal privacy commercial
- 15. Can each ofour organizations support real- time querying of billions of full-text items? Can we provide tools for collection analysis and visualization? Can we support the frequent downloading by researchers of collections that may be over 200 TB each? These are among the questions that all of our institutions are grappling with as we build large digital collections and discover new ways in which they can be used.
- 16. So what areour institutions doing about preservation and access to our Big Collections and Big Data?
- 17. Collaboration www.digitalpreservation.gov/ndsa The National Digital Stewardship Alliance is an initiative of the National Digital Information Infrastructure and Preservation Program at the Library of Congress, with almost 100 member organizations that share a sense of dedication to digital preservation, and want to work collaboratively across the community. The NDSA operates through five working groups: Content; Standards and Practices; Infrastructure; Innovation; and Outreach.
- 18. Tool Development All stewardshiporganizations can and should participate in the development and use of open access tools for use across the community. NDIIPP is revising its Tools and Services Directory to include a broader range of projects, some of which are always looking for other organizations to contribute to! http://www.digitalpreservation.gov/partners/resources/tools
- 19. As an Example… Seeingand Sharing Digital Cultural Heritage Collections Differently with ViewShare/Recollection
- 20. bigish ideas › heterogeneousdata › one big distributed collection › open distributed infrastructure › mindset: records -> data
- 21.
- 22.
- 23. the ViewShare idea digitalcultural heritage collections include temporal, locative, and categorical data that, could be tapped to better dynamically interact with and understand those collections.
- 24. the challenges › weall have different kinds of metadata › that data is in different kinds of systems › much of that data is messy › much of that data is not in the format we might wish it was
- 25.
- 26.
- 27.
- 28.
- 33. ingest collection descriptions from spreadsheets,MODS records, or ATOM and RSS
- 34. Augment: derive ISO dates, latitudeand longitude coordinates, and break apart data
- 35. design views: graphical interface forassembling views
- 36. publish views onthe site or embed views with one line of javascript into any HTML document.
- 38.
- 39. share data andviews share not only the end results, but also the raw data for other others to create their own views. data use and re-use
- 40. recent work › supportfor public/private views and data › beta support for OAI and ContentDM data loading › full open source release on SourceForge: http://sourceforge.net/projects/loc-recollect/
- 41. what’s next? › viewshare.orgpublic launch on November 1, 2011 › big data sets: in a while › remix across data sets: long view
- 42. contact us › Letus know if you are interested in participation in the NDSA through the web site › Let us know if there is a tool or service that is missing from our directory › visit http://recollection.zepheira.com/ to get a sneak peek at ViewShare › email [email protected] if you are interested in a ViewShare account
- 43. Questions? Leslie Johnston [email protected]