1-15-14 DuraSpace Solutions Webinar: The National Agenda for Digital Stewardship presentation slides

Editor's Notes

• #17: Organizational Roles, Policies and Practices describes the overarching challenges faced by the digital preservation community articulates the problem that it is increasingly difficult to adequately preserve our valuable digital content it’s difficult because of a complex set of interrelated societal, technological, financial and organizational pressures Pressures outside our organizations include: At the broadest context (a global problem) is that there are rapidly accelerating technological changes – as well as changes in how people use technology – and an increasing complexity and increasing volume of data Evolving data management and security policies Complex and evolving rights management landscape Pressures in our organizations include: Increased costs, decreased resources Lack of adequate digital preservation staff – both in sheer numbers of personnel as well as a lack of expertise. Increased scope of responsibilities Lack of prioritization of digital preservation by higher administrators (those controlling the budgets) The agenda describes one potential solution for us as a community to campaign for more resources; and work to raise awareness about the real cost and importance of digital preservation through outreach activities However, even if we increase the resources available for digital prservatoin, we have to make sure that we are spending the money our organizations have for digital preservation (and the money that funders invest in digital preservation) as wisely as possible. In the 2014 National Agenda, we suggest that the way to do this is to dramatically increase cross-organizational cooperation, which you could think of as smart collaborations. Impractical for every institution to develop expertise in every aspect of the digital preservation challenge. It’s smarter to divide up the work Share capabilities – leverage specialized resources and expertise Share responsibilities Collaboration frees us up to put investments where they can make a large impact The agenda lists several areas we should apply this cross-organizational cooperation and this section concentrates on three of them: 1) Provision of preservation services – network of preservation service providers that can provide specialized services so every organization does not have to perform all aspects of digital preservation itself. (examples: DPN, OCLC’s idea for SWAQT sites) 2) Share responsibility for shepherding and promoting the use of relevant standards – so that the digital preservation community as a whole as coverage on the relevant standards bodies without each organization needing to participate in every body. 3) Share digital preservation training and staffing resources (curricula training materials, position descriptions, etc). We can use these materials in a variety of digital preservation education experiences for example * Immersive internships and fellowships for undergrads, graduate students and post-graduates Workforce or professional development educational programs that are affiliated with data management or scientific disciplines
• #18: Organizational Roles, Policies and Practices describes the overarching challenges faced by the digital preservation community articulates the problem that it is increasingly difficult to adequately preserve our valuable digital content it’s difficult because of a complex set of interrelated societal, technological, financial and organizational pressures Pressures outside our organizations include: At the broadest context (a global problem) is that there are rapidly accelerating technological changes – as well as changes in how people use technology – and an increasing complexity and increasing volume of data Evolving data management and security policies Complex and evolving rights management landscape Pressures in our organizations include: Increased costs, decreased resources Lack of adequate digital preservation staff – both in sheer numbers of personnel as well as a lack of expertise. Increased scope of responsibilities Lack of prioritization of digital preservation by higher administrators (those controlling the budgets) The agenda describes one potential solution for us as a community to campaign for more resources; and work to raise awareness about the real cost and importance of digital preservation through outreach activities However, even if we increase the resources available for digital prservatoin, we have to make sure that we are spending the money our organizations have for digital preservation (and the money that funders invest in digital preservation) as wisely as possible. In the 2014 National Agenda, we suggest that the way to do this is to dramatically increase cross-organizational cooperation, which you could think of as smart collaborations. Impractical for every institution to develop expertise in every aspect of the digital preservation challenge. It’s smarter to divide up the work Share capabilities – leverage specialized resources and expertise Share responsibilities Collaboration frees us up to put investments where they can make a large impact The agenda lists several areas we should apply this cross-organizational cooperation and this section concentrates on three of them: 1) Provision of preservation services – network of preservation service providers that can provide specialized services so every organization does not have to perform all aspects of digital preservation itself. (examples: DPN, OCLC’s idea for SWAQT sites) 2) Share responsibility for shepherding and promoting the use of relevant standards – so that the digital preservation community as a whole as coverage on the relevant standards bodies without each organization needing to participate in every body. 3) Share digital preservation training and staffing resources (curricula training materials, position descriptions, etc). We can use these materials in a variety of digital preservation education experiences for example * Immersive internships and fellowships for undergrads, graduate students and post-graduates Workforce or professional development educational programs that are affiliated with data management or scientific disciplines
• #19: Overview: The Content WG, which now has more than 100 members, discussed content types most at risk of loss and decided that the 4 types defined in this document are our priority to address in the 2014 year. Overarching challenges across content types that we discussed include: First, that different content types present a variety of challenges related to preservation including, for example: o size/scale of content (web archives, big data), o selection of content when we can't preserve everything - how do we define what is important to preserve. Secondly, when we select content to preserve, audiences increasingly expect expanded access options - tools for engaging and interacting with the content, usability standards met, involvement in the selection of content to preserve. Four priority content areas selected for focus in 2014 are, in no specific order: Electronic Records, Research Data, Web and Social Media, and Moving Image and Recorded Sound. Each was selected for the 2014 list because of its importance to many of our organizations, and the already observed loss of this type of digital content. Electronic Records that need to be preserved range from personal records such as electronic diaries, to business and organization electronic records, and to records created by all levels of government. Issues related to electronic records are: -Knowledge related to selection, including retention schedules, legal requirements, and researchers' interests is lacking, and knowledge is lacking about preservation strategies for his content. - Current infrastructure is inadequate in many organizations/government agencies to implement preservation of e-records. Collaborative models need further development. Research Data are an acute challenge already recognized by many organizations. Challenges include: - the scale and often heterogeneous and diverse nature of the data and the many different information standards used. - costs, which are perhaps the overriding challenge. There is significant need for the development of sustainable economic models to cover management of research data throughout the lifecycle. - and, a gap in the knowledge base for stewarding the preservation of research data exists. In most organizations, there is no clear understanding of who is responsible for preserving research data. This area, in particular, seems to be an area in which the digital preservation and curation community could help. Moving Image and Recorded Sound How many in this room have a smartphone with a audio and video recorder or editor on it?   -This is an area where the creation of content has shifted dramatically with the advent of easy tools that fit in your pocket, the sheer amount of content being created is overwhelming stewards - flood of digital content poses new challenge: -storage and management of large files -ability to play back in the future - vital for content creators and stewards work together to develop new standards -not only are we talking about born digital, but also content converted from older storage media, which has a short lifespan, but then the new digital files need preservation Web and Social Media More organizations than ever preserving web content, but… -Websites are more complex than ever and tools in use for crawling and access not able to keep up -Full text indexing of web archives remains a challenge, though the international community has been working on this, there are still no easy solutions -Social media companies don't have preservation as a business model, and we are concerned about the ability to preserve with the constant and ever-changing barriers that companies put in place -Privacy issues, particularly when it comes to social media preservation, present challenges as well. Even though there is a significant need for research related to web archives preservation and access, very little funding has been allocated to this end, that we are aware of.    -There is some encouraging news about a recent grant by the Mellon Foundation to Columbia University Libraries to pursue collaborative web archiving initiatives, including the development of software, however the micro-grants that Columbia will pass out will be relatively small (maximum 25K each). -There was also a prior grant by the IMLS, who funded work on automatic classification of web pages from a collaborative End of Term government web archive
• #20: Overview: The Content WG, which now has more than 100 members, discussed content types most at risk of loss and decided that the 4 types defined in this document are our priority to address in the 2014 year. Overarching challenges across content types that we discussed include: First, that different content types present a variety of challenges related to preservation including, for example: o size/scale of content (web archives, big data), o selection of content when we can't preserve everything - how do we define what is important to preserve. Secondly, when we select content to preserve, audiences increasingly expect expanded access options - tools for engaging and interacting with the content, usability standards met, involvement in the selection of content to preserve. Four priority content areas selected for focus in 2014 are, in no specific order: Electronic Records, Research Data, Web and Social Media, and Moving Image and Recorded Sound. Each was selected for the 2014 list because of its importance to many of our organizations, and the already observed loss of this type of digital content. Electronic Records that need to be preserved range from personal records such as electronic diaries, to business and organization electronic records, and to records created by all levels of government. Issues related to electronic records are: -Knowledge related to selection, including retention schedules, legal requirements, and researchers' interests is lacking, and knowledge is lacking about preservation strategies for his content. - Current infrastructure is inadequate in many organizations/government agencies to implement preservation of e-records. Collaborative models need further development. Research Data are an acute challenge already recognized by many organizations. Challenges include: - the scale and often heterogeneous and diverse nature of the data and the many different information standards used. - costs, which are perhaps the overriding challenge. There is significant need for the development of sustainable economic models to cover management of research data throughout the lifecycle. - and, a gap in the knowledge base for stewarding the preservation of research data exists. In most organizations, there is no clear understanding of who is responsible for preserving research data. This area, in particular, seems to be an area in which the digital preservation and curation community could help. Moving Image and Recorded Sound How many in this room have a smartphone with a audio and video recorder or editor on it?   -This is an area where the creation of content has shifted dramatically with the advent of easy tools that fit in your pocket, the sheer amount of content being created is overwhelming stewards - flood of digital content poses new challenge: -storage and management of large files -ability to play back in the future - vital for content creators and stewards work together to develop new standards -not only are we talking about born digital, but also content converted from older storage media, which has a short lifespan, but then the new digital files need preservation Web and Social Media More organizations than ever preserving web content, but… -Websites are more complex than ever and tools in use for crawling and access not able to keep up -Full text indexing of web archives remains a challenge, though the international community has been working on this, there are still no easy solutions -Social media companies don't have preservation as a business model, and we are concerned about the ability to preserve with the constant and ever-changing barriers that companies put in place -Privacy issues, particularly when it comes to social media preservation, present challenges as well. Even though there is a significant need for research related to web archives preservation and access, very little funding has been allocated to this end, that we are aware of.    -There is some encouraging news about a recent grant by the Mellon Foundation to Columbia University Libraries to pursue collaborative web archiving initiatives, including the development of software, however the micro-grants that Columbia will pass out will be relatively small (maximum 25K each). -There was also a prior grant by the IMLS, who funded work on automatic classification of web pages from a collaborative End of Term government web archive
• #21: the set of interconnected technical elements that provides a framework for supporting an entire structure of design, development, deployment and documentation in service of applications, systems, and tools for digital preservation includes hardware, software, and systems. Organizational policies, practice, and regulation inform many of the observations and recommendations for the development of digital stewardship infrastructure. Main areas of need: File Format Action Plan Development Many formats to manage Standards needed/guidelines Interoperability and Portability in Storage Architectures Sharing data tapes/storage……. plug and play Integration of Digital Forensics Tools Research needed Ensuring Content Integrity Migration assuring content Fixity checks/validity checks clear need for organizations to share assessments of institutional risk and plans for mitigating those risks; to develop use‐case driven best practices for fixity in particular system designs and configurations; to move the basic research in digital forensics tools from research to implementation in production workflows for organizations. need for integration interoperability portability related standards and protocols
• #22: the set of interconnected technical elements that provides a framework for supporting an entire structure of design, development, deployment and documentation in service of applications, systems, and tools for digital preservation includes hardware, software, and systems. Organizational policies, practice, and regulation inform many of the observations and recommendations for the development of digital stewardship infrastructure. Main areas of need: File Format Action Plan Development Many formats to manage Standards needed/guidelines Interoperability and Portability in Storage Architectures Sharing data tapes/storage……. plug and play Integration of Digital Forensics Tools Research needed Ensuring Content Integrity Migration assuring content Fixity checks/validity checks clear need for organizations to share assessments of institutional risk and plans for mitigating those risks; to develop use‐case driven best practices for fixity in particular system designs and configurations; to move the basic research in digital forensics tools from research to implementation in production workflows for organizations. need for integration interoperability portability related standards and protocols
• #23: So I’m am going to provide a quick overview of the Agenda’s final section, the Research Priorities section. Research in digital stewardship faces a number of challenges – digital preservation is, of course, a long-term responsibility and its benefits serve broad and diverse communities and can be difficult to quantify and funding in this area tends to be project or content specific. But beyond those challenges the agenda identifies 5 areas of opportunity. I’ll discuss each research priority individually, but the priorities are organized around five areas: Applied Research for Cost Modeling and Audit Modeling Understanding Information Equivalence and Significance Policy Research and Trust Frameworks Preservation at Scale And Strengthening the Evidence Base for Digital Preservation Topic Area 1: Applied Research for Cost Modeling and Audit Modeling There is currently little research in cost modeling beyond that focusing on the capacity and density of specific storage media and costs of storage through time. The national agenda recommends more research into the benefits and value of preserved content through time – be it through research use, cultural documentation, transactional accountability, or other uses. Cost and audit modeling need to move beyond analyzing pure infrastructure costs. Similarly, developments like cost calculators that could allow institutions to match content types, infrastructure needs, and resource capacity can help with appraisal and management decisions as far as digital content selected for preservation. Lastly, cost and audit models need to account for multiple storage methods: local, consortial, distributed, cloud, hybrid approaches – the many options for storing and auditing digital content can present stewards with many choices, choices that additional research can support. Topic Area 2: Understanding Information Equivalence and Significance The ease of duplication and transmission of digital objects is one of their most appealing qualities, but also one of the most challenging when it comes to identifying uniqueness and significance. Information content and information context are often divergent. At the same time, qualities that signal or encapsulate divergence, for instance technical metadata, are often invisible to users and even collection managers. Further research into the grades of instantiation of digital materials and how these differences and equivalences impact selection and preservation will enhance future stewardship practices. Tools like fuzzy hashing to recognize bitstream similarity, tools to characterize fidelity and compare image patterns or frame rates – these can facilitate improvements to how we manage and preserve large sets of digital objects. Topic Area 3: Policy Research and Trust Frameworks Digital preservation research and literature has identified a range of specific risks to digital content: failure of many types -- software, hardware, media, organizational – as well as obsolescence, custodial risk, natural disaster, etc – these are all well defined and have led to the emergence of collaborative and consortial efforts to spread and reduce risk, efforts like LOCKSS, Data-PASS, MetaArchive, and Digital Preservation Network. But study of the reliability and design of these frameworks has not kept pace. Better tools are need to assess and measure, quantifiably, existing policy models and trust frameworks. An area of potential research is testing methods of self-evaluation for compliance with standards like ISO 16363 (TRAC) and these can be matched with peer-reviewed and community-driven assessment. Modularity of auditing is another area of potential interest. Lastly, policy and trust frameworks continue to operate at a high, often purely conceptual level that makes implementation, not to mention comprehension, difficult for some organizations. Further research in these areas can lead to the design and evaluation of policy and trust frameworks that better mitigate risk through time and better facilitate the overall preservation of digital content. Discussion at our prior presentation of the agenda focused largely on the chance to extend and test collaborative models, especially research focused on governance models. Topic Area 4: Preservation at Scale As the size of digital collections continues to grow, better understanding is needed into the ramifications of big data on matters of preservation and access. The prototyping and development of new technologies that will allow stewards to scale up activities like indexing, validating, querying, and delivering digital content must continue at the same pace as digital content creation. The current demands on software and infrastructure made by big data is compounded by changing researcher and user expectations about access, especially research dependent on the availability of large data sets. In addition, data collected and preserved and intended for use at scale also raises questions about privacy, confidentiality, and personally-identifiable information. Digital forensics toolkits, data mining algorithms, an other technologies offer promise; and collaborative, community driven approaches to shared infrastructure and scalability may alleviate some of the demands of big data on the technology stack. These potential solutions will benefit greatly from additional research. Topic Area 5: Strengthening the Evidence Base for Digital Preservation Finally, underpinning all these topics is a notable lack of empirical evidence in digital stewardship research. Questions of cost, value, scope, scale, statistics on the rates of bit corruption or hardware failure, obsolescence, reliability and replication – all require a shared evidence base to drive future research and technology and policy development. Use cases, formal test beds of content available for research, and other community-supported bodies of evidence need to be available for testing in many different technology and institutional environments. Evidence-based methodologies from other disciplines can provide a model for this work and outcome-based and replicable testing and experimentation using standardized sample data in operational environments will help extend the evidence base for digital preservation. Areas of discussion at DP13 included ensuring that research in these areas can be translated to smaller organization, making innovation more applicable and better navigating the move from innovation and development work to implementation, capitalizing on demonstrated use cases and social interest to better define the value proposition inherent in digital stewardship, better coordination between content types and selection and appraisal decisions, and an overall increase in precise and quantifiable goals and outcomes for digital stewardship research.
• #32: Thanks!