Wednesday 6 May: Hand me the data! What you should know as a humanities researcher before asking for data from a web archive, Ulrich Have, NetLab/DIGHUMLAB, Aarhus University

Editor's Notes

• #2: Hello everyone. Welcome to this short presentation: Hand me the data! What you should know as a humanities researcher before asking for data from a web archive
• #3: Hello, my name is Ulrich. I am a research IT developer at NetLab. I work with both researchers and developers. On a daily basis I assist researcher from the humanities and social science in using web archive data in their research. I have a humanities background from Aarhus University but also spent some time in computer science in a program called Software Construction. My own interest in the web dates back to 1996. It was a glorious time when we used Netscape Navigator and a few of us were using e-mail.
• #4: These are some of the things I will cover: What is web archive data? How do you get it and how do you handle it? And do you need to be a computer scientist? I have to note that this account is about a danish research project using the Danish web archive. The Danish web archive is a national archive run by the Royal Danish Library under the legal deposit law. It is only open to researchers because of laws governing personal data and copyright. The access points are a Wayback Machine url-search and full-text search offering a few basic search options. Not all web archives are the same; they offer different user interfaces and different possibilities when it comes to data analysis.
• #5: We normally browse web archive data. This data is just like any other web content. It is being served by a server and rendered in your browser. Chances are that your national web archive has created some collections or aggregate data. They may also have created an index of all the content. This means that a parser has been parsing the information and put it into a structured form. The information can now be searched in various ways and particular types of data can be explored.
• #6: The mHealth project is an interesting case because it has a mixed team of health and communication researchers. The project contacted NetLab for help in finding a way to explore “mHealth” in 10 years of web data. mHealth is a term used for mobile health technologies. mHealth is part of the larger domain of eHealth. It has no authoritative definition and is constantly evolving as new technologies and practices emerge.
• #7: When the team approaced us they had already been granted access to the national web archive and had tried to do some research.The usual browsing of the web archive quickly turned out to be very hard because of the time frame of 10 years. There were simply too many pages. Furthermore, it appeared that there was a variety of uses of the term mHealth by different actors in different contexts. So it made sense to ask for the complete data in order to study it using computational methods.
• #8: So we tried asking for the data...
• #9: Exactly what do we mean by data? Clearly when you are using a web archive you are interacting with data. But you don’t have a clue about how data is stored in the archive or if any metadata is available. So your first step would be to understand your web archive in more detail. In the mHealth project we first looked at which types of data existed in the archive. Luckily all content has been indexed which means that there is a big pool of data. It is usually hidden to the user but as it turns out it can be valuable for doing computational analysis. The mHealth team decided to ask for these data types: links, text and metadata. It also tried to get the information in CSV files instead of WARC files. This choice was a good one because we would get medium sized files that could be quickly delivered.
• #10: This work would lead to a specifications document called the ETL document. The ETL clearly specifies queries and data fields. The queries can be used to define a certain collection or corpus for example ”all documents in Danish containing the word mHealth excluding documents from Twitter.” The data fields define data sets for metadata, text and links. The links and text derive from the web pages. The metadata can be information about the page such as date, content type, title and so on. As you will see, having a data set w. metadata is very useful for exploring a collection.
• #11: The mHealth project was one of the first projects to have data delivered from the Danish web archive. Through agreement the web archive would deliver data to the project owner who would then become responsible for the data. One of the first steps was to make sure that data could be stored and shared safely. The team would have to develop a data management mindset and some IT skills. Plus some infrastructure was needed, but not much.
• #12: To sum it up, having data from a web archive comes with some extra requirements. Such as IT infrastructure, software and legal issues.
• #13: Data was delivered via secure file sharing service hosted by DeiC, The Danish e-infrastructure Cooperation (DeiC), a national body that coordinates Danish digital infrastructure as an umbrella for the eight Danish universities. It is possible to send many gigabytes of data per delivery. The project owner and all participants have to comply with data laws. This means that data must be stored and transferred in a secure way. The mHealth project chose to use a university server which was only accesible to the team and only via the university network.
• #14: So this is how the project work was organised: data was stored on the university server and only accessible to the project team. In order to comply w. data laws, all access was logged. The code and reports were stored in a private repository on GitHub. Team members could then share work documents in a consistent workflow.
• #15: The main focus was on exploratory data analysis. The central tool for the mHealth team was the notebook. A notebook is a document with both text and code. Various programming languages support the notebook method, in the case of mHealth we used the statistical programming language R to produce the notebooks. One advantage of the notebook is that it allows researchers to use a shared notebook to perform their own interactive exploration of data by setting variables and executing code. For instance, the mHealth notebook would allow for a year-by-year analysis of the data. The first analysis would use the metadata. Many insights into mHealth could be made by simply looking at the dataset from different angles: which domains had most pages? Which month or year stood out?
• #16: Another advantage was that the team could produce their own project deliverables directly in the notebook. No need for special software or servers.
• #17: So no, you don’t need to be a computer scientist! As we learned from the mHealth project exploratory data analysis is possible for all team members with very little effort in terms of IT skills and data management. Using the right setup and a data management mindset the infrastructure for doing computational analysis is available to any small to mid size organisation. Modern computer technology, networks and security technology makes it possible to safely work on web archive data. The researcher can explore data, perform different kinds of analysis, and create visualizations and derived datasets. Most importantly the research team can work better as a team by building a shared workflow.
• #18: In conclusion the mHealth case shows Data delivery from a national web archive brings freedom of analysis because you are not restricted to a certain research infrastructure. Having data shifts concerns towards being a manager not just a user. This requires data management skills something that is of value to researchers not just in working with web archives. Having a shared workflow is useful for teams. Just like in software development code and documents can be stored in a repository on the internet or in the organisation.
• #19: Finally I hope to have inspired some of you to learn more about your national web archive. Ask them if they offer data. Some, like the UK Web Archive, have datasets that you can download. Maybe your web archive has created aggregate datasets or even an index. Ask if they offer IT services for a specific research task. They often know the possibilities and limitations of their archive. Use your own instutition to learn and educate yourself in IT and data management. And don’t forget to give credit - web archives are doing a great service with often very limited resources!
• #20: Thank you!