Details of the data can be obtained from here

In this project, we do both association mining and sequence mining of web log data from NASA Kennedy Space Center. The goal is to find interesting insights as to which pages are popular and visited together in July and August of 1995. The association mining will give us insights into which pages were typically visited together in a single session – this will allow NASA to place emphasis on which web pages should be updated based on this historical trend. The way a user navigates the website will also be evident from the sequence mining. The sequences mined would give an insight on how NASA can improve their website design based on how a user traversed the page in 1995.

Due to the nature of the web logs, which are in Apache web-server log format, a lot of pre-processing of the data was required to get the web logs into a format that was acceptable by the arules package for association mining and arulessequences package for sequence mining. We also filtered out error responses and media (images and videos) so that we focused on the actual pages visited. Also, sessionization of the web logs was done at 2 levels – host only and host-date with the time sessionized to 30 minutes.

Initial data exploration of the data was done, giving us insights into the most frequently accessed pages. There was a huge spike of views on 13th July 1995. This could be explained by the launch of STS -70 from the Kennedy Space Center in Florida on this day. For evaluation of the rules, we split the logs into train and test sets. The train data was based in July, and the test data was based in August.

We first mined the associations with the logs sessionized based on the hostname with 30 mins sessions. When the generated rules were investigated it was found that a lot of the rules related to the Apollo 13 mission, likely because of a popular movie released at the end of June 1995 popularizing the space mission. The generated rules were also visualized using the arulesviz package in R and rules with high support, confidence and lift were identified.

We also ran the sequence mining package on the host-only dataset. We noticed a pattern in the websites accesses pertaining to the NASA space missions launching around the same time frame. For both the association and the sequence mining, the precision and recall values were noted on the test set, along with the number of rules generated. We noticed a clear trend of precision-recall trade-off in our findings when we used varying values of support and confidence which might be due to the number of rules generated by each set of values.

There were 2 key pieces of information obtained. First, movies like Apollo 13 are responsible for influencing the general public’s browsing of NASA’s website. Secondly, we found that space launches and missions conducted by NASA influences the public’s browsing patterns. To really generalize the association/sequence mining models, web logs would have to be collected over a longer time frame. Similar launches in future may follow the same web page access pattern, and thus these insights will serve well as a starting point for any re-design work NASA may want to introduce for their web pages.