Evaluating a new electronic service using mixed methods

Editor's Notes

• #3: – overview of the sessionThis session will introduce you to a project that we have been running at Teesside University to introduce a new electronic service at Teesside University. In September we introduced LibGuides a hosted web based services to support information literacy and so this presentation will be about the various methods we have been using to evaluate this new service and reflecting on our approach. So we will start by explaining the background and what LibGuides are and how they have been developed. We will share with you how we have tried to gain information from stakeholders and what information we have gained so far (eg on their usage) and what we plan to do next.There will be an opportunity for your questions in the end.
• #4: Denise – Before we implemented Libguides, we had links to all of the resources we had for a subject in subject guides, but these were very static, and the structure of our web site meant that the help for the resources was separate from the actual resources. The guides themselves were just a list of links and the headings for the sections of the subject guides had be agreed as each subject area needed to use the same headings.
• #5: So for example this is what the business subject guide – looked like. As I’ve said just a list of links to various types of resources. There was no focus on a task a student might want to perform they needed to understand what online databases via Search It meant – there was information below to explain a little more but do people read it?Students got to their subject guides from the navigation on the left – because of the timing of the project teaching material had already been designed telling students to click on the link to Subject Guides and colleagues were reluctant to replace this with a LibGuide link and so for this year we kept both the original Subject Guides and introduced the new LibGuides.
• #6: BusinessLibGuideThis is what the new Business LibGuide looks like. For anyone not familiar with LibGuides the new service uses a library content management system to present information tailored to a specific discipline. It is very easy to create guides and all staff in my team were involved where in the past only a small number of the team would have had the skills needed to create a webpage like this. We aimed to bring together all of the most useful resources for a subject, along with instructions, tutorials and details of the academic librarians supporting that subject - in effect creating a one stop shop.We have one of these replacing each of our original subject guides. Initally we created a template which was used by all subjects, this speeded up conversion to the new guides and meant that core information is only input once and then cascaded to all LibGuides. We are now working on developing them to be more discipline specific. The structure isTab-based, organised by a mixture of different formats of information and tasks. This is the welcome screen: our strategy is to push relevant info at students through the VLE rather than to expect them to navigate our web site. Each Blackboard module will link to the relevant subj guide. We are using RSS feeds to add the latest news to this page and a calendar of events that updates to show the latest month . This is also the opportunity to introduce the subject team.
• #7: Through areas like Finding books we can push key texts to students. Through the e-books – area we can promote the eversion of heavily used print titles , as these can be fairly hidden from students. In the section we are looking at now we can point out major journals collections and online for that discipline together with factsheets, tutorials etc. shows students how to reference this type of material – in a relevant style for their school.
• #8: Denise –So in summary, some of the positive aspects of libguides.Customisable so able to tailor them to the needs of a subject in consultation with academic staff
• #10: Denise We have been monitoring usage of LibGuides since they were introduced and take up has been good. 161,465 hits so far in comparison to the Old subject Guides which over the same time period last year = had 41,747 hits. this years usage of the old guides has dropped to just 10,725 a drop of just over 74%. (74.31%)
• #11: Statistics from Sep-April.Show graph of under-used guides – take to librarians meeting?Compare a few well-used guides – top 5 – any trends?
• #13: Limitations of libguides statistics – no idea of the path that users taking or time spent or how reached the site – so implemented Google Analytics. Any results?
• #15: One of the early decisions we made on planning the evaluation of LibGuides was to use a feedback form. This is a facility that LibGuides gives you and building a form is very easy. We implemented a feedback form on every LibGuide. Asking users to rate the LibGuide out of 5 based on how useful it was. Space was also available to add comments. Despite high usage we have only received 85 feedback forms from the site. None from the Business School who had the most used guide! Most respondents scored the guides highly and based on the score we could have got we achieved almost maximum scores. Some students also left comments and many of these were also positive. Plus there were some useful suggestions about how we could improve guides.
• #16: However, only 24% of the returned forms included comments and where the score was low there was often no comment at all which was frustrating. Unlike other methods we had used for forms in the past like Survey Monkey of Google Docs the responses came as individual emails and so even with the low response it was time consuming to collate the feedback. We didn’t know who was returning the forms and so We thought this would give us a good opportunity to obtain feedback on the Libguides – but it hasn’t proved as useful as we had initially hoped. For next year, we may not abandon feedback but will consider using our own survey form which will mean feedback will automatically be collated.
• #17: Swap to Sue – going now from general feedback to our evaluation with specific user groups, beginning with academic staff.Introduced guides to them before we launched them to students – but only interested staff attended. Feedback from those staff was very positive. Subject librarians publicised the guides with their schools – again feedback was positive, but we don’t know whether the staff then promoted the guides with students.We gave a presentation on LibGuides to our Teesside Learning and Teaching Conference this year. We asked whether libguides should be extended to include “study skills” and How can we involve academic staff?
• #18: When we asked whether libguides should be extended to include “study skills” these were the areas that were mentioned.When we looked at the feedback from the L&T Conference even though only one group mentioned Dissertation, there was lots of feedback that could be encompassed in a guide to writing a dissertation eg report writing, time management – we have a separate project working on this. So we found this feedback useful.
• #19:  We didn’t find these responses particularly useful – quite random statements that didn’t always link to the questions, but maybe that this was due more to the limitations of the conference session:Limitations of methodology eg range of attendees (also included learning technologists and ICT), difficulty of engaging them with actual question within short timescalePut into groups and asked as a group to write on post it's and struggled to get feedback so 5 mins is not enough. Could have asked for longer session but an hour too long?In hindsight should have asked them to start individually and then discuss(NHS not able to access due to firewalls. Problems with social media and ICT departments.)
• #20: Focus group earlier this month.
• #21: Strong agreement on positives – especially that they’re easy to create, also easy to use, brings everything together, and looks good
• #22: More varied comments – concerns over info overload, content being hidden, “dumping ground” (negative side of easy to create). Also finding a balance between shared template and freedom to ‘do your own thing’
• #23: A lot of these thoughts were what we’d already decided before the session – but it worked well to allow colleagues to see this for themselves from the evidence rather than just tell them – eg have fewer guides.Questions about whether the headings are meaningful especially More Resources were addressed by the usability study
• #24: Students are the most important user group. When we thought about what we wanted to learn from them, these were our key questions:Expectations – look back at advantages that we mentioned at the start of this presentation and seeing if reality matchesCan students find what they need? Do students find them overwhelming? (info overload concern of librarians)Shared problems – reference:LibGuides Usability TestingCustomizing a Product to Work for Your UsersRachel Hungerford, Lauren Ray, Christine Tawatao, Jennifer WardUniversity of Washington Libraries – Seattle, Washington USA https://digital.lib.washington.edu/dspace/bitstream/handle/1773/17101/UWLibGuidesUsability-2010LAC-tawatao.pdfTesting carried out in 2009.Also research from:
• #25: Usability protocol – based on research from Uni of Washington, but also Nielsen’s web usability advice and our own experience of testing our web site several years ago.Also informed by all our evaluation up to this point – aspects that we particularly want to know about such as whether students find the tabs and sub-tabs easy to use Specific tasks - As well as 5 Tasks, X and Os exercise when students indicate useful and less useful parts of pages, also more general questions about how use libguides – up to 1 hour in total.
• #26: Tasks – start with easy (basic) question2. and 3 - Key tasks that want students to be able to do – find expensive resources –ebook and journal article
• #27: Two tasks based on specific concerns:4. Do students realise what the resource icons do (for factsheet) – do they understand that online databases are within Finding Journal Articles5. Newspaper database beneath More Resources – does this hide them? (particular concern of librarians)Early days – can only report on 2 students experience – but demonstrated keenness to use search box rather than navigate the site. Also just as Google Analytics is giving us more than we expected, these usability studies are also providing us with much richer data.
• #28: Work in progress – not yet completed usability study but also needs to be iterative process (make changes, test, make changes ….)
• #29: 20 mins here - Sue - We are at an early stage and so value both your questions but also your feedback