Two ways to improve your survey, webinar for Delib 2023.pptx
Download as PPTX, PDF0 likes174 views
The document discusses ways to enhance the effectiveness of surveys, emphasizing the importance of identifying the most crucial question and recognizing burning issues that respondents want to convey. It highlights the need for trust in the survey process, balancing perceived effort with potential rewards, and outlines how to structure surveys to obtain relevant data for decision-making. Additionally, the document suggests methods for improving survey design and the significance of representativeness in the respondents' pool.
More Related Content
Similar to Two ways to improve your survey, webinar for Delib 2023.pptx (20)
Two ways to improve your survey, webinar for Delib 2023.pptx
- 1. Two ways to improve your surveys The Most Crucial Question and the Burning Issue Caroline Jarrett @cjforms #SurveysThatWork2023
- 2. Caroline Jarrett @cjforms (CC) BY SA-4.0 2 Agenda A bit about me Surveys and consultations Big Honkin’ Surveys and Light Touch Surveys Improve your survey’s effectiveness 1. The Most Crucial Question 2. The Burning Issue Representativeness The survey process
- 3. Caroline Jarrett @cjforms (CC) BY SA-4.0 3 I’m interested in questions because I’ve worked on forms for over 30 years Image credit: Caroline Jarrett
- 4. Caroline Jarrett @cjforms (CC) BY SA-4.0 4 Tax stuff arrives in brown envelopes
- 5. Caroline Jarrett @cjforms (CC) BY SA-4.0 5 The envelope made me think of goals Allows someone to achieve a goal
- 6. Caroline Jarrett @cjforms (CC) BY SA-4.0 6 People separated the form immediately Looks like a form and works like a form Allows someone to achieve a goal
- 7. Caroline Jarrett @cjforms (CC) BY SA-4.0 7 Within that, there are lots of questions Looks like a form and works like a form Asks questions and expects answers Allows someone to achieve a goal
- 8. Caroline Jarrett @cjforms (CC) BY SA-4.0 8 I turned to the survey literature to learn more, especially about questions Image credit: Caroline Jarrett
- 9. Caroline Jarrett @cjforms (CC) BY SA-4.0 9 $1 in the envelope beats $10 guaranteed later Image credit: Caroline Jarrett
- 10. Caroline Jarrett @cjforms (CC) BY SA-4.0 10 Response depends on effort, reward and trust People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response. Diagram from Jarrett, C, and Gaffney, G (2008) “Forms that work: Designing web forms for usability” inspired by Dillman, D.A. (2000) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method”
- 11. Caroline Jarrett @cjforms (CC) BY SA-4.0 11 Perceived effectiveness can be rewarding People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response. I want to help this organisation to make a good decision Diagram from Jarrett, C, and Gaffney, G (2008) “Forms that work: Designing web forms for usability” inspired by Dillman, D.A. (2000) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method”
- 12. Caroline Jarrett @cjforms (CC) BY SA-4.0 12 Agenda A bit about me Surveys and consultations Big Honkin’ Surveys and Light Touch Surveys Improve your survey’s effectiveness 1. The Most Crucial Question 2. The Burning Issue Representativeness The survey process
- 13. Caroline Jarrett @cjforms (CC) BY SA-4.0 13 What’s your recent experience with surveys? • Think of a survey you’ve seen recently in local government • What was it about? • What level of response did it get? • What decision(s) did it contribute to?
- 14. Caroline Jarrett @cjforms (CC) BY SA-4.0 14 We looked for some example surveys Leighton Buzzard Milton Keynes West Northants Dacorum Consult on change of use of council-owned land Feedback after booking an appointment at the tip Dog poo survey as part of researching “Public spaces protection order” Change a regulation to reflect a change of the name of a law Thanks to Jane Matthews for her help
- 15. Caroline Jarrett @cjforms (CC) BY SA-4.0 15 This Leighton Buzzard consultation closes on 6 June 2023 Have your say on plans to improve the area south of High Street | Central Bedfordshire Council
- 16. Caroline Jarrett @cjforms (CC) BY SA-4.0 16 Milton Keynes asks for feedback on every transaction
- 17. Caroline Jarrett @cjforms (CC) BY SA-4.0 17 West Northamptonshire tangled with dog poo Proposal 1. Failure to clean up after your dog. Persons in control of a dog must clean up and remove its faeces, poo or mess from the area and place it in a bin or take it home for disposal. Question 1A To what extent do you agree or disagree with Proposal 1? Strongly agree Agree Neither agree nor disagree Disagree Strongly disagree Don’t know (Public Pack)Agenda Document for Cabinet, 11/10/2022 18:00 (moderngov.co.uk) – questionnaire results from page 159 Public Spaces Protection Order Consultation: ownership of dogs and prohibition of smoking in certain places. - West Northamptonshire Council - Citizen Space
- 18. Caroline Jarrett @cjforms (CC) BY SA-4.0 18 This Dacorum consultation got no responses Marlowes Amendment Order 2023 (dacorum.gov.uk)
- 19. Caroline Jarrett @cjforms (CC) BY SA-4.0 19 Let’s think about effectiveness and response High response Low response High effectiveness Low effectiveness
- 20. Caroline Jarrett @cjforms (CC) BY SA-4.0 20 This is where I’d put three of my examples Dacorum Change a regulation to reflect a change of the name of a law West Northamptonshire Dog poo survey “Public spaces protection order” Milton Keynes Feedback after booking an appointment at the tip High response Low response High effectiveness Low effectiveness
- 21. Caroline Jarrett @cjforms (CC) BY SA-4.0 21 At first, I rated the Leighton Buzzard one highly Dacorum Change a regulation to reflect a change of the name of a law West Northamptonshire Dog poo survey “Public spaces protection order” Leighton Buzzard Change of use of council-owned land Milton Keynes Feedback after booking an appointment at the tip High response Low response High effectiveness Low effectiveness
- 22. Caroline Jarrett @cjforms (CC) BY SA-4.0 22 But I looked into it some more and now, maybe not High response Low response High effectiveness Low effectiveness Dacorum Change a regulation to reflect a change of the name of a law Milton Keynes Feedback after booking an appointment at the tip West Northamptonshire Dog poo survey “Public spaces protection order” Leighton Buzzard Change of use of council-owned land Background papers describe a survey with 120 responses from community groups about their use of space Consultation is about removing car-parking spaces to make them available for development
- 23. Caroline Jarrett @cjforms (CC) BY SA-4.0 23 What about the examples you had? High response Low response High effectiveness Low effectiveness
- 24. Caroline Jarrett @cjforms (CC) BY SA-4.0 24 Agenda A bit about me Surveys and consultations Big Honkin’ Surveys and Light Touch Surveys Improve your survey’s effectiveness 1. The Most Crucial Question 2. The Burning Issue Representativeness The survey process
- 25. Caroline Jarrett @cjforms (CC) BY SA-4.0 25 Interviewers used to visit every respondent Image credit: http://www.census.gov/history/www/genealogy/decennial_census_records/
- 26. Caroline Jarrett @cjforms (CC) BY SA-4.0 26 The 1950s mindset was “Ask Everything” Survey = Big Honkin’ Survey
- 27. Caroline Jarrett @cjforms (CC) BY SA-4.0 27 Now we can do Light Touch surveys • Choose one question • Find a person • Ask the question • See if you can make one decision • Improve, iterate, increase
- 28. Caroline Jarrett @cjforms (CC) BY SA-4.0 28 You can iterate to a larger Light Touch survey Time for new question
- 29. Caroline Jarrett @cjforms (CC) BY SA-4.0 29 Agenda A bit about me Surveys and consultations Big Honkin’ Surveys and Light Touch Surveys Improve your survey’s effectiveness 1. The Most Crucial Question 2. The Burning Issue Representativeness The survey process
- 30. Caroline Jarrett @cjforms (CC) BY SA-4.0 30 A survey is an ‘ask’ method to find out ‘how many’ Observe Ask Why? qualitative How many? quantitative Usability test Field study Interview Survey Analytics
- 31. Caroline Jarrett @cjforms (CC) BY SA-4.0 31 Establish your goals for your survey What do you want to know? Why do you want to know? What decision will you make based on these answers? What number do you need to make the decision?
- 32. Caroline Jarrett @cjforms (CC) BY SA-4.0 32 Discussing “the number” is the biggest challenge What do you want to know? Why do you want to know? What decision will you make based on these answers? What number do you need to make the decision? This number is the “survey statistic”. It’s the overall result of the survey such as “93% agreed”. It’s not the number of responses.
- 33. Caroline Jarrett @cjforms (CC) BY SA-4.0 33 I try to focus the team on a very specific decision What do you want to know? “We want to know which social media our customers use” Why do you want to know? “We want to know how to use our social media advertising budget” What decision will you make based on these answers? “We will choose where to place our advertisements” What number do you need to make the decision? “The proportion of our customers who use each type of social media”
- 34. Caroline Jarrett @cjforms (CC) BY SA-4.0 34 It’s not unusual to have a sequence like this What do you want to know? “We are worried about our job application process” Why do you want to know? “We want to make improvements” What decision will you make based on these answers? “We will focus on the biggest pain point” What number do you need to make the decision? “The point where we get the biggest dropout”
- 35. Caroline Jarrett @cjforms (CC) BY SA-4.0 35 Surveys are for counting not discovery Observe Ask Why? qualitative How many? quantitative Usability test Field study Analytics Interview Survey If you don’t need to - ask questions AND - get a number, try something else.
- 36. Caroline Jarrett @cjforms (CC) BY SA-4.0 36 Let’s go back to our four challenges What do you want to know? Why do you want to know? What decision will you make based on these answers? What number do you need to make the decision?
- 37. Caroline Jarrett @cjforms (CC) BY SA-4.0 37 What’s the Most Crucial Question? “ The Most Crucial Question is the one that makes a difference. It’s the one that will provide essential data for decision-making. You’ll be able to state your question in these terms: We need to ask ___________________ So that we can decide ___________________” Caroline Jarrett (2021) “Surveys that work: A practical guide for designing and running surveys”
- 38. Caroline Jarrett @cjforms (CC) BY SA-4.0 38 Takeaway Be clear on the Most Crucial Question for every survey
- 39. Caroline Jarrett @cjforms (CC) BY SA-4.0 39 Prune: remove questions you don’t need Image credit: https://portal.ct.gov/- /media/CAES/DOCUMENTS/Publications/Fact_Sheets/Plant_Pathology_and_Ecology/ PRUNINGANINTRODUCTIONTOWHYHOWANDWHEN0415Rpdf.pdf
- 40. Caroline Jarrett @cjforms (CC) BY SA-4.0 40 A Crucial Question: can we lose parking places?
- 41. Caroline Jarrett @cjforms (CC) BY SA-4.0 41 Is this enough context for the MCQ? “A larger redevelopment site is a more attractive opportunity to developers and regeneration can make a bigger difference to the town centre. Car parking studies have shown that although Duncombe Drive car park is well used, there is plenty of parking elsewhere in the town” “How far do you agree or disagree that the Council should redevelop the land it owns south of the High Street?” “How far do you agree or disagree that a community / cultural facility is important to include in the redevelopment potential of the site?”
- 42. Caroline Jarrett @cjforms (CC) BY SA-4.0 42 Do usability testing on your questions Get someone to fill in your questionnaire while you watch Extra tip: Get them to tell you what they are thinking as they work out their answers to the questions Image credit: Drawn by Tasia Graham, illustration from Caroline Jarrett, “Surveys that work” (2021)”
- 43. Caroline Jarrett @cjforms (CC) BY SA-4.0 43 Takeaway It’s OK to include a few extra questions to set the context for the MCQ. Make sure you test them.
- 44. Caroline Jarrett @cjforms (CC) BY SA-4.0 44 Agenda A bit about me Surveys and consultations Big Honkin’ Surveys and Light Touch Surveys Improve your survey’s effectiveness 1. The Most Crucial Question 2. The Burning Issue Representativeness The survey process
- 45. Caroline Jarrett @cjforms (CC) BY SA-4.0 45 Response depends on effort, reward and trust People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response. Diagram from Jarrett, C, and Gaffney, G (2008) “Forms that work: Designing web forms for usability” inspired by Dillman, D.A. (2000) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method”
- 46. Caroline Jarrett @cjforms (CC) BY SA-4.0 46 Burning Issues are things that people want to tell you
- 47. Caroline Jarrett @cjforms (CC) BY SA-4.0 47 Interview users about the topics in your survey • Who are they? • How will you find them? • Do they want to answer your questions? • What are their Burning Issues? • Do they understand your questions? Image credit: design by Julia Allum, words by Caroline Jarrett
- 48. Caroline Jarrett @cjforms (CC) BY SA-4.0 48 Survey methodologists start with interviews Observe Ask Why? qualitative How many? quantitative Usability test Field study Analytics Interview Survey
- 49. Caroline Jarrett @cjforms (CC) BY SA-4.0 49 A ‘further comments’ box is also a space for Burning Issues
- 50. Caroline Jarrett @cjforms (CC) BY SA-4.0 50 Takeaway It’s good to make space for people to tell you what they want to tell you
- 51. Caroline Jarrett @cjforms (CC) BY SA-4.0 51 Agenda A bit about me Surveys and consultations Big Honkin’ Surveys and Light Touch Surveys Improve your survey’s effectiveness 1. The Most Crucial Question 2. The Burning Issue Representativeness The survey process
- 52. Caroline Jarrett @cjforms (CC) BY SA-4.0 52 “Every single time” surveys get low response rates • Who will look at the response? • What decision will get made?
- 53. Caroline Jarrett @cjforms (CC) BY SA-4.0 53 Keep it realistic. Extremely short fails, too.
- 54. Caroline Jarrett @cjforms (CC) BY SA-4.0 54 Is this any better? I’ve added: • A “Question of the day” • A representativeness question, to help us understand who answered
- 55. Caroline Jarrett @cjforms (CC) BY SA-4.0 55 Takeaway Check these four things about each question 1. Does it help us to understand who answered? 2. Is there a way for people to tell us their Burning Issues? 3. Is it the Most Crucial Question? 4. It is essential to lead into the Most Crucial Question?
- 56. Caroline Jarrett @cjforms (CC) BY SA-4.0 56 Agenda A bit about me Surveys and consultations Big Honkin’ Surveys and Light Touch Surveys Improve your survey 1. The Most Crucial Question 2. The Burning Issue Representativeness The survey process
- 57. Caroline Jarrett @cjforms (CC) BY SA-4.0 57 Here is my process in stages Establish your goals for the survey Decide who to ask and how many Build the questionnaire Run the survey from invitation to follow-up Clean and analyse the data Present the results Goals Sample Questionnaire Fieldwork Responses Reports Test the questions Questions
- 58. Caroline Jarrett @cjforms (CC) BY SA-4.0 58 You get a better survey by doing many things well Establish your goals for the survey Decide who to ask and how many Build the questionnaire Run the survey from invitation to follow-up Clean and analyse the data Present the results Questions you need answers to The right people in the sample Goals Sample Questionnaire Fieldwork Answers from the right people Responses Reports Accurate Answers Useful Decisions Test the questions Questions Questions people can answer Questions people can interact with
- 59. Caroline Jarrett @cjforms (CC) BY SA-4.0 59 Caroline Jarrett twitter @cjforms [email protected] www.effortmark.co.uk
Editor's Notes
- #46: People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response.
- #58: The seven steps are: - Goals, where you establish your goals for the survey and create questions that you need answers to - Sample, where you decide who to ask and how many and end up with people you will invite to answer - Questions, where you test the questions and end up with questions that people can answer - Questionnaire, where you build the questionnaire and end up with questions people can interact with - Fieldwork, where you run the survey from invitation to follow-up and end up with people who actually answer - Responses, where you clean and analyse the data and end up with answers - Reports, where you present the results and end up with decisions
- #59: The seven steps are: - Goals, where you establish your goals for the survey and create questions that you need answers to - Sample, where you decide who to ask and how many and end up with people you will invite to answer - Questions, where you test the questions and end up with questions that people can answer - Questionnaire, where you build the questionnaire and end up with questions people can interact with - Fieldwork, where you run the survey from invitation to follow-up and end up with people who actually answer - Responses, where you clean and analyse the data and end up with answers - Reports, where you present the results and end up with decisions