Downloaded 12 times
![Experimentally manipulate additional prompts
Clickable link. + Prompts embedded into hints
[Click here to learn about the “What? Why? How?” strategy]](https://image.slidesharecdn.com/hcibrownbagv4smrefsnodata-131120154214-phpapp02/75/Using-Experiments-and-Cognitive-Science-Research-to-Improve-the-Design-of-Online-Resources-for-Learning-12-2048.jpg)
Uploaded byJoseph Jay Williams
Using Experiments and Cognitive Science Research to Improve the Design of Online Resources for Learning
1) Joseph Jay Williams conducts research using experiments and cognitive science to improve the design of online learning resources. 2) His work focuses on explaining concepts, teaching learning strategies, adding motivational messages, developing experimental paradigms, and designing experiments into online courses. 3) Some findings include that explaining "why" helps learning more than simply stating facts, teaching self-questioning strategies like "What? Why? How?" improves understanding, and embedding growth mindset messages increases student motivation more than positive messages alone.
More Related Content
![Evaluating Educational Technology in Early Childhood [At the McCormick Center]](https://cdn.slidesharecdn.com/ss_thumbnails/evaluatingedtechmccormickleadership2012mcmanisforposting1-120529092903-phpapp02-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Using Experiments and Cognitive Science Research to Improve the Design of Online Resources for Learning
Using Experiments and Cognitive Science Research to Improve the Design of Online Resources for Learning
- 1. Using Experiments andCognitive Science Research to Improve the Design of Online Resources for Learning Joseph Jay Williams [email protected] www.josephjaywilliams.com/researchoverview 1
- 2. Online Education &Learning Online • New research area? • Convergence of computational & behavioral science NIPS “Data-Driven Education” New ACM conference “Learning at Scale” CHI
- 3. Novel Research Opportunity:Real-World + Laboratory
- 4. Integrate Research &Practice • Randomized assignment • Experimental Control • Rich data • Generalizable theories • “in vivo” experiments • Diverse populations • Real-world environment • Authentic activities • Practical Challenges • Practical improvements • Disseminate research • Generate Funding
- 5. Overview • • • • • Explanation & Learning TeachingLearning Strategies Motivational Messages Experimental Paradigm Experiment-focused Design 5
- 6. Why does explaining“why?” help learning? • General boost to Learning Engagement vs. • The Subsumptive Constraints Account: Interpret target of why-explanation in terms of a broader generalization (Williams & Lombrozo, 2010) • • 2x3=6 Why? 6
- 7. Explanation and Learning:Lab Online • Discovery & transfer GLORP (Williams & Lombrozo, 2010, Cognitive Science) • Use of prior knowledge (Williams & Lombrozo, 2013, Cog. Psych.) • Erroneously overgeneralize at expense of exceptions • Promotes belief revision – given sufficient anomalies (Williams et al, 2013, JEP: General) (Williams, Walker, Maldonado & Lombrozo, 2012; 2013, Cog Sci Conference; in prep) • Prompts in online (math) exercises (Williams, Paunesku, Haley, Sohl-Dickstein, 2013, AIED Moocshop; ongoing) DRENT
- 8. Learning Task &Experimental Paradigm • Online (Math) Exercise 1. Number of Problems Completed 2. Percent Correct 8
- 9. Logic of myPrevious Research Explain why this is correct. Elaborate on what you are thinking now. • • • • Post-Study test questions Transfer/Generalization questions Questions about key principle Memory for details 9
- 10. Future: Generate, Receive,Compare Generate E.g. Explain why this is correct. Receive Compare Geza Kovacs Simultaneously Learning AND Crowdsourcing Improvement of Learning Resources Williams, Thille, Siemens, Trumbore, Stigler. How online resources can facilitate interdisciplinary collaboration. Invited talk to be presented at SIG on Computer and Internet Applications in Education, AERA 10 2014.
- 11. Teaching Learning Strategies •Spend time teaching specific content, or general strategies? • Online: Collect data that would be extremely difficult to get in the real world • Online: Repeatedly reinforce habits & educational behaviors • Teach “What? Why? How?” selfquestioning/explanation strategies (Palinscar & Brown, 1984, Cognition and instruction; McNamara, 2004, Discourse Processes; Williams & Lombrozo, 2010, Cognitive Science) • Understanding vs. Problem-solving • vs. Interpreting vs. Practice-as-Usual 11
- 12. Experimentally manipulate additionalprompts Clickable link. + Prompts embedded into hints [Click here to learn about the “What? Why? How?” strategy]
- 13. Self-questioning strategy: What?Why? How?
- 14. Embedded Prompts betweenHint/Solution Steps
- 15. Prompts Prompt type Promptedunderstanding What? What does thisstep mean to you? Why? Why is what you are currently doing Why is it helpful to helpful? Why is it take this step? useful for achieving your goal? How? How well is your current approach to this problem working? How do you know this step is right? Promptedproblem-solving Promptedinterpretation What are you doing What is this step or thinking right saying? Restate it in now? your own words.
- 16. Explanation study Practice-asusual Control Self-Regulating Thinking Reflecting on Meaning Problems Attempted Problems Correct PracticeTasks Completed Mastery Tasks Completed Problem Accuracy 16
- 17. Explanation study Practice-as-usual No Textbox Textbox ProblemsAttempted Problems Correct Practice Tasks Completed Mastery Tasks Completed Problem Accuracy 17
- 18. Learning Behavior Support •Clickable link to Drop-Down text with suite of strategies: Are you stuck? Click here for some tips. • Provide previously examined prompts. • Use mouseover and drop-down text to reveal information “as requested”, rich traversal of options, guided by student
- 19. Natural Link toLearning Strategy Training • If you want to learn more about strategies to keep motivated and learn well, go to tiny.cc/learningassistant or XX or YY
- 20. 3. Add motivationalmessages Practice-as-usual Growth Mindset Message Remember, the more you practice the smarter you become! 20
- 21. 3. Embedded invivo Experiment • Benefit of Growth Mindset Message? • Practice-as-usual Jascha Sohl-Dickstein • Growth Mindset Message • • "Remember, the more you practice the smarter you become.”, "Mistakes help you learn. Think hard to learn from them.” 21
- 22. Results: More motivated? •Growth Mindset Message > Practice-asUsual • extra problems attempted • more problems correct • Percent Correct: Problems correct/Problems attempted • increase in Percent Correct 22
- 23. 3. Add motivationalmessages Practice-as-usual Message Growth Mindset Message Positive Remember, the more you practice the smarter you become! Some of these problems are hard. Do your best! 23
- 24. Does any positivemessage work? • Practice-as-usual • Growth Mindset Message • • "Remember, the more you practice the smarter you become.”, "Mistakes help you learn. Think hard to learn from them.” • Positive Message • • "Some of these problems are hard. Just do your best." "This might be a tough problem, but we know you can do it.” 24
- 25. Effects of PositiveMessages? • Positive Messages ~= Practice-asUsual • Growth Mindset > Positive • extra problems attempted • more problems correct • increase in Percent Correct 25
- 26. Computational Modeling • • • • • • Williams, Mitchell,Heffernan. MOOC Research Initiative grant from Gates Foundation & Athabasca. Investigating the benefits of embedding motivational messages in online exercises. 2 million users on 12 kinds of fractions exercises, ~100 problems each Moderators & Mediators Item Response Theory Non-parametric Bayesian clustering of Users (CrossCat, JMLR) Model latent knowledge states 26
- 27. Synthesize Scientific Findings • Williams,J.J. (2013)Improving Learning in MOOCs by Applying Cognitive Science. Paper presented at the MOOCshop Workshop, International Conference on Artificial Intelligence in Education, Memphis, TN. • www.josephjaywilliams.com/education 27
- 28. Experimental Paradigm: R.E.P.E.A.T. • • Williams,J. J. (2013). Finding connections between basic experimental research and realistic online education contexts. In J. J. Williams (chair), Online Learning and Psychological Science: Opportunities to integrate research and practice. Symposium conducted at the annual convention of the Association for Psychological Science. Williams, J. J., Renkl, A., Koedinger, K., Stamper, J. (2013). Online Education: A Unique Opportunity for Cognitive Scientists to Integrate Research and Practice. In M. Knauff, M. Pauen, N. Sebanz, & I. Wachsmuth (Eds.), Proceedings of the 35th Annual Conference of the Cognitive Science Society. 28 Austin, TX: Cognitive Science Society. (pdf)
- 29. Experiments • • Williams, J.J. &Williams, B.A. (under review). Online A/B Tests & Experiments: A Practical But Scientifically Informed Introduction. Course proposal submitted to ACM CHI Conference on Human Factors in Computing Systems. Toronto, Canada. (pdf) Williams, J.J., Heffernan, N., & Koedinger, K. Experiments at Scale: Instrumenting MOOCs for experimentation and course-improving data analysis. Tutorial proposal submitted to the First Annual ACM Conference on Learning at Scale. 29
- 30. Experiment-Focused Design • Williams,J.J. & Williams, B. A. (2013). Using Interventions to Improve Online Learning. Paper to be presented at the NIPS 2013 Workshop on Data Driven Education. 30
- 31. Review • • • • • • Explanation & Learning TeachingLearning Strategies Motivational Messages Experimental Paradigm Experiment-focused Design Williams, J.J., Klemmer, S., Kizilcec, R., & Russel, D. (under review). Learning Innovations at Scale. Workshop proposal submitted to ACM CHI Conference on Human Factors in Computing Systems. Toronto, Canada. 31
- 32. Acknowledgements • • • • • Jascha Sohl-Dickstein Jace Kohlmeier& Khan Academy Sam Maldonado Lytics Lab (lytics.stanford.edu) VPOL (Vice Provost of Online Learning, online.stanford.edu) 32
Editor's Notes
- #2 Originally from Trinidad. I’m a Research Fellow in theLytics Lab & Office of Online Learning. Cognitive Science background – experimental psychology, statistical modeling and machine learning.Core interest is being a knowledge broker – reviewing and synthesizing the thousands of published studies in cognitive science, education research, the learning sciences to see which ones are relevant to real-world outcomes, and building products or conducting experiments that improve practically ad financially valuable outcomes. Illustrate this approach with an experiment that shows how to improve students’ motivation while learning from mathematics problems on Khan Academy.
- #7 Not traditional, but ubiquitous
- #8 Not traditional, but ubiquitous