Investigating learning strategies in a dispositional learning analytics context: the case of worked examples
Download as PPTX, PDF2 likes480 views
This document discusses a study analyzing how students use worked examples, tutored problem-solving, and untutored problem-solving in an online math tutorial system called SOWISO. The study examines how these learning modes relate to student performance and dispositions. Key findings include: (1) engagement with tutorials and mastery of content strongly predicts exam scores; (2) students frequently use worked examples and rarely use hints for tutored problem-solving; (3) adaptive dispositions correlate with timely preparation and less example use, while maladaptive dispositions correlate with less preparation and more example use.
Investigating learning strategies in a dispositional learning analytics context: the case of worked examples
- 1. School of Business and Economics Analysing the use of worked examples and tutored and untutored problem-solving in a dispositional learning analytics context Dirk Tempelaar, Maastricht University School of Business and Economics Bart Rienties, Open University UK, Institute of Educational Technology Quan Nguyen, Open University UK, Institute of Educational Technology
- 2. School of Business and Economics Aim Connecting: • Computer Assisted Formative Assessment & Practicing in the domain of mathematics • Learning Analytics based on track data generated by e-tutorial systems • Learning dispositions approach proposed by Buckingham Shum & Deakin Crick (LAK2012) as a framework for learning analytics applications that integrate track data with ‘intentional data’ based on self- report instruments, …to generate learning feedback
- 3. School of Business and Economics
- 4. School of Business and Economics Educational context National (SURF projects, both ‘Testing and Test-based Learning’, and Learning Analytics Stimulus Program): improve transfer high school university, and diminish dropout 1st year, for math, by creating adaptive learning paths using e-tutorials for practicing & formative assessment.
- 5. School of Business and Economics
- 6. School of Business and Economics Educational context National (SURF projects, both ‘Testing and Test-based Learning’, and Learning Analytics Stimulus Program): improve transfer high school university, and diminish dropout 1st year, for math, by creating adaptive learning paths using e-tutorials for practicing & formative assessment. Local (Maastricht): do so for a large, very international and very heterogeneous population of business & economics 1st year students. Data: ‘17/’18 cohort, 1029 students, 75% international.
- 7. School of Business and Economics Learning context Blended/technology enhanced learning, flipped-classroom • Face-to-face component: PBL, problem-based learning tutorials, in small tutorial groups, student-centered, 2nd year students as tutor (72 TGs, 14 students) • Technology-component: 2 digital learning labs: math: SOWISO & stats: MyStatLab (Pearson)
- 8. School of Business and Economics Learning design & feedback in SOWISO SOWISO system provides different instructional designs (McLaren) with different types of learning feedback (Narciss) • Ask Hint learning design represents tutored problem-solving: students receive feedback in the form of hints and an evaluation of provided answers, both during and at the end of the problem-solving steps, helping to solve the problem step-by-step. This feedback is of Knowledge of Result/response, KR feedback type . • Ask Solution learning design represents the worked-out example approach. Many lab- based educational studies find it the most efficient design: ‘examples relieve learners of problem-solving that – in initial cognitive skill acquisition when learners still lack understanding – is typically slow, error-prone, and driven by superficial strategies’ (Renkl). It corresponds with Knowledge of the Correct Response, KCR type feedback. • Check Answer learning design represents untutored problem-solving: feedback is restricted to the evaluation of provided answers at the end of the problem-solving steps. In feedback terms: Multiple-Try type feedback.
- 9. School of Business and Economics Learning aids • Learning aids in SOWISO: – Check: untutored problem- solving – Theory: view a theory page explaining the mathematical principle – Solution: worked example – Hint: tutored problem-solving
- 10. School of Business and Economics Solution in SOWISO • The solution page provides a full worked example. After studying this, the next step is to ‘redo’: another version of the exercise loads (through parameterization), students can prove mastery in that new version.
- 11. School of Business and Economics Educational aim: predictive modelling, signalling students at risk / in need for help Being in a ‘data rich’ learning environment providing multi-modal data, our prime educational aim is to find the best predictors for students most in need of additional help. Data available for this predictive modelling: 1. LMS, BlackBoard trace data, click and time-on-task 2. E-tutorial trace data (SOWISO, MyLab): mastery data, time-on-task data, attempts, hints, solutions, theory views data 3. Learning dispositions (learning styles, goals, motivation, engagement, expectancy-value data, self-theories of intelligence, effort beliefs, …) 4. Prior education, diagnostic prior knowledge tests 5. Formative assessments (quizzes) performance data 6. Final exam
- 12. School of Business and Economics Previous research on predictive power multi-modal data: 1 In previous research (CinHB 2015, 2018) we analysed the predictive power of the different types of data. Outcomes are best expressed as a sequence of steps in predictive modelling of several outcome variables (vertical axis: multiple correlation): 1. BlackBoard trace data: very modest, mostly Week0 data (prior course use) 2. E-tutorial trace data: practicing math and stats, mastery and time 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Week0Week1Week2Week3Week4Week5Week6Week7 MExam2013 SExam2013 MQuiz2013 SQuiz2013 MExam2014 SExam2014 MQuiz2014 SQuiz2014 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Week1 Week2 Week3 Week4 Week5 Week6 Week7 MExam2013 SExam2013 MQuiz2013 SQuiz2013 MExam2014 SExam2014 MQuiz2014 SQuiz2014 E-tutorial trace dataBB trace data Tempelaar, D. T., Rienties, B., & Giesbers, B. (2015). In search for the most informative data for feedback generation: Learning Analytics in a data-rich context. Computers in Human Behavior, 47, 157-167.
- 13. School of Business and Economics Previous research on predictive power multi-modal data: 2 2. E-tutorial trace data: practicing math and stats, mastery and time (repeated) 3. Cognitive data added: diagnostic entry tests & quizzes (formative/low stakes summative assessments) 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Week1 Week2 Week3 Week4 Week5 Week6 Week7 MExam2013 SExam2013 MQuiz2013 SQuiz2013 MExam2014 SExam2014 MQuiz2014 SQuiz2014 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Week0 Week3 Week5 Week7 MExam2013 SExam2013 MQuiz2013 SQuiz2013 MExam2014 SExam2014 MQuiz2014 SQuiz2014 Entry test & quiz dataE-tutorial trace data Tempelaar, D. T., Rienties, B., & Giesbers, B. (2015). In search for the most informative data for feedback generation: Learning Analytics in a data-rich context. Computers in Human Behavior, 47, 157-167.
- 14. School of Business and Economics Previous research on predictive power multi-modal data: 3 3. Cognitive data added: diagnostic entry tests & quizzes 4. All data (learning disposition data added) Intermediate conclusion: we find a very strong causal chain of relations: • Engagement with e-tutorials → Mastery in e-tutorials → Quizzes → Exam. 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Week0 Week3 Week5 Week7 MExam2013 SExam2013 MQuiz2013 SQuiz2013 MExam2014 SExam2014 MQuiz2014 SQuiz2014 Entry & quiz data 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Week0 Week1 Week2 Week3 Week4 Week5 Week6 Week7 MExam2013 SExam2013 MQuiz2013 SQuiz2013 MExam2014 SExam2014 MQuiz2014 SQuiz2014 All data Tempelaar, D. T., Rienties, B., & Giesbers, B. (2015). In search for the most informative data for feedback generation: Learning Analytics in a data-rich context. Computers in Human Behavior, 47, 157-167. Tempelaar, D. T., Rienties, B. Mittelmeier, J. Nguyen, Q. (2018) Student profiling in a dispositional learning analytics application using formative assessment. Computers in Human Behavior, 78, 408-420. doi: 10.1016/j.chb.2017.08.010.
- 15. School of Business and Economics Next question: can we increase our knowledge on why, how and when students engage with e-tutorials? Given the crucial role of engagement with e-tutorials, this research tries to discover other, prior links in the chain: 1. The timing decisions of students: when do students practice? Three subsequent moments: – Tutorial session preparation – Quiz session preparation – Exam preparation 2. The learning mode decisions of students; preference for: – Worked examples – Tutored problem-solving – Untutored problem-solving 3. How do both decisions relate to learning dispositions?
- 16. School of Business and Economics RQ1: Descriptive statistics: timing decisions How do students plan their learning, given the three crucial moments: • Weekly tutorial sessions, Week1 … Week7 • Biweekly quiz sessions, Week3, Week5, Week7 • Final exam, end of course TutorSession QuizSession Exam 0 10 20 30 40 50 60 70 80 90 Week1 Week2 Week3 Week4 Week5 Week6 Week7 TutorSession QuizSession Exam Example: Week1 topics are discussed in tutorial session of Week1, quizzed in Week3, tested in Week8. Conclusion: Quiz is most important learning target. Tutorial mastery: 27% Quiz mastery: 63% Exam mastery: 71% on average (mastery% is cumulative) Mastery%
- 17. School of Business and Economics RQ2: Descriptive statistics: learning mode decisions How do students plan their learning in terms of learning mode decisions: • What is the total number of attempts, averaged per exercise • What share of these attempts are of worked example type: Solutions • What share of these attempts are of tutored problem-solving type: Hints Averages: • Attempts= 1.78, so on average, every exercise is loaded 1.78 times • Hints= 0.07, so on average, 7 hints are called for every 100 exercise attempts • Solutions= 0.76, so on average, 0.76 solutions are called for every exercise attempt Conclusion: students use of tutored problem-solving is quite low, but frequent use of worked examples (43% of attempts are solutions called for). Hints Solutions Attempts0.0 0.5 1.0 1.5 2.0 2.5 Week1 Week2 Week3 Week4 Week5 Week6 Week7 Hints Solutions Attempts
- 18. School of Business and Economics RQ2: Course performance, timing and learning mode To what extent do timing and learning mode decisions matter? Relationships with three course performance measures. Beyond the control variables: • Mastery positively predicts performance, but only timely mastery: Tutorial & Quiz • Attempts positively predicts performance, but only timely attempts: Tutorial & Quiz • Solutions called for negatively predicts performance, but only timely calls: Tutorial & Quiz • Hints called for unrelated to performance • Views called for negatively predicts performance
- 19. School of Business and Economics RQ3: Learning dispositions and learning decisions How do learning dispositions relate to timing decisions, and to learning mode decisions? In describing some of the outcomes, we distinguish adaptive from maladaptive dispositions. Beyond the control variables: 1. Timely preparation (tutorial & quiz session) is positively related to adaptive dispositions as planning skills and persistence, negatively related to self-sabotage. Disengagement negatively relates all phases of preparation. 2. Preparation is positively related to externally regulated learning (being dependent on others to regulate learning, although external regulation is regarded as maladaptive. 3. The use of Solutions (worked examples) is negatively related to the adaptive valuing school, positively to planning, and with regard to maladaptive dispositions, negatively related to disengagement, self-sabotage, failure avoidance. 4. The use of Solutions is positively related to externally regulated learning, unrelated to self-regulation.
- 20. School of Business and Economics Conclusions and limitations • High level of predictive power in predicting course performance with trace and dispositions data; main predictor mastery in e-tutorials. • That mastery construct is itself related to two important learning regulation decisions students make: timing of learning, and learning mode (worked examples, tutored/untutored problem-solving). • Learning regulation decisions are related to learning dispositions, such as cognitive processing strategies, metacognitive regulation strategies, motivation and engagement variables. • Most of these relations are of the nature that adaptive dispositions predict more preparation, more timely preparation and less use of worked examples, and maladaptive dispositions vice versa. • There are exceptions to this: external learning regulation (rather than self- regulation) predicts preparation, and valuing school is negatively related to preparation. • This may connect with the limitation of this study: we can only observe one component of the learning blend, the digital one, and lack any measurement for the face-to-face part. It is highly likely that the most adaptive students (high in self-regulation, high in valuing school) focus on this second component, and are less dependent on the digital mode.
- 21. School of Business and Economics Analysing the use of worked examples and tutored and untutored problem-solving in a dispositional learning analytics context Dirk Tempelaar, Maastricht University School of Business and Economics Bart Rienties, Open University UK, Institute of Educational Technology Quan Nguyen, Open University UK, Institute of Educational Technology
- 22. School of Business and Economics Hints in SOWISO • Hints for tutored problem-solving are context dependent: the stage of solving the exercise determines the hint
- 23. School of Business and Economics Theory views in SOWISO • Theory pages provide a short theoretical summary of the solution steps
- 24. School of Business and Economics Learning disposition surveys • Epistemological self-theories of intelligence (Dweck) • Epistemological views on role effort in learning (Dweck, Blackwell) • Epistemic learning emotions (Pekrun) • Cognitive learning processing strategies (Vermunt) • Metacognitive learning regulation strategies (Vermunt) • Subject (math & stats) attitudes (Schau) • Academic motivations (Deci & Ryan) • Achievement goals (Elliott) • Achievement orientations (Dweck & Grant) • Learning activity emotions (Pekrun) • Motivation & Engaggement wheel (Martin) • Cultural intelligence (SFCQ) • Help seeking behaviour
- 25. School of Business and Economics Example: Martin’s ‘Motivation & Engagement Wheel’ Four quadrants based on: • Thoughts (Cognitions) Behaviours • Adaptive MalAdaptive