Presentation LMU Munich: The power of learning analytics to unpack learning and teaching: a critical perspective
Download as PPTX, PDF1 like465 views
The document discusses the critical perspective on learning analytics, detailing its implementation at the Open University and its potential to improve learning outcomes. It highlights various examples of using analytics to track student performance and engagement, addressing both the benefits and challenges of data utilization in education. The findings suggest that learning design and analytics can significantly enhance student success and support personalized learning experiences.
Presentation LMU Munich: The power of learning analytics to unpack learning and teaching: a critical perspective
- 1. @DrBartRienties Professor of Learning Analytics The power of learning analytics to unpack learning and teaching: a critical perspective 30th of October 2019 Ludwig-Maximilians-Universität München Fakultät für Psychologie und Pädagogik
- 2. First an apology Adeniji, B. (2019). A Bibliometric Study on Learning Analytics. Long Island University. Retrieved from https://digitalcommons.liu.edu/post_fultext_dis/16/ There is too much exciting stuff happening at the OU!!!
- 3. Agenda 1. What is learning analytics? 2. Exemplar 1: How is the OU implementing LA on a large scale? 3. Exemplar 2: Can we track how good learners are in search- skills? 4. Exemplar 3: Can we predict what is a good learning design? 5. What are the main affordances and limitations of Learning Analytics in terms of data?
- 4. (Social) Learning Analytics “LA is the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs” (LAK 2011) Social LA “focuses on how learners build knowledge together in their cultural and social settings” (Ferguson & Buckingham Shum, 2012)
- 5. Dyckhoff, A. L., Zielke, D., Bültmann, M., Chatti, M. A., & Schroeder, U. (2012). Design and Implementation of a Learning Analytics Toolkit for Teachers. Journal of Educational Technology & Society, 15(3), 58-76.
- 6. Dyckhoff, A. L., Zielke, D., Bültmann, M., Chatti, M. A., & Schroeder, U. (2012). Design and Implementation of a Learning Analytics Toolkit for Teachers. Journal of Educational Technology & Society, 15(3), 58-76.
- 8. Hlosta, M., Herrmannova, D., Zdrahal, Z., & Wolff, A. (2015). OU Analyse: analysing at-risk students at The Open University. Learning Analytics Review, 1-16.
- 9. Prof Paul Kirschner (OU NL) “Learning analytics: Utopia or dystopia”, LAK 2016 conference
- 10. 1. Increased availability of learning data 2. Increased availability of learner data 3. Increased ubiquitous presence of technology 4. Formal and informal learning increasingly blurred 5. Increased interest of non-educationalists to understand learning (Educational Data Mining, 4profit companies) 6. Personalisation and flexibility as standard
- 11. Exemplar 1: How is the OU implementing LA on a large scale: Leading global distance learning, delivering high-quality education to anyone, anywhere, anytime The Open University Largest University in Europe No formal entry requirements enter with one A-level or less 33% 38% of part-time undergraduates taught by OU in UK 173,927 formal students 55% of students are 'disadvantaged' FTSE 100 have sponsored staff on OU courses in 2017/8 60% 66% of new undergraduates are 25+ 1,300 Open University students has a disability (23,630) 1 in 8 Students are already in work 3 in 4 employers use OU learning solutions to develop workforce
- 12. 13,206 learners 6,000 learners 123,000 learners • A pioneering Moodle-based award winning site providing free access to learning – 1,000+ courses, plus articles and videos. • The OU publishes curriculum as short courses onto OpenLearn and creates free Badged Open Courses (BOCs) which reward informal learners with a badge / certificate of achievement. • Links from BBC broadcasting with themed pages developed to support the content of many series. In addition, print resources can be ordered such as posters. 7.8m new learners each year 60m since launch in 2006 Opening Access: OpenLearn
- 13. Innovative and engaging content for broadcast audiences and rich resources for students. 2016/17: the OU co-produced 35 series: • Generated 250 million viewing and listening events across channels and platforms. • Directed 1.2 million viewers to OpenLearn. Dedicated channel on YouTube with bite- sized learning. It is the largest educational presence on YouTube in EU with: • Over 1,700 public videos with 50 million views. • Over 166,000 subscribers to our channel, more than any other UK educational institution • Reaching over 3 million learners per year. The OU now engages with learners on Facebook: • Viewed 6.2 million times by 2.5 million users. • We run Facebook live sessions to engage around topical issues. • We will be doing more with Facebook on the DfE Flexible Learning Fund in 2019. Reaching out to millions more 250m views 2016/17 3m learners per year 2.5m in 2017
- 14. • 2m transactions a day • 6m quiz questions answered a year • 4500 tutors • Groups of 15-20 • 8-16 hours a week • Home, work, trains, prisons, submarines • 130 degrees and other qualifications • From 450+ modules Study Experience Overview
- 15. STUDENT SUCCESS ANALYTICS 15 O R G AN I S AT I O N AL C APAB I LT I E S Productionised output and MI Strategic analysis Modelling / AI Data collection Data storage and access Technology architecture Learning design and delivery Student lifecycle managemen t Continuous improvemen t and innovation Creation of actionable insight Availability of data Impact the student experience Adapted from Barton and Court (2012) - https://hbr.org/2012/10/making-advanced-analytics-work-for-you
- 16. HOW ANALYTICS SUPPORTS STUDENT SUCCESS STUDENT SUCCESS ANALYTICS KPIs STUDENT SUCCESS PRIORITIES DATA & ANALYTICS WILL SUPPORT, ALIGN TO, AND PROACTIVELY DRIVE THE RECRUITMENT AND STUDENT SUCCESS STRATEGY THROUGH CONTINUOUS INNOVATION CONVERSION STUDY CONTINUATION EMPLOYABILITY STUDY ENGAGEMENT WEBSITE APP ALVLECHATBO T PHONECOMMUNICATION CHANNELS DATA ANALYTICS CAPABILITY CRM & DIGITAL (ANALYTICS) INTEGRATION LAYER ENTERPRIS E DATA HUB BI ENABLEMENT DATA SCIENCE ENABLEMENT ANALYTICS INDUSTRIALISATION DATA SECURITY, PRIVACY & GOVERNANCE Build Your Own / Self-serveStrategic Analysis Canned Reporting BUSINESS INTELLIGENCE DIGITAL EXPERIENCE OFFLINE CHANNELS Statutory Reporting ANALYTICS & DATA SCIENCE Modelling, AI & Machine Learning Test & LearnKnowledge Graph Survey Analytics Sentiment Analysis Trigger Based Interventions Digital Journey Optimisation Personalised Prospect Targeting Personalised Recommendations ANALYTICS CAPABILITY 16 SUPPORTED OPEN ENTRY IMPROVED COMMUNICATIONS FLEXIBLE STUDY INTENSITY INCREASED RETURN RATES INCREASED YR3 CONTINUATION RATES INCREASED SATISFACTION IMPROVED PERSONAL OUTCOMES INCREASED PASS RATES IMPROVED CAREER OUTCOMES D ATA & AN ALY T I C S V I S I O N
- 17. Using Predictive Learning Analytics A quick history of OU Analyse Pres. Scope Delivery 2014B 2 modules, selected course members Excel spreadsheet, sent by email, manually generated 2014J 12 modules, selected course members Automated pred., excel spreadsheet 2015J Made available to tutors in selected modules Dashboard, within OU network 2016J ~1000 users (338 accessed) , 785 tutors (305 accessed) Dashboard on Internet, no VPN needed, grade predictions 2017J,18 B 37 modules, 375 users (240 accessed), 323 tutors (204 accessed) Mostly STEM pilots, user acceptance 2018J 250 modules in OUA, ~3500 users, including mostly tutors, module chairs, staff tutors, cluster managers, SSTs Dashboard combining OUA predictions and SIO Student Probabilities
- 18. 18 What does it do? It produces predictions as to whether students are at risk of failing their studies. The model predicts on a weekly basis whether or not a given student will submit their TMA. It uses a traffic light system to pinpoint in red students at risk, in amber those with a moderate probability of failing and in green those who are unlikely to fail. OU ANALYSE https://analyse.kmi.open.ac.uk/#dashboard
- 19. Herodotou, C., Hlosta, M., Boroowa, A., Rienties, B., Zdrahal, Z., Mangafa, C. (2019). Empowering online teachers through predictive learning analytics. British Journal of Educational Technology. 50(6), 3064-3079. Impact factor: 3.142.
- 20. Herodotou, C., Hlosta, M., Boroowa, A., Rienties, B., Zdrahal, Z., Mangafa, C. (2019). Empowering online teachers through predictive learning analytics. British Journal of Educational Technology. 50(6), 3064-3079. Impact factor: 3.142. “Teachers in the “average” OUA frequency group had significantly better average student’s performance in course presentations in which they used OUA than in presentations without OUA. These findings may be explained by existing non- PLA-related approaches and thus used to monitor students’ engagement with the course material. As shown in other studies (Herodotou et al., 2017), OUA helped teachers in monitoring their students’ behavior online in an effective manner; rather than having to search for disperse student’s engagement information in, for example, forums and the VLE, teachers had all relevant information gathered together in a single space (OUA).”
- 21. 21 2017/18: Use of OUA
- 22. In summary • Uptake of LA a long and non-linear process • Use of OUA by teachers significantly influences students’ performance • Large differences in practices (i.e., discipline, module, teacher characteristics)
- 23. Exemplar 2: Are students well skilled in searching the internet? ● Across the globe people are assumed to have good internet searching skills ● However, recently there is a debate whether this is actually the case? ● In particular, some have raised concerns about a widely used self-report instrument called Internet-Specific Epistemic Questionnaire (ISEQ)? ● Are students well skilled in searching the internet? ● (How would you set up a design to test this?) 23
- 24. • Lab study whereby 269 students worked in dyads on complex red yeast rice case • We monitored which websites they visited (and which they did not) • We analysed chat data and final dyad answer to government advice Knight, S., Rienties, B., Littleton, K., Mitsui, M., Tempelaar, D. T., Shah, C. (2017). The relationship of (perceived) epistemic cognition to interaction with resources on the internet. Computers in Human Behavior, 73, August 2017, 507–518
- 25. Knight, S., Rienties, B., Littleton, K., Mitsui, M., Tempelaar, D. T., Shah, C. (2017). The relationship of (perceived) epistemic cognition to interaction with resources on the internet. Computers in Human Behavior, 73, August 2017, 507–518
- 26. ● No relation between ISEQ and what students actually do online Knight, S., Rienties, B., Littleton, K., Mitsui, M., Tempelaar, D. T., Shah, C. (2017). The relationship of (perceived) epistemic cognition to interaction with resources on the internet. Computers in Human Behavior, 73, August 2017, 507–518
- 27. Exemplar 3: linking existing datasets • Learning design data (>300 modules mapped) • VLE data • >140 modules aggregated individual data weekly • >37 modules individual fine-grained data daily • Student feedback data (>140) • Academic Performance (>140) • Predictive analytics data (>40) • Data sets merged and cleaned • 111,256 students undertook these modules
- 28. Learning Design is described as “a methodology for enabling teachers/designers to make more informed decisions in how they go about designing learning activities and interventions, which is pedagogically informed and makes effective use of appropriate resources and technologies” (Conole, 2012).
- 29. Assimilative Finding and handling information Communication Productive Experiential Interactive/ Adaptive Assessment Type of activity Attending to information Searching for and processing information Discussing module related content with at least one other person (student or tutor) Actively constructing an artefact Applying learning in a real-world setting Applying learning in a simulated setting All forms of assessment, whether continuous, end of module, or formative (assessment for learning) Examples of activity Read, Watch, Listen, Think about, Access, Observe, Review, Study List, Analyse, Collate, Plot, Find, Discover, Access, Use, Gather, Order, Classify, Select, Assess, Manipulate Communicate, Debate, Discuss, Argue, Share, Report, Collaborate, Present, Describe, Question Create, Build, Make, Design, Construct, Contribute, Complete, Produce, Write, Draw, Refine, Compose, Synthesise, Remix Practice, Apply, Mimic, Experience, Explore, Investigate, Perform, Engage Explore, Experiment, Trial, Improve, Model, Simulate Write, Present, Report, Demonstrate, Critique Conole, G. (2012). Designing for Learning in an Open World. Dordrecht: Springer. Rienties, B., Toetenel, L., (2016). The impact of learning design on student behaviour, satisfaction and performance: a cross-institutional comparison across 151 modules. Computers in Human Behavior, 60 (2016), 333-341 Open University Learning Design Initiative (OULDI)
- 30. Toetenel, L., Rienties, B. (2016). Analysing 157 Learning Designs using Learning Analytic approaches as a means to evaluate the impact of pedagogical decision-making. British Journal of Educational Technology, 47(5), 981–992.
- 33. Nguyen, Q., Rienties, B., Toetenel, L., Ferguson, R., Whitelock, D. (2017). Examining the designs of computer-based assessment and its impact on student engagement, satisfaction, and pass rates. Computers in Human Behavior. DOI: 10.1016/j.chb.2017.03.028. 69% of what students are doing in a week is determined by us, teachers!
- 34. Constructivist Learning Design Assessment Learning Design Productive Learning Design Socio-construct. Learning Design VLE Engagement Student Satisfaction Student retention 150+ modules Week 1 Week 2 Week30 + Rienties, B., Toetenel, L., (2016). The impact of learning design on student behaviour, satisfaction and performance: a cross-institutional comparison across 151 modules. Computers in Human Behavior, 60 (2016), 333-341 Nguyen, Q., Rienties, B., Toetenel, L., Ferguson, R., Whitelock, D. (2017). Examining the designs of computer-based assessment and its impact on student engagement, satisfaction, and pass rates. Computers in Human Behavior. DOI: 10.1016/j.chb.2017.03.028. Communication
- 35. So what happens when you give learning design visualisations to teachers? Toetenel, L., Rienties, B. (2016) Learning Design – creative design to visualise learning activities. Open Learning: The Journal of Open and Distance Learning, 31(3), 233-244.
- 36. Toetenel, L., Rienties, B. (2016) Learning Design – creative design to visualise learning activities. Open Learning: The Journal of Open and Distance Learning, 31(3), 233-244.
- 37. Conclusions I 1. A lot of data is coming into (and out of) education 2. A lot of “semi-standardised” data is gathered within and across institutions 3. Great opportunities to harvest fine- grained and longitudinal data
- 38. Conclusions II 1. What about the ethics? 2. What can be standardised (and what not)? 3. Are we optimising the record player?
- 39. The power of learning analytics to visualise evidence of learning T: drBartRienties E: [email protected] W: www.bartrienties.nl W: https://www.organdonation.nhs.uk/ W: https://www.sportentransplantatie.nl/
Editor's Notes
- #16: ‘Creating better opportunities with and for learners, enabling more to achieve their goals’ Creating– includes our content, people, systems and support services Better – means continuous improvement, always striving to be better than we were yesterday Opportunities – different learning options to meet different needs With and For Learners– all learners from all backgrounds etc. which recognises their diversity and emphasising partnership Enabling – making it easy to engage with us and our products More – again, continuous improvement, striving to attract more students To achieve their goals – which may be a new job or career, a qualification, an accreditation or learning for pleasure.
- #19: Anna
- #28: 5131 students responded – 28%, between 18-76%
- #30: Explain seven categories
- #32: For each module, the learning design team together with module chairs create activity charts of what kind of activities students are expected to do in a week.
- #35: Cluster analysis of 40 modules (>19k students) indicate that module teams design four different types of modules: constructivist, assessment driven, balanced, or socio-constructivist. The LAK paper by Rienties and colleagues indicates that VLE engagement is higher in modules with socio-constructivist or balanced variety learning designs, and lower for constructivist designs. In terms of learning outcomes, students rate constructivist modules higher, and socio-constructivist modules lower. However, in terms of student retention (% of students passed) constructivist modules have lower retention, while socio-constructivist have higher. Thus, learning design strongly influences behaviour, experience and performance. (and we believe we are the first to have mapped this with such a large cohort).