Downloaded 36 times
Uploaded byKerry Rice
Educational Data Mining in Program Evaluation: Lessons Learned
The document details various studies conducted by Boise State University focusing on educational data mining to evaluate and improve program effectiveness in K-12 and online graduate teacher education contexts. Key methods employed include decision tree analysis, cluster analysis, and time series analysis to predict student performance, identify at-risk students, and enhance course design. The findings highlight the importance of engagement, course load, and demographic factors in student outcomes and underscore challenges in data collection and analysis within educational environments.
In this document
Powered by AI
Overview of Boise State University's educational data mining research and focus on program evaluation.
Introduction to different educational data mining techniques: Decision Tree Analysis, Cluster Analysis, Sequential Association Analysis, and Time Series Analysis.
Analysis of teacher training workshops using data mining techniques to improve program outcomes with 103 participants and 31,417 learning logs.
Data mining in online education to identify struggling students and improve course design using 2,744,433 Moodle server logs.
Large scale evaluation of K-12 online programs utilizing data mining and survey data, covering 7,500 students and over 23 million logs.
Evaluation of K-12 blended programs using survey and demographic data, focusing on decisions at course and institutional levels.
Real-time identification of at-risk students in online graduate education through time series analysis with 509 enrollments.
Validation of predictive models based on data from the previous study to assess at-risk student identification effectiveness.
Highlighting key engagement variables used in data mining to analyze student performance.
In-depth look at student clustering results showing shared participation characteristics and performance outcomes.
Distribution of student clusters across multiple courses, highlighting performance levels.
Insights into decision tree analysis predicting course satisfaction and student performance based on various factors.
Exploration of course design correlation with learner outcomes using sequential association analysis.
Overview of time series analysis techniques applied to course access and discussion board replies.
Conclusions on student performance patterns, emphasizing the role of engagement and course load on success.
Identifying successful student attributes in K-12 education related to engagement and enrollment in advanced courses.
Profile of at-risk students in K-12 education based on age, course type, and engagement levels.
Challenges faced in data collection for educational data mining, including inconsistencies and missing data.
Discussion of the complexities and challenges of applying data mining techniques in educational settings.
Citations of relevant studies and research articles on educational data mining and program evaluation.
More Related Content
Similar to Educational Data Mining in Program Evaluation: Lessons Learned
Educational Data Mining in Program Evaluation: Lessons Learned
- 1. © 2016 BoiseState University 1 Kerry Rice, Jui-Long Hung, Yu-Chang Hsu, Brett E. Shelton Department of Educational Technology Boise State University Educational Data Mining in Program Evaluation: Lessons Learned AECT 2016, Las Vegas
- 2. © 2016 BoiseState University 2 MET Ed. D. Ed. S. Graduate Certificates: Online Teaching Technology Integration Specialist School Technology Coordinator K-12 Online Teaching Endorsement College of Education
- 3. © 2016 BoiseState University 3 Go Broncos!
- 4. © 2016 BoiseState University 4 Decision Tree Analysis (performance prediction) Cluster Analysis (engagement) Sequential Association Analysis (path analysis) Educational Data Mining Applications Time Series Analysis (future performance prediction)
- 5. © 2016 BoiseState University 5 Study #1: Teacher Training Workshops 2010 • Survey Data + Data Mining + Student Outcomes • Research Goal: – Program improvement – Satisfaction – Impact on practice • Blackboard • 103 participants • 31,417 learning logs • Cluster Analysis; Sequential Association Analysis; Decision Tree Analysis;
- 6. © 2016 BoiseState University 6 Study #2: Online Graduate Teacher Education 2010 • Data Mining + Student Outcomes (no demographic data) • Research Goal: – Identify struggling students – Adjust teaching strategies – Improve course design – Data Visualization • Study Design – Comparative (between and within courses) – Random course selection • Moodle • Two graduate courses (X and Y) • Each with two sections – X1 (18 students) – X2 (19 students) – Y1 (18 students) – Y2 (22 students) • 2,744,433 server logs • Cluster Analysis; Sequential Association Analysis; Decision Tree Analysis; Data Visualization
- 7. © 2016 BoiseState University 7 • Data mining + Demographics + Survey Data + Student Outcomes • Research Goal: Large scale program evaluation – Support decision making at the course and institutional level – Identify key variables and relationships between teacher and course satisfaction, student behaviors, and performance outcomes Study #3: End of Year K-12 Online Program Evaluation 2012 • Blackboard LMS • 7500 students • 883 courses • 23,854,527 learning logs (over 1 billion records) • Cluster Analysis; Decision Tree Analysis
- 8. © 2016 BoiseState University 8 Study #4: End of Year K-12 Blended Program Evaluation 2012 • Blackboard LMS • 255 Enrollments • 33 course sections • 17 unique courses • Satisfaction Survey • Data from 2011 pilot study • Cluster Analysis; Decision Tree Analysis • Demographics + Survey Data + Data Mining + Student Outcomes • Research Goal: Test Framework in Blended Learning – Support decision making at the course and institutional level? – Identify key variables and relationships between teacher and course satisfaction, student behaviors, and performance outcomes
- 9. © 2016 BoiseState University 9 Study #5: Online Graduate Teacher Education, 2014 • Moodle LMS • 509 Enrollments • 25 course sections • 12 unique courses • 431,708 records • Time Series Analysis • 34 original and derived variables –static (demographic) –dynamic (engagement) • Demographics + Data Mining + Student Outcomes • Research Goal: – Could we identify at-risk students in real time? – When?
- 10. © 2016 BoiseState University 10 Study #6: Online Graduate Teacher Education, 2015 • Moodle LMS • 661 Enrollments • 31 course sections • 18 unique courses • 546,965 records • Time Series Analysis • 34 original and derived variables –static (demographic) –dynamic (engagement) • Demographics + Data Mining + Student Outcomes • Research Goal: – Did the model developed in Study #5 work with new semester data? – Were the predictive (timing) results the same? – Were frequency data more predictive?
- 11. © 2016 BoiseState University 11 Variables - actual and derived Engagement
- 12. © 2016 BoiseState University 12 Cluster Analysis Student clustering which describes shared characteristics of students who passed or failed their courses
- 13. © 2016 BoiseState University 13 Cluster Analysis: Relative Participation Levels and Final Grades • Average Time Spent • Average Days Participated • Average Frequency of Mouse Clicks • Average Time Spent per Session • Average Frequency of Mouse Clicks per Session Participation Variables (Engagement)
- 14. © 2016 BoiseState University 14 Cluster Analysis: Student Characteristics Cluster 1 – Low-High, 119 students: Low average participation and higher performance levels. Cluster 2 – High-High, 60 students: High average participation and high performance levels. Cluster 3 – Low-Low, 76 students (46% remedial): Low average participation and low performance levels.
- 15. © 2016 BoiseState University 15 Cluster Distributions in Courses High percentage of Low-Low students in Chemistry CP, CP Pre-Calculus, English II, English III, and Pre-Calculus (H)
- 16. © 2016 BoiseState University 16 Decision Tree Analysis Perception and performance predictions which identify key predictors of course satisfaction, instruction satisfaction, and final grade
- 17. © 2016 BoiseState University 17 Decision Tree Analysis: Predictors of Student Performance Decision Tree Analysis • Identified at risk • Number of Courses Taken • Average Clicks per Week • Average Time Spent per Week, and • Average Time Spent per Session Other factors • Gender • Ethnicity • Reason for taking a course
- 18. © 2016 BoiseState University 18 Sequential Association Analysis Course X Course Y Does the design of the course (path to learning) predict learner outcomes?
- 19. © 2016 BoiseState University 19 Sequential Association Analysis
- 20. © 2016 BoiseState University 20 Time Series Analysis
- 21. © 2016 BoiseState University 21 Time Series Analysis – course access A B F Week 10 Spring Break
- 22. © 2016 BoiseState University 22 Time Series Analysis – DB Replies A B F Week 10 Spring Break
- 23. © 2016 BoiseState University 23 Overall Analysis • Students who took fewer courses performed significantly better than those who took more courses. • Engagement is a significant factor. High-engaged students performed better than low-engaged students. • Students identified as at risk performed differently than all other students. • Type of engagement matters. Students who accessed their courses more often performed better than those who had more interactions within the course. Consistent interaction over time is a better predictor of performance. (higher ed only) • Advanced courses - High-engagement and high performance (K-12) • Entry level courses - Low performance regardless of engagement (K-12) • Gender and ethnicity (higher ed) were identified as significant factors • Satisfaction did not always equate to higher performance
- 24. © 2016 BoiseState University 24 Characteristics of successful students • Female (k-12) • Younger (k-12) • Were enrolled in advanced courses (k-12) • Took fewer courses • Were more engaged overall • Were consistently engaged
- 25. © 2016 BoiseState University 25 Characteristics of at-risk students • Male (K-12) • Older (K-12) • Took entry-level courses (K-12) • Took a greater number of courses • Were low engaged overall • Were inconsistent in their engagement
- 26. © 2016 BoiseState University 26 Data Collection Challenges • Bb activity accumulator grouped wide ranging behaviors into only five useful categories • Missing data (empty fields – ex. internal handler) • Mismatched data fields/data stored in the wrong fields • Inconsistent data collection (i.e. failure to track every forum reply) • Partial or missing timestamp (needed for sequential analysis) • Course or student ID not linked to survey • Demographic data not linked to course or program • Inconsistent course models (blended)
- 27. © 2016 BoiseState University 27 Educational Data Mining Special Challenges • Learning behaviors are complex • Target variables (learning outcomes/performance) require wide range of assessments and indicators • Goal of improving teaching and learning is hard to quantify • Limited number of DM techniques suitable to meet educational goals • Only interactions that occur in the LMS can be tracked through data mining • Still a very intensive process to identify rules and patterns
- 28. © 2016 BoiseState University 28 References •Hung, J. L., Rice, K., & Saba, A. (2012). An educational data mining model for online teaching and learning. Journal of Educational Technology Development and Exchange, 5(2), 77-94. •Hung, J. L., Hsu, Y.-C., & Rice, K. (2012). Integrating data mining in program evaluation of K-12 online education. Educational Technology & Society, 15(3), 27-41. •Rice, K., & Hung. J. (2015). Data mining in online professional development program: An exploratory case study. International Journal of Technology in Teaching and Learning, 11(1), 1-20. •Shelton, B., Hung, J. L., & Baughman, S. (2015). Online graduate teacher education: Establishing and EKG for student success intervention. Technology, Knowledge and Learning. •Rice, K., & Hung, J. L. (2015). Identifying variables important to the success of K-12 students in blended learning. Paper presented at the Northern Rocky Mountain Educational Research Association Conference, Boise, Idaho. •Shelton, B. E., Hung, J. L.., & Lowenthal, P. (under review). Predicting student success by modeling student interaction in online courses.