Download as PDF, PPTX
Uploaded byIskandar Noor
Measuring Learning Impact
This document summarizes a knowledge sharing session on measuring learning impact. The session: 1) Discussed the importance of learning and development programs in organizations and different models for evaluating training, including Kirkpatrick's four levels of evaluation. 2) Presented two sample training courses and asked how evaluation could be conducted for each, focusing on which level of evaluation is most appropriate. 3) Noted that for the sample courses, level 4 evaluation measuring business impact would not be possible due to challenges in collecting relevant data. 4) Emphasized that the purpose of learning interventions is to impact the business through revenue generation, cost reduction, process improvement and people engagement.
More Related Content
Similar to Measuring Learning Impact
Measuring Learning Impact
- 1. Knowledge Sharing Session MeasuringLearning Impact 1 Facilitator: Noor Iskandar 28 May 2021
- 2. LET’S START BYPUTTING THINGS IN PERSPECTIVE
- 3. The Importance OfLearning And Development In An Ever- Changing World • When looking at how to keep an organization in top form, learning and development can be one of the best ways. The Association for Talent Development, based in the United States, concludes that “organizations are more competitive, agile, and engaged when knowledge is constantly and freely shared.” • In fact, after surveying around 830 companies in 2015, they found that organizations with a culture of learning were some of the highest performers. They also found that they attracted more talented employees and had higher levels of customer satisfaction. • Research done through LinkedIn found that the main way for employees to be productive and achieve goals is for employers to provide them with opportunities for continuous learning. This helps employees continue to improve their skills and be more flexible and adaptable, as changes related to technology and other workplace matters can arise quickly. When continuous learning is supported employees feel valued, helping boost employee engagement and retention. • Seeing how beneficial learning and development programs are, why is it that only 31% of organizations surveyed had well developed learning and development programs? 3 https://gethppy.com/talent-management/importance-of-learning-and-development extracted on 24 May 2021
- 4. ROLE OF LEARNINGAND DEVELOPMENT IN AN ORGANIZATION The Learning & Development Manager is responsible for developing and implementing an organization’s learning strategy. Some but not limited are as below: • Develop and implement learning strategies and programs that meet business needs • Evaluate organizational as well as individual employee development needs • Ensure alignment of learning & development department with business goals • Optimize training processes to improve ROI • Manage and procure L&D technologies to facilitate learning and management of learning initiatives • Design eLearning courses, interactive group training, and other training content • Implementing effective training methods • Manage personnel within the learning & development department • Maintaining the learning & development budget within an organization • Championing and marketing training programs to get managerial buy-in as well as employee commitment The learning & development manager is central to ensuring that employees are able to successfully meet the challenges of their jobs and that they are properly aligned with the business goals of the company.
- 5. 5 Importance of impact of L&Dand measuring them.
- 6. 6 Why the needto Evaluate Learning & Development Interventions
- 7. MARCH/APRIL 2020 trainingwww.trainingmag.com Business leaders’ thoughts and perceptions of L&D Mind Tools for Business survey report, March 2021 The Needs Analysis Playbook: How to Make L&D a Trusted Partner in Your Organization ©2021 All rights reserved. www.sweetrush.com
- 8. L&D is necessaryin developing not just Capability, also Maturity
- 9. First Edition, Jan1990 Learning Organization
- 10. Last but notleast, something very recent 2020 2021
- 11.
- 12. Measuring Learning Impact beginswith the Design of the Learning Solutions 12
- 13. Effective Training Effective trainingis considered as an important factor in determining the efficiency of an organization which depends upon the capability of its employees. For training evaluation to be truly effective, the training and development itself must be appropriate for the person and the situation. Even though evaluation is listed at the last phase, evaluation actually happens during all the phases. It is used during the training process to evaluate the training process itself. Evaluation is not just for the trainer or organization it is absolutely important for the learner too. One new model was found namely Back planning model. Which is same as Kirkpatrick’s model but the process occurs in the reverse order and continues in a cyclic process. Analysis and Evaluation of Training Effectiveness Akhila Kunche , Ravi Kumar Puli , Sunitha Guniganti , Danaiah Puli N.I.T. Warangal Andhra Pradesh, India
- 14.
- 15.
- 16. 2 Commonly UsedModels in Developing Training 16
- 17.
- 18. 6Ds Approach toLearning Development Process 18
- 19. Good Learning Occurs… Effective Delivery Contextual Relevancy Content Validity Hence,learning evaluation will be based on all of them
- 20.
- 21. Before we measurelearning impact, let’s get to know some fundamentals 21
- 22. Two technical concepts Testing(Measuring), Evaluation & Assessment Reliability and Validity
- 23. Testing (Measuring), Evaluation& Assessment In case you are curious, here are some definitions: • A test or quiz is used to examine someone's knowledge of something to determine what he or she knows or has learned. Testing measures the level of skill or knowledge that has been reached. • Evaluation is the process of making judgments based on criteria and evidence. • Assessment is the process of documenting knowledge, skills, attitudes and beliefs, usually in measurable terms. The goal of assessment is to make improvements, as opposed to simply being judged. In an educational context, assessment is the process of describing, collecting, recording, scoring, and interpreting information about learning.
- 24. Reliability and Validity •These two terms, reliability and validity, are often used interchangeably when they are not related to statistics. When critical readers of statistics use these terms, however, they refer to different properties of the statistical or experimental method. • Reliability is another term for consistency. If one person takes the same personality test several times and always receives the same results, the test is reliable. • A test is valid if it measures what it is supposed to measure. If the results of the personality test claimed that a very shy person was in fact outgoing, the test would be invalid. • Reliability and validity are independent of each other. A measurement maybe valid but not reliable, or reliable but not valid. Suppose your bathroom scale was reset to read 10 pound lighter. The weight it reads will be reliable (the same every time you step on it) but will not be valid, since it is not reading your actual weight.
- 25.
- 26. What are thecommonly available training evaluation models? The Kirkpatrick Model Phillip’s ROI Model The CIPP Model The CIRO Model The Brinkerhoff Model Kaufman's Model of Learning Evaluation Anderson Model of Learning Evaluation 26 1 2 3 4 5 6 7
- 27.
- 28. 28 Developed by Dr. JackPhillips Kirkpatrick’s FOUR Levels
- 29. Purpose of Learningand Training vis a vis Evaluation 29 4 3 2 1
- 30. Types of EvaluationEmployed by Organizations 30
- 31. Develop Tests –from example Objectives below Learning Objectives Given the ABC system, display the customer file in 30 seconds and on the first try. Faced with a fire, select appropriate procedure for combating it. Name the capitals of all provinces in Canada, matching each capital to its provinces with no errors. From a collection of bones, identify the femur. Test Suggestion Performance Test. Give the learner a customer name. Have the learner interact with the ABC system to display the file. Check for accuracy, time and number of attempts. Recall Test. Show different types of fires (eg Electrical, gas, flammable liquid). Have learner select the procedure from memory. (We have not asked him/her to combat the fire, only to select the procedure.) Verify accuracy of match. Recall Test. Have learners recite the provinces and their capitals from memory. Verify accuracy. Recognition Test. Display bones. Name femur. Have learner point to it.
- 32.
- 33. Let’s Focus onKirkpatrick’s Levels of Training Evaluation 33
- 34. Donald Kirkpatrick • Kirkpatrickdeveloped a model of training evaluation in 1959. • Arguably the most widely used approach. • Simple, Flexible and Complete • 4-level model.
- 35. Why Evaluate? • Shouldthe program be continued? • How can the program be improved? • How can we ensure regulatory compliance? • How can we maximize training effectiveness? • How can we be sure training is aligned with strategy? • How can we demonstrate the value of training?
- 36. The Four Levelsof Evaluation • Level I: Evaluate Reaction • Level II: Evaluate Learning • Level III: Evaluate Behavior • Level IV: Evaluate Results • Fifth level was recently “added” for return on investment (“ROI”) but this was not in Kirkpatrick’s original model
- 37. Relationship Between Levels •Each subsequent level is predicated upon doing evaluation at lower level • A Level 3 will be of marginal use, if a Level 2 evaluation is not conducted Level 1 - Reaction Was the environment suitable for learning? Level 2 - Knowledge Did they learn anything Level 3 - Behavior KSA being used on the job? Level 4 - Results Was it worth it?
- 38. Slide 38 Improve Learning Environment Improve Knowledge/Skilltransfer Check Performance Environment Check Requirements, Systems and Processes Only by assessing each level can we yield actionable results Level 1 - Reaction Was the environment suitable for learning? Level 2 - Knowledge Did they learn anything Level 3 - Behavior KSA being used on the job? Level 4 - Results Was it worth it?
- 39. Types of AssessmentsUsed at Each Level Level 1 - Reaction Was the environment suitable for learning? Level 2 - Knowledge Did they learn anything Level 3 - Behavior KSA being used on the job? Level 4 - Results Was it worth it? Type Form Summative Correlation of business results with other assessment results Summative Observation of Performance 360° Survey Diagnostic Summative Self-assessment Test Reaction Formative Survey Real-time Polling Quizzing
- 40. Summary of Toolsto Purpose If you want to… Level 1 Level 2 Level 3 Level 4 Continue program X X Improve program X X X Ensure compliance X Maximize training effectiveness X Align training with strategy X X Demonstrate training’s value X X X X
- 41. Evaluation Techniques Evaluation Levels Methods 1 Reaction 2 Learning 3 Behavior 4 Results Survey● ● ● ● Questionnaire/Interview ● ● ● ● Focus Group ● ● ● ● Knowledge Test/Check ● Skills Observation ● ● Presentations ● ● Action Planning ● ● ● Action Learning ● Key Business HR Metrics ●
- 42. Let’s do acouple of exercises to see if we know this stuff… Before we do that…jeng jeng jeng 42
- 43. Sample #1: PlantOperation and Maintenance Training COURSE OBJECTIVES 1. Describe in detail how to read P & I diagrams and the wealth of information they contain 2. Explore the main challenges of plant start-up, continuous operation & optimization, and shutdown 3. Equip participants with the ability to operate plants efficiently and quantify plant operations through performance indicators and technical calculations 4. Develop participants’ capacity for troubleshooting methodologically using available process trends and events recording 5. Present real case scenarios and examples for materializing the troubleshooting process and focusing on the importance of root causes 6. Investigate plant maintenance strategies and priorities 7. Empower participants to plan turnarounds, execute, and evaluate them. 8. Emphasize the importance and relevance of core operations values and safety principles on operations processes 9. Explain the creation and implementation of SOP • How would you see the evaluation look like? • Which level of Evaluation should you employ? • How do you think the training should be conducted?
- 44. Sample #2: ManagementTraining for the New Manager or Supervisor • Including how to: 1. Build a successful team. 2. Prioritize new management responsibilities. 3. Effectively communicate with new staff. • Participants will learn to: 1. Differentiate between “managing” and “leading”. 2. Effectively transition from staff to management. 3. Establish performance standards and expectations for new teams. 4. Create action plans to reach goals. 5. Delegate effectively. 6. Deliver effective feedback, both positive and negative. 7. Facilitate conflict resolution among staff. 8. Understand the work style differences among different generations. 9. Establish trust and credibility while building team culture. • How would you see the evaluation look like? • Which level of Evaluation should you employ? • How do you think the training should be conducted?
- 45. Based on thetwo public courses examples, you will notice there is no Level 4 – Impact Evaluation Why? 45
- 46. Purpose of Learningand Training vis a vis Evaluation 46
- 47. To Summarize thissection… Revenue Generation Cost Reduction Process Improvement People Engagement Purpose of Learning Interventions are to Impact the Business for:
- 48. ROI Model andProcess How to Measure Training Results : A Practical Guide to Tracking the Six Key Indicators Hardcover – February 27, 2002 by Jack Phillips (Author), Ron Stone (Author) Many organizations have challenges in collecting and extracting data. Hence, conducting ROI for training will be difficult.
- 49. Summary Analysis •What Knowledge? (L2) •What Skills or Behavior? (L3) •What Business Impact?(L4) •Is your training worth the investment? (L5) Design •Learning Outcome to solve issues in Analysis. •What test or measurement do we use to determine if Learning Outcome is achieved? Develop •Training & Learning Materials to support Analysis and Design, including the test / evaluation. Implement (Delivery) •What delivery style or approach will be able to meet learning outcome and the test / evaluation? Evaluate •Actual evaluation happens 49
- 50. Finally, looking atADDIE model Analyze Design Develop Implement Evaluate Analyze Design Develop Implement Evaluate Delivery Areas L&D should be good at
- 51.
- 52. We are donefor this session… 52
- 53. Knowledge Sharing Session MeasuringLearning Impact 53 Thank you