Understanding Metrics – What to Measure, and Why - John Custy
13 likes•9,232 views
The document outlines the importance and purpose of metrics in IT service management, emphasizing how they inform stakeholders, promote organizational value, and drive performance improvement. It categorizes different types of metrics, including quantitative and qualitative measures, and discusses how to effectively report on performance to ensure alignment with business goals. Additionally, it identifies common service desk metrics and methods for reporting performance to various stakeholders.
Understanding Metrics – What to Measure, and Why - John Custy
- 1. #atlassian
- 2. Understanding Metrics: What to Measure and Why JOHN CUSTY • MANAGING CONSULTANT • JPC GROUP • @ITSMNINJA
- 3. About John Custy Service Management Practitioner, Consultant And Educator • Distinguished Professional in IT Service Management • ITIL Expert & ITIL Service Manager • ITIL Intermediate – SS, SD, ST, SO, CSI, OSA, SOA, PPO, RCV • KT Certified Instructor • ITIL Accredited Trainer • KCS Verified Consultant • ISO/IEC 20000 Consultant • ISFS, ISMAS based on ISO/IEC 27002 • HDI Faculty & Certified Instructor
- 4. A story about metrics…
- 5. Source: Joëlle Flumet’s Broken Bed Installation
- 9. The customer rating is 5/5, but the customer WON’T return!
- 10. What are you measuring? Why are you measuring it?
- 11. What to Measure and Why PURPOSE OF METRICS TYPES OF METRICS COMMON SUPPORT METRICS METHODS OF REPORTING QUESTIONS
- 12. PURPOSE OF METRICS How to use metrics
- 13. PURPOSE OF METRICS How to use metrics • Inform your stakeholders • Report measurements so that stakeholders can understand activities and results • Promote the value of the organization • Determine the best way to communicate the information to the stakeholders • Perform better stakeholder analysis to facilitate stakeholder buy-in • Improve performance - people do what is measured
- 14. PURPOSE OF METRICS What are we trying to accomplish?
- 15. ENSURE ALIGNMENT • Account for IT Processes and Deliverables • Inform stakeholders • Understand IT Performance COMPLIANCE • Achieve certifications; ISO/IEC 20000, COBIT • Measure progress to goals/ objectives OPERATIONAL EXCELLENCE • Measure IT Performance • Control IT Processes • Maximize IT Productivity (people) • Report Costs • Demonstrate value of IT Organization PURPOSE OF METRICS What are we trying to accomplish?
- 16. The Relationship of Metrics to Goals
- 17. The Relationship of Metrics to Goals VISION MISSION GOALS OBJECTIVES CSF KPI METRICS MEASUREMENT
- 18. PURPOSE OF METRICS Sharing Accomplishments What should you report?
- 19. PURPOSE OF METRICS Sharing Accomplishments What should you report? Key performance indicators Critical success factors Variances to baseline Progress towards targets Annotate milestones and abnormalities Service improvement projects
- 20. Understanding the Different Types of Metrics
- 21. DIFFERENT TYPES OF METRICS Metrics & Characteristics Quantitative • How much or how many • Ex. The number of times customers contact the service desk % :)
- 22. DIFFERENT TYPES OF METRICS Metrics & Characteristics Quantitative METRICS ! ! •Performance indicators (PI) ! •Key performance indicators (KPI) ! •Key results indicators (KRI) • How much or how many • Ex. The number of times customers contact the service desk CHARACTERISTICS ! ! •Efficiency vs. effectiveness ! •Leading vs. lagging % :)
- 23. DIFFERENT TYPES OF METRICS Efficiency vs. Effectiveness Quantitative • How much or how many • Ex. The number of times customers contact the service desk % :)
- 24. DIFFERENT TYPES OF METRICS Efficiency vs. Effectiveness Quantitative EFFICIENCY ! ! •How • How fast? much or how many ! •• Ex. The number of times How customers many? contact the service ! desk •Transactional Cost ! •Incident/Request/Access Management ! •Departmental Goals EFFECTIVENESS ! ! •Accuracy ! •Customer Satisfaction ! •Total Organizational Cost ! •Problem Management ! •Enterprise Objectives % :)
- 25. DIFFERENT TYPES OF METRICS Quantitative vs. Qualitative Quantitative • How much or how many • Ex. The number of times customers contact the service desk % :)
- 26. DIFFERENT TYPES OF METRICS Quantitative vs. Qualitative Quantitative QUANTITATIVE ! •How much or how many ! •Ex. The number of times customers contact the service desk • How much or how many • Ex. The number of times customers contact the service desk QUALITATIVE ! •How well something or someone is performing. ! •Ex. Customer Satisfaction, Employee Satisfaction, stock price. % :)
- 27. Service Metrics
- 28. • Time to process an order • Time to check inventory item • Time to send/receive an e-mail • Time to … @ITSMNinja End-to-End Performance
- 29. • Cost Per Transaction • Cost Per User @ITSMNinja End-to-End Performance
- 30. Uptime – compared to … • Downtime per Service • Frequency and total amount of time • Number of incidents (type/category) • Number of recurring incidents • Time per incident @ITSMNinja Service Availability
- 31. • Problems identified per Service • # incidents per problem • Lost time per problem • Changes Per Service • # successful changes (time, budget) • Lost time due to changes – incidents and requests • # Service Requests due to changes • # problems due to changes – IT and business lost time • % improvements due to changes @ITSMNinja Service Availability
- 32. BALANCE NEEDED Operational metrics allow you to understand where to improve Service metrics report on the overall performance of the service
- 33. What Type of Metrics Are Reported?
- 34. What Type of Metrics Are Reported? IT INTERNAL METRICS SENIOR MANAGEMENT SERVICE MANAGEMENT BUSINESS UNIT METRICS REGULATORY/ COMPLIANCE
- 35. Four Types of Process Metrics PROGRESS EFFICIENCY EFFECTIVENESS COMPLIANCE ! IN PROCESS MATURITY ! USE OF RESOURCES ! CORRECT AND COMPLETE THE FIRST TIME ! TO PROCESS AND REGULATORY REQUIREMENTS
- 36. Identifying Common Service Desk Metrics
- 37. @ITSMNinja Typical Operational Measurements • Response • % connected immediately (Real-Time) • Abandon Rate • Wait (hold/queue) Time • Average Speed to Answer (ASA) • Response Time service level XX% in YY seconds • Call-Back Time • Desktop (PC)
- 38. @ITSMNinja Typical Operational Measurements • Resolution • Resolved First Contact • Resolved X hours, Y hours, Z hours • Cases re-opened, Repeats • Requests resolved without assistance (self-help) • Calls/Cases avoided due to self-help
- 39. @ITSMNinja Typical Operational Measurements • Cost • Call/Contact • Customer/User • Total Cost (TCO) of Support/Service
- 40. @ITSMNinja Typical Operational Measurements • Service Desk • Volumes, trends • Performance to goals • Incident Management • Volumes, trends, repeat incidents • Reduction in restoration time • Performance to goals • Customer Satisfaction
- 41. @ITSMNinja Typical Operational Measurements • Request Fulfillment • Volumes, trends • Time to complete requests • Performance to goals • Customer satisfaction
- 42. @ITSMNinja Typical Operational Measurements • Customer Satisfaction • Employee Satisfaction
- 43. @ITSMNinja Typical Operational Measurements • Knowledge Base Usage • Accesses/Searches per contact • # solutions per search • # solutions searched/opened/viewed • Time spent reviewing solutions • Ease of finding solutions • Quality of solutions (ability to use solutions)
- 44. @ITSMNinja Typical Operational Measurements • Service Asset and Configuration Management • Errors in CMDB • Resources improvement utilizing CMDB • Change Management • Number of incidents/requests due to the change • Additional (reduction) workload due to changes • Release Management • Number of incidents/requests due to the release • Additional (reduction) in workload due to releases
- 46. SERVICE DESK METRICS WORKLOAD • • Volumes • Calls/Cases per customer per month • Number of registered users/ Total number of users • Time spent contacting users • Time spent on change related incidents/requests
- 47. SERVICE DESK METRICS WORKLOAD • • Volumes • Calls/Cases per customer per month • Number of registered users/ Total number of users • Time spent contacting users • Time spent on change related incidents/requests INDIVIDUALS • Number of calls taken • Average Handle Time (AHT) • Avai labi l i ty • Occupancy • Number of incidents/requests c l o s e d o n f i r s t contact • Customer Satisfaction • Contribution to Knowledge base
- 48. SERVICE DESK METRICS WORKLOAD • • Volumes • Calls/Cases per customer per month • Number of registered users/ Total number of users • Time spent contacting users • Time spent on change related incidents/requests INDIVIDUALS • Number of calls taken • Average Handle Time (AHT) • Avai labi l i ty • Occupancy • Number of incidents/requests c l o s e d o n f i r s t contact • Customer Satisfaction • Contribution to Knowledge base CUSTOMERS • Customer Satisfaction • Frequency of surveying, Number not responding • Volumes • Calls/Case
- 49. SERVICE DESK METRICS WORKLOAD • • Volumes • Calls/Cases per customer per month • Number of registered users/ Total number of users • Time spent contacting users • Time spent on change related incidents/requests INDIVIDUALS • Number of calls taken • Average Handle Time (AHT) • Avai labi l i ty • Occupancy • Number of incidents/requests c l o s e d o n f i r s t contact • Customer Satisfaction • Contribution to Knowledge base CUSTOMERS • Customer Satisfaction • Frequency of surveying, Number not responding • Volumes • Calls/Case RESPONSE • Average Speed to Answer (ASA) • % calls answered live vs.. queued • Call back time • Abandon Rate (ABA) • Responses within Service Level & Outside service l e v e l
- 50. INCIDENT, REQUEST AND ACCESS MANAGEMENT
- 51. INCIDENT, REQUEST AND ACCESS MANAGEMENT RESOLUTION • • Incident closure ( f rom time of submission) • Mean Time for Service Restoration (MTSR) for Levels 1 , 2 , & 3 • Incidents matched (KE) • Incidents Re- Opened • Closed First Contact • Escalations for resolut ion • Remote tool u t i l i z a t i o n • Desk-side visits • Incidents closed v i a s e l f - h e l p
- 52. INCIDENT, REQUEST AND ACCESS MANAGEMENT RESOLUTION • • Incident closure ( f rom time of submission) • Mean Time for Service Restoration (MTSR) for Levels 1 , 2 , & 3 • Incidents matched (KE) • Incidents Re- Opened • Closed First Contact • Escalations for resolut ion • Remote tool u t i l i z a t i o n • Desk-side visits • Incidents closed v i a s e l f - h e l p VOLUME • Total number of incidents/requests ( b y p r i o r i t y & category) ! • Security related incidents
- 53. INCIDENT, REQUEST AND ACCESS MANAGEMENT RESOLUTION • • Incident closure ( f rom time of submission) • Mean Time for Service Restoration (MTSR) for Levels 1 , 2 , & 3 • Incidents matched (KE) • Incidents Re- Opened • Closed First Contact • Escalations for resolut ion • Remote tool u t i l i z a t i o n • Desk-side visits • Incidents closed v i a s e l f - h e l p VOLUME • Total number of incidents/requests ( b y p r i o r i t y & category) ! • Security related incidents RESPONSE TIME • Service Desk performance ! • Level 2/3 – same as SD metrics
- 54. INCIDENT, REQUEST AND ACCESS MANAGEMENT RESOLUTION • • Incident closure ( f rom time of submission) • Mean Time for Service Restoration (MTSR) for Levels 1 , 2 , & 3 • Incidents matched (KE) • Incidents Re- Opened • Closed First Contact • Escalations for resolut ion • Remote tool u t i l i z a t i o n • Desk-side visits • Incidents closed v i a s e l f - h e l p VOLUME • Total number of incidents/requests ( b y p r i o r i t y & category) ! • Security related incidents RESPONSE TIME • Service Desk performance ! • Level 2/3 – same as SD metrics ESCALATION • Time to escalate ! • % Escalated to correct group ! • Technical & Hierarchical
- 55. INCIDENT, REQUEST AND ACCESS MANAGEMENT
- 56. INCIDENT, REQUEST AND ACCESS MANAGEMENT CUSTOMER SATISFACTION • Incident, Request & Access Management processes • SELF-SERVICE • Number of unique users ! • Average t ime per user ! • # pages viewed
- 57. Methods to Report Performance
- 58. Methods to Report Performance PERFORMANCE REPORTS BALANCED SCORECARD SUPPORT SCORECARD
- 59. Factors to Consider When Reporting
- 60. Factors to Consider When Reporting • Who are the stakeholders? • How does what you are reporting impact the stakeholders? • Reports must be easy to read and understood, thus they need to be developed with the stakeholder in mind. • Reports need to show how the support center is contributing to the goals of each stakeholder and the business. • Reports must identify the appropriate channels to communicate with each of the stakeholders.
- 65. Present your achievements with scorecards
- 66. Scorecards Drive Transformation I Relevant High (Business Low (Process Improvement) Moderate Transformation) to IT H General Specific T ISO/IEC 17799 ITIL/ISO/IEC 20000 C COBIT Six Sigma ISO/IEC 9000 Malcolm Baldrige Award Scorecards Standards help us to ensure that IT is aligned to meet business objectives
- 67. • Simple indicator • Reference base • Measures the key issues • Reports on progress to goals @ITSMNinja Support Scorecard
- 68. Scorecard Criteria: Operational Performance COLUMN TITLE COLUMN TITLE COLUMN TITLE Overall performance Response time Resolution time Closed first call Abandon time Wait time Status time Backlog aging
- 69. Scorecard Criteria: Operational Performance COLUMN TITLE COLUMN TITLE COLUMN TITLE Overall performance Response time Resolution time Closed first call Abandon time Wait time Status time Backlog aging
- 70. Scorecard Criteria: Operational Performance SERVICE SCORECARD The Acme Support Center Scorecard provides a weekly report of performance on our Service CS Overall Support Center Product Level commitments. Key Service Area Goal Actual Response Time Front-line Measurements: • Call Pick-Up Time - All incoming calls are answered by a support consultant • Call Waiting Time - Average is less that 3 minutes • Back-line Measurements: • Non-Accepted Call Back Time - All customers not responded to on the initial call by a support consultant will be called back within 30 minutes. 80% 3 minutes 90% Resolve Time Resolved on First Contact - 30% resolved first call (4 month goal is 50%) Resolved Same Day - 40% resolved within 1 business day (4 month goal is 60%) Resolved Same Week - 85% resolved within 5 business days 30% 40% 85% Status Priority 1 Issues - Customer provided status every 4 hours until resolved Priority 2 Issues - Customer provided status every 24 hours until resolved or workaround provided Call Aging - Manage backlog so that no more than 20% over 2 weeks and 5% over 30 days 80% 80% 80% Backlog (Average age of open items) 3 days 10 days Event Survey Overall satisfaction rating on a 1-5 scale 4.1 COLUMN TITLE
- 71. Service Desk Scorecard COLUMN TITLE Overall performance Response time Resolution time Closed first call Abandon time Wait time Status time Backlog aging COLUMN TITLE Customer sat/quality Overall satisfaction Response satisfaction Resolution satisfaction Status satisfaction Improvement goals Alignment to business
- 72. Acme Service Desk Scorecard
- 73. Balanced Scorecard Business Goals Financial Perspective COLUMN TITLE Provide a good return on investment on IT-enabled business investments Manage IT-related business risks Improve corporate governance and transparency Customer Perspective Improve customer orientation and service Offer competitive products and services Establish service continuity and availability Create agility in responding to changing business requirements Achieve the cost optimization of service delivery Obtain reliable and useful information for strategic decision making Internal Perspective Improve and maintain business process functionality Lower process costs Provide compliance with external laws, regulations and contracts Provide compliance with internal policies Manage business change Improve and maintain operational and staff productivity Learning and Growth Perspective Manage product and business innovation Acquire and maintain skilled and motivated personnel
- 74. IT Balances Scorecard @ITSMNinja Source: ITIL Continuous Service Improvement
- 76. You cannot manage what you cannot CONTROL You cannot control what you cannot MEASURE You cannot measure what you cannot DEFINE
- 77. What do you need to measure? What should you be doing you do with the metrics you produce?
- 78. Thank you! Join me tomorrow @2PM to learn Knowledge Centered Support (KCS) – The Methodology that Really Works John Custy • Managing Consultant • JPC Group • @ITSMNinja