Effectively managing project performance reporting
1 like•1,564 views
This white paper provides guidance for project teams on managing performance reporting for IT support and maintenance projects, focusing on essential performance measures to track progress, product quality, and customer satisfaction. It illustrates a case study involving Mindtree's collaboration with a Fortune 500 bank, emphasizing the establishment of common performance reporting processes and metrics to enhance accountability and efficiency. The document outlines the PEMARI framework for measurement planning, performance analysis, and continuous improvement, ensuring that project metrics are accurately defined and communicated.
Effectively managing project performance reporting
- 1. WHITE PAPER Effectively managing project performance reporting.
- 2. Summary This white paper helps project teams identify performance measures for Information Technology (IT) support and maintenance projects, which collect and report data to help track and assess project progress, product quality, project success and customer satisfaction of a project. An effective set of performance measures provide actionable information on a focused set of metrics. This in turn provides a balanced view of project performance that can be used to improve the project management process. Performance measures also provide for accountability by laying out what is expected, when it is expected and what action will be taken if planned achievements do not occur. The white paper highlights how Mindtree partnered with a Fortune 500 bank where the basic metrics of the team’s efficiency was measured and published. The Mindtree’s data warehouse team demonstrated how they established a common project performance reporting process between Mindtree and the customer’s IT team. They published the comparative project efficiency and team productivity metrics on a periodic basis. Contents 1 Performance reporting 03 2 Performance reporting methodology – PEMARI 03 3 Mindtree – Banking and Financial Services (BFS) project case study 03 3.1. Measurement planning 04 3.2 Establishing and updating measures 05 3.3 Measuring performance 05 3.4 Analyzing data 05 3.5 Performance reporting 10 3.6 Continuous improvement 10 4 Summary 12 White paper 02
- 3. 1. Performance reporting 2. Performance reporting methodology – PEMARI It’s difficult to manage what you can’t measure and PEMARI (stands for measurement planning, establishing document. Every project needs to answer certain questions and updating measures, measuring performance, analyzing like, “Is the project successful? Is the new process data, performance reporting and continuous improvement) working? Is the project team efficient? Is the outsourcing is a project measurement framework which systematically cost effective?” In order to answer these questions, it is approaches performance through an ongoing process important to have actual data which can back up the story. of establishing metrics, collecting, analyzing, synthesizing, Hence there needs to be a mechanism where the project is reviewing and reporting performance data (fig.01). measured for performance and adds value to the organization. Each stage of PEMARI has to be thought with respect to the specifications of the project requirements of Benefits of performance reporting performance measurement and reporting in order to make 1. Customer can fine-tune efforts based on actual it the most effective solution. performance by targeting areas of low performance, in areas of quantity, quality, cost or target solutions. 3. Mindtree – Banking and Financial Services (BFS) 2. Customer can enter into service level agreements project case study reasonably sure of meeting the service levels. Project Description 3. The customer can quantify and prove his team’s value to Mindtree started the journey towards building a partnership his organization and therefore, justify asking for with US based fortune 500 commercial banks providing more resources. a diverse spectrum of services – support, user requests, change requests and enhancements. From the customer perspective the focus of this engagement was on round the clock productivity benefits, value added insights, faster and more flexible setup of the offshore team. The range of services Mindtree provided is explained in fig. 02. Fig.01: Performance reporting methodology _ PEMARI Measurement planning Establishing & updating measures Measuring performance ƒƒ Identify the performance ƒƒ Develop a list of ƒƒ Identify data source measurement team potential measures ƒƒ Plan for data collection ƒƒ Identify project areas to be ƒƒ Plan for data collection ƒƒ Communicate to data source measured ƒƒ Communicate to data source what is expected of them ƒƒ Identify the tools and what is expected of them ƒƒ Collect data for analysis technologies ƒƒ Collect data for analysis ƒƒ Ensure data quality ƒƒ Ensure data quality Analyzing data Performance reporting Continuous improvement ƒƒ Analyzing of data ƒƒ Develop a communication ƒƒ Review and revise the target ƒƒ Calculate the metric values plan defining metrics value ƒƒ Analyse and validate results ƒƒ Events ƒƒ Learn from feedback ƒƒ Perform benchmarking and ƒƒ Target audience ƒƒ Formally collect comparative analysis ƒƒ Message lessons learned ƒƒ Objective ƒƒ Target action items to achieve ƒƒ Impact target set ƒƒ Comment ƒƒ Share results with stakeholders White paper 03
- 4. Fig. 02: Range of services provided by MIndtree Scope of services: Support 8X5; DW uptime 7 AM PST Support time critical DW & 700 + BI users Data load User service Analytical data State BI Downstream Enhancements support requests marts reports applications Data load support Complexity & critical Data storage : Oracle 10g, MA SLA for DW uptime : 7 AM PST tools access, Flat files User dependency : More than 700 Warehousing : Business objects data business users tools integrator, P / SQL, Technicality : More than 60 RTL jobs, 15 Essbase, Sagent universes, 300 reports Analytics : Business objects XI, Integration level : More than 25 tools Webl, Excel upstream applications Process : Project map vision’s Dependency : More than 25 and documentation, upstream applications project tool kits Publishing tools : Infoburst (documentation / reports), MS Power Point, .Net applications 3.1 Measurement planning 3) Project estimation template – A common project The Mindtree team is measured on the performance estimation template has been built and agreed upon between of the various deliverables as listed below: Mindtree and customer IT team to arrive at the project ƒƒ Production support work breakdown structure, deliverables and estimated ƒƒ Resolving user requests (tickets) effort. With this, both the teams can make sure that the ƒƒ Working on enhancements and projects estimations done at any point in time for any project will be Mindtree approach for measuring team efficiency as same. The sample of splitting the deliverables / outputs for compared to the customer is explained through the a project is shown in Fig. 05. block diagram. Fig. 03. 4) Project estimation mechanism – The project estimation mechanism is to arrive at the list of deliverable for In order to make sure that the efficiency comparison each project. Every deliverable estimated in the estimation between Mindtree and customer team is base-lined on the sheet will be assigned with the effort unit, complexity and common ground, Mindtree defined common tools confidence level. to support metrics calculations. Some of those are 5) Project complexity definition – Mindtree and client listed below. agreed upon the common complexity definition for different type of deliverables across various technologies. 1) Ticket complexity – The ticket complexity code and The table (Fig. 06) has been used to define the complexity the complexity level has been arrived after studying level for every different kind of deliverables those are the six months history of tickets. The sample is attached available in the project currently. in Fig. 04. 6) Project sizing mechanism – The complexity of project 2) Ticket assignment SPC (Single Point of Contact) – The deliverables have been arrived at based on the Mindtree assignment of ticket has to be done by SPC for the entire standard followed in the data warehouse practice team making sure the equal distribution is made. for arriving at the size of the deliverable. Fig. 07 explains about the size calculation. White paper 04
- 5. 3.2 Establishing and updating measures ƒƒ Calculate the metrics values for the Mindtree defined and established the metrics based on the pre-determined metrics measurement criteria. Some of the key metrics have been ƒƒ Validate the results through the Mindtree quality explained in Fig. 08. function team to ensure all the calculations and data are correct (Fig. 09) 3.3 Measuring performance Performance measurement is based on the availability of Sample metrics calculation the artifacts below: A sample metrics called “Average Ticket Cycle Time” and the ƒƒ Data sources – ticketing tool, effort logging tool, calculation method has been provided below. project / CR plan, tier 4 escalation log, tickets design and code review log Step 1: Extract data from Ticket Logging Tool into a work ƒƒ Data collection – Monthly dump of all the data book for the current month sources mentioned above. Mindtree derives the data Step 2: Cycle time for a ticket = Ticket closed date – Ticket for the tickets based on the static report developed created date for the same. Some of the data has to be manually Step 3: Average of Cycle Time for all tickets integrated together Step 4: Carve out the trend of the “Average Ticket Cycle Time” for both the Mindtree and customer 3.4 Analyzing data IT team Below are the steps involved in data analysis process: Step 5: Build the chart for the above trend ƒƒ Analyze the data to make sure that all the required data is available in the required format to make the metrics calculation Fig.03: Mindtree approach for measuring team efficiency as compared to the customer Project efficiency calculation Customer size project Size of project Customer project Customer total Total effort (Hrs) throughput effort (Hrs) Mindtree Mindtree size project project Size of project Mindtree efficiency project Mindtree total Total effort (Hrs) throughput Mindtree efficiency effort (Hrs) Average (Mindtree project efficiency, Tickets efficiency calculation Mindtree tickets Customer # efficiency) of tickets # of tickets Customer tickets Customer total Total effort (Hrs) throughput effort (Hrs) Mindtree Mindtree # tickets of tickets # of tickets Mindtree efficiency tickets Mindtree total Total effort (Hrs) throughput effort (Hrs) White paper 05
- 6. Fig. 04. Ticket complexity – sample Complexity Ticket Ticket type Complexity level code category 01 BI Webi reports scheduling request Simple 02 BI Desk reports scheduling request Medium 03 BI Creating new reports Complex 04 BI Universe changes with direct object changes Simple 05 BI Universal changes with filter Medium 06 BI Universal changes with join or context changes Complex 07 DW Retriggering product jobs Simple 08 DW BODI changes (adding or modifying straight columns) Medium 09 DW BODI plans (creating new plans) Complex Fig. 05. The sample of splitting the deliverables / outputs for a project Req. Req. Requirement summary D1 (Effort) D2 (Technical D2 (Confidence ID Sub ID complexity) level) 01 1.1 Interface requirement 3 X effort unit Medium >=80% 02 2.1 Design client interface 5 X effort unit Complex >=80% 03 3.1 Design for accounts and balances 5 X effort unit Complex >=80% 04 4.1 Design staging interface 5 X effort unit Complex >=80% 05 5.1 Design staging to data warehouse mapping 5 X effort unit Complex >=80% 06 6.1 Design monthly revenue 5 X effort unit Complex >=80% 07 7.1 Design periodic cash payment 5 X effort unit Complex >=80% White paper 06
- 7. Fig. 06. Complexity level for every different kind of deliverable across various technologies Tool name version Tool specific unit Featured distribution Complexity Size of measurement (units) Business XI Report <20 objects with simple formatting for Simple 1 Object all reports, no sections, simple straight report forward calculations, no prompts <40 objects, <20 objects formatting, <5 Medium 2 sections, multidata providers cross tab report, <10 derived objects, <2 charts, simple prompts, 1-2 derived table <40 objects with formatting, >5 sections, Simple 3 multi data providers, sub report drill down, <10 derived objects, >2 derived tables and customization effort Business XI Report <20 columns with simple formatting for all Simple 1 Objects web reports, no grouping, no derive tables <40 columns, <20 column formatting, <5 Medium groups, multiple commands, cross tab 2 report, <10 derived colums, <2 charts, simple parameters >40 columns with formatting, >5 groups, Complex multiple commands, sub report drill 3 down, >10 derived columns, >2 charts, security and special features (Ex, multi-lingual, cascading prompts etc.) and customization effort Business XI Universe <50 tables and simple joins, OLAP data Simple 1 Objects model, <5 derived objects, simple designer prompts, default hierarchy setup, no row level security, no loop or SQL traps, no derived tables, no aggregate awareness <10 tables and simple conditional joins, Medium 2 OLAP data model, <10 derived objects, simple prompts, customized hierarchy setup for drill down, row level security, <3 loops, No SQL traps, <2 derived tables, no aggregate awareness White paper 07
- 8. Business XI Universe >10 tables and complex / conditional joins, Complex 3 object OLTP data model, >10 derived objects, designer cascading prompts, customized hierarchy setup for drill down, row level security, >3 loops, availability of SQL traps, >2 derived tables, availability of aggregate awareness, special features (EX, multi- lingual implementation etc) BODI Data flow One-to-one mapping, <25 transformation Simple 1 rules, <2 look ups, no custom functions, <3 joins <50 transformation rules, <5 look ups, <3 Medium 2 custom functions, <5 joins >50 transformation, >5 look ups, <3 Complex 3 custom functions, <5 joins PL / SQL Oracle KLOC <500 LOC (including PL / SQL block, Simple 1 10g exception handling, comment), no analytical function, re-use existing logics <2000 LOC (including PL / SQL block, Medium exception handling, comment), simple analytical function, simple collections 2 <500 LOC (including PL / SQL block, Complex 3 exception handling, comment), complete analytical function and collections Impact DART <3 simple downstream, <10 ETL programs Simple 1 analysis <10 downstream, <30 ETL programs Medium 2 10 downstream, >30 ETL programs Complex 3 Data <3 simple downstream, <10 ETL programs Simple 1 loading for downstream <10 downstream, <30 ETL programs Medium 2 testing 10 downstreams, >30 ETL programs Complex 3 Requirements <3 simple downstream, <10 ETL programs Simple 1 analysis and design <10 downstream, <30 ETL programs Medium 2 10 downstreams, >30 ETL programs Complex 3 White paper 08
- 9. Fig. 07: Size calculation table DW practice benchmark – effort (PD) Simple Medium Complex BI 5 7 9 ETL 4 6 8 PL-SQL 3 5 8 DW practice benchmark – from effort benchmark (units) Simple Medium Complex BI 1.7 2.3 3.7 ETL 1.3 2.0 2.7 PL-SQL 0.0 1.7 2.7 1.0 2.0 2.8 Average 1 2 3 Fig. 08: Key metrics Service type Metrics name Measurement criteria / calculation formula Tickets Volume of over-all tickets Resolve equal amount of tickets as compared to customer IT team Volume of complex, Resolve equal amount of complex and medium tickets as compared to medium tickets customer IT team Cycle time for tickets Time duration between ticket created date and closed date Tickets DRE Number of tickets passed design review / total design reviews conducted Tickets efficiency Number of tickets / total effort spent Support number of tier 4 escalations No. of time Mindtree team escalated to customer for incident resolution (ideally zero) Projects / CR Support time Show reduction in time spent by Mindtree team in production support Schedule variance (Planned duration – actual duration) / planned duration Effort variance (Planned effort – actual effort) / planned effort White paper 09
- 10. 3.5 Performance reporting a) Visual display board – The visual display board is a Performance reporting is the final step in reporting mechanism to set the target for each individual on a project performance to the management, both at Mindtree monthly basis, based on the project / tickets pipeline and and to the customer. This primarily involves making the track it accordingly on almost near real time basis. This presentation with all the metrics values defined earlier is an intelligent dashboard which measures the tickets with the scorecard of the current month’s achievements resolution performance of the team at any point in time. It against the set targets. This presentation also talks about enables the team to re-plan, if needed, during the month in the trend of improvements that has been seen over the order to avoid surprises at the end of the tenure. period of time. b) Value stream analysis – The value stream analysis is A snapshot from the presentation has been shown in the methodology which can carve out the various stages fig. 10 which talks about the trending of the tickets of the process in order to capture the Cycle Time and resolution metrics. The data in the graphs below have been Value Time of each of the stages in that process and bring masked for the security reasons. out the difference between Cycle Time and Value Time. This methodology was applied to the Tickets resolution 3.6 Continuous improvement / projects execution process. As a result of this, the team The project and the underlined operations improvement could identify some of the stages in the process which can be achieved only by establishing the continuous really had a significant gap in Cycle Time and Value Time. improvement process. As part of this project the team has As a next level analysis the team got the reasons behind taken up certain measures in order to understand the gaps these gaps and then worked towards mitigating the gaps. in the existing processes and arrive at the steps for focusing This analysis is done on monthly basis in order to identify on the continuous improvements needed. Primarily the two the avenues of continuous improvement and has shown steps adopted by the team are listed as follows: tremendous results. The sample of Value Stream analysis that was carried out for one of the months is shown in Fig. 11. Fig. 09: Some data collected for metric calculations Summary White paper 10
- 11. Fig. 10: A snapshot from the presentation Efficiency - Trend MindTree Tickets Efficiency MindTree Projects Efficiency MindTree Ticket Efficiency MindTree Project Efficiency 10 4 3.77 3.60 8.21 7.59 3.5 Efficiency Number 8 Efficiency Number 3 5.71 6 2.5 2 2 2 2 1.96 4.08 1.75 2 4 2 2 2 2 1.5 2 1 0.5 0 0 Mar-09 Apr-09 May-09 Jun-09 Mar-09 Apr-09 May-09 Jun-09 Customer MindTree Customer MindTree MindTree Overall Efficiency Overall Efficiency Trend MindTree Overall Efficiency Overall Efficiency Trend 9 8.20 7 7.59 5.90 8 6 Efficiency Number 7 Efficiency Number 4.67 5.90 5.71 5 3.92 6 3.83 5 4.67 4.08 3.92 4 3.60 3.76 3.83 4 3 2 2 2 2 3 2 2 2 1.76 2 1.96 2 2 1 1 0 0 Mar-09 Apr-09 May-09 Jun-09 Mar-09 Apr-09 May-09 Jun-09 Customer MindTree Ticket Ef f iciency Customer MindTree MindTree Project Ef f iciency MindTree Overall Ef f iciency CONFIDENTIAL: For limited circulation only © 2009 MindTree Limited Slide 1 Fig. 11: The sample of Value Stream Analysis that was carried out for one of the months User Service Ticket logged Ticket assigned Ticket Req. SPC to ODC ODC Acceptance Request in Infradesk to DART Analysis CT=18 VT=13 CT=45 VT=10 CT=15 VT=1 CT=33 VT=2 CT=33 VT=36 Create release Dev. & Testing Create Design DART Impact Ticket Req. Design Review Pkg. for UAT (Dev Env) Doc Analysis Clarification CT=13 VT=13 CT=4100 VT=61 CT=60 VT=18 CT=100 VT=100 CT=38 VT=38 CT=600 VT=22 Deploy & Test Create release Deploy & Smoke Infoburst Close Ticket User Acceptance in UAT Pkg. for Prod. Test in Prod. Scheduling in Infradesk CT=34 VT=17 CT=75 VT=13 CT=300 VT=1 CT=15 VT=1 CT=230 VT=8 CT=138 VT=2 White paper 11
- 12. Mindtree established a common project performance reporting process by the publishing the comparative efficiency and productivity metrics on a periodic basis. Mindtree made expected improvements towards the target set by customer in the defined period. This not only enabled the customer to take an intelligent decision based on our productivity, they also realized the actual cost arbitrage and recognized the value added by Mindtree’s outsourcing team. About the author: Shubha Krishnamurthy is currently being employed in Mindtree as Program Director and has more than 14 years of IT experience in managing large scale Data Warehouse and Business Intelligence projects. She is currently managing delivery for all the Data Warehouse and Business Intelligence projects under banking and financial services and insurance industry group. Prior to Mindtree she has worked in Wipro Technologies and Patni Computers as software engineer developing data warehouse and business intelligence applications. She holds a B.E with specialization in Electronics and Computer Science stream from S.D.M Engineering College of Technology, Dharwad. White paper 12