OOW16 - Oracle E-Business Suite 12 Upgrade Experience for a 14 TB Oracle E-Business Suite Instance [CON4913]
- 1. 1 9/15/2016 GE Healthcare R12 Upgrade Overview - Umesh Date ( Tech. Program Mgr.- R12 Upgrade)
- 2. GE HEALTHCARE TODAY • $18 billion: global business unit of GE • 46,000: number of employees worldwide • $1 billion+ per year investment in R&D • 35,800 patents filed by GE since 2000 • Core strengths in bio-sciences, medical imaging & information technologies • Headquartered in Chicago, USA
- 3. 3 9/15/2016 BROAD SOLUTIONS FOR HEALTHCARE Diagnostic and Interventional Imaging Patient Monitoring Maternal and Infant Care Anesthesia and Respiratory Care Diagnostic Cardiology Healthcare IT Life Sciences Healthcare Consulting Global Services
- 4. CUSTOMERS OUR KEY CUSTOMERS INCLUDE: • Healthcare systems, hospitals, and clinics – Physicians – Clinicians – Nurses – Executives and administrators – IT managers – Biomedical engineers – Construction managers • Government officials • Pharmaceutical firms • Genetics researchers • Bio-science researchers
- 5. 5 9/15/2016 GE Healthcare ERP Context 4 Pillar instances – SAP, PS & EBS ( 2) Largest EBS instance upgraded to 12.2.4 80% of revenue transacted 53 countries Approx. 10000 users DB Size approx. 14 TB 150+ companies Upgraded from 11.5.10 to 12.2.4
- 7. Upgrade vs Re-implement R12 Technical Upgrade R12 Upgrade Plus R12 Reimplementation Shortest Tech upgrade to R12.2 • Customization remediation • Integration and Bolt-on checks • Regression & Performance Testing • New Features: SLA, inv org, multi- org, role- based access controls • Retire Customizations: • Convert GL to Corporate COA New R12.2 Instance • Rationalize customizations • Leverage new R12 features • Potential for Phased Roll-out Lowest Lowest Supportability Real-time Patching Timeline Cost Funct. Resource Needs Business Benefits New Features Retire Customizations Corporate COAs + + Medium - Long Longest Medium - High Highest Medium - High Highest Supportability+ Re-engineering Technical Upgrade allowed for other business priorities. Incremental Change in Business process instead of Big Bang
- 8. 8 9/15/2016 Initial Project Timeline Code Remediation 2014 2015 Q3 Q4 Q1 Q2 CoE Tech Steady increase in program resourcing needs over time Global Teams Core Team Planned Timeline Resources Leveraging a small core team to maximize value R12 Upgrade Testing
- 9. 9 9/15/2016 Global Functions SVCGSC FIN Commercial ENG IT Technical / Quality Team Global Management Team Senior Leadership Team Program Management Region / P&L Region/ Modality CIOs Functional Program Manager VP- Service VP-IT VP- Supply Chain CFO CTO Regional GMs Supply chain GSC Reginal GM Fin FIN Sales GM C.E. GM Engg ENG CIO Fin Leader IT Fin Leader GSC Leader IT GSC Leader Service Leader IT SVC Leader ENG Leader IT Eng Leader Bus Integration Business Integration Leader Region / P&L Integration Leader IT Program Leader Quality & Compliance Leader Commercial Ex Leader Comm. IT Leader Program Organization SVC Service GMS Technical Program Manager
- 10. 10 9/15/2016 Team Functional IT Senior Leadership Team (SLT) • Top Down Support – CAP / Communication • Strategic Direction • Help with staffing engagement • Top Down Support – CAP / Communication • Strategic Direction • Help with staffing engagement Guidance Management Team (GMT) • Drive func leadership selection • Monitor progress, provide help needed • Authorize major project variances • Manage conflicting priorities • Assist with functional staffing • Make the final Go / No-Go decision • CIO engagement of functional leadership • Drive IT leadership selection • Approve budget changes • Resolve cross program issues • Monitor progress, provide help needed • Make the final Go / No-Go decision Overall Program Executive • Manages SLT engagement • Organize and Lead GMT Sessions • Develops overall program strategy & organization • Manages top-level risk mitigation Program Managers • Organizes the overall program and mobilizes teams • Drives overall program operating mechanisms, issues, and risk mitigation • Manages program Plan, risk mitigation, tollgates • Manages program budget • Manages communication plan Business Integration Leader • Creates Overall Program Test strategy • Establishes and drives test reporting • Drives cross functional integration • Identifies simplification opportunities for future releases Region Integration Leader • Drives regional engagement through best practices and common work structure • Pivots between program, global processes and region / P&L leaders • Plans the work for the regional leads and monitors the overall region plan • Identifies top regional issues and drives to closure • Leads commercial staffing, supports functional staffing Top Management Support Staff Leaders Make Go/No- go Leadership Engagement Mobilize & Orchestrate Cross- functional Consistency & Integration Lead Region Engagement Roles & Responsibilities
- 11. 11 9/15/2016 Team Functional IT Global Process Leader • Recommend Go / No-Go • Assure functional coverage & obtains resources • Validate functional test plan • Escalates critical risks • Organizational readiness • Identifies process eco-system • Develop staffing coverage strategy and resource plan • Staffs GE & contractor positions • Drives issue resolution • Liaison's to function Global Track Leader • Defines critical processes • Defines test scenarios • Manages test execution • Escalates critical issues • Identifies testing coverage for customizations & interfaces • Gathers data for testing • Drives fixing defects • Drives system documentation maintenance Regional Leader • Engages regional leadership and commercial teams for testing & readiness • Identifies local system integrations, country localizations, and CFD’s • Coordinates regional testing as needed – particularly where local tools are involved • Identifies regional risks and issues and drives to closure • Define the region go-live readiness (acceptance) criteria IT Technical Program Leader • N/A • Plans & manages outage window execution • Performs stress test & tunes system • Governs customization designs • Drives Overall eco-system connectivity • Manages validation & compliance • Leads central break/fix for Oracle defects • Identify & modify BI fixes Organizational Readiness Process & Module Readiness Run the business… Find the Bugs! Run the Systems… Fix the Bugs! Assure Regional Readiness Upgrade Execution & System Performance Roles & Responsibilities
- 12. 12 9/15/2016 2014 2015 2016 Actual Program Timeline NovOct Nov Dec Jan Feb Mar Apr May DecJun Jul Sep OctAug Go Live Jan CRP1 CRP2 UAT Validation Test Development End to End Process Testing User Acceptance Test Compliance documentation Training & Communication Defect Resolution Defect Resolution Key Activities: CRP1 – Code remediation and functionality exploration CRP2 – End to end testing and Bug fixes UAT – User acceptance testing Upgrade simulations to optimize the time Back out planning and Cutover planning Simulate Back out plans. Communication & Training Go-Live 6 Upgrade Simulations
- 13. 13 9/15/2016 Upgrade completed on time – system came up 6pm Sunday as planned Critical Business transactions exceeded daily averages (starting day 1 ) No post go-live outages Successful January Close Program Go-Live Highlights Post Go-Live Executive Summary “Just an awesome, awesome job!!!!” ~ CFO GEHC “Best upgrade I have seen in 20 Years” ~CFO SC P&L “As close to perfect as you could expect in an ERP upgrade” ~ VP & CIO GEHC
- 15. 15 9/15/2016 Key Learnings ERP is not standalone 350 Interfaces ;Inbound and outbound 120 connecting systems 3 BI tools Transactions flow from Source to ERP to destination systems Code remediation and testing done across systems …… But… How do we shutdown this web of systems? Key is to understand dependencies and cross-impacts How do we startup this web of systems? A pilot can fly alone easily but when 15 pilots fly together, it’s a different experience…. That’s how it feels like in the ecosystem !! Think about your Ecosystem; not just the ERP
- 16. 16 9/15/2016 Key Learnings Use of tools to remediate code Automation of code remediation saved us months in the project schedule Code remediation activity could be run like a factory Total number of code Objects remediated ~ 2300 …… But… We found that like all software, our tools had issues.. We missed code in ecosystem, missed SSWA objects, missed embedded queries in apps like QA, Pricing etc. Online patching compliance Trust your tools but verify the results !
- 17. 17 9/15/2016 Key Learnings Test plan and Test coverage We executed 13000 test scenarios in CRP’s Fantastic functional team engagement Defect monitoring and root cause analysis helped Exited UAT with 12 defects across all tracks. Load testing and Regression Testing …… But… Initially we did not test transactions which start in 11i but end in R12. Practice period close. Get used to period close analyzers. Our root cause analysis did not look at underlying trends and systemic defects…. We improved on that. Start your Load testing early. If using i-modules, please test them from outside the firewall. Test boundary conditions & exceptions !
- 18. 18 9/15/2016 Key Learnings Appreciate your team Any Oracle upgrade is very demanding on people Work hours will be long and often at odd hours Workload for different teams like developers, middleware team, testers, users will be fluctuating. Workload for DBA’s tends to be high all the times. Keep morale up. Take pride in milestone accomplishments and learn from every incident. This is team work. Trust is key. Help in building it …… But… Think about back ups Think about shifting work hours Think if you run a risk of knowledge concentration in a few Think about your communication plan across the globe if you are global Remember DBA’s are the centerpiece of this project
- 19. 19 9/15/2016 Key Learnings Functional Users Need a Strong buy-in. Knowledgeable super users are the key to success Users need to test the system as they use it everyday Reports, labels are the key Spend ample time in communications, awareness and training. Posters and banners at major hubs and Mfg/ Dist sites. Think about having a dedicated communications leader Involve them in sanity testing during simulations Use your super users as your messengers and damage controllers in case of small hick ups. Think about your communication plan across the globe if you are global Project success = Happy Users
- 20. 20 9/15/2016 Key Learnings Upgrade Considerations from a DBA perspective Archive or Purge unwanted data Refresh all MVs to reduce upgrade time Concentrate on long running jobs Parallelize tasks if possible Clean up database objects before enabling ADOP (duplicate tables etc.) Implement SPMs and SQL profiles for long runners …… But… Do end to end simulations Plan for backups before critical steps Check logs always for any unexpected errors Follow the upgrade manual from Oracle. No Short cuts please !!
- 21. 21 9/15/2016 Key Learnings Cutover planning & execution We had a 1900 line project plan just for the cutover tasks Understand the dependencies of tasks. Understand dependencies of people. They need to trust each other. Handshakes on tasks are important Co-location helps a lot Simulate like it is Production. Avoid the temptation of shortcuts. …… But… Note every single incident for investigation later Think user lock down is simple…it might not be. Plan your simulations for every single failure. Don’t take things for granted. Practice, practice and practice….
- 22. 22 9/15/2016 R12 Upgrade Outage plan Provide high quality instance as efficiently as possible What actions were taken? • Practiced and improvised the shutdown process. • Created save points which were key to BCP • Worked with Oracle to get best practices • Oracle fix was practiced and verified • Conducted multiple simulations to iron out the process • Added time for GS and introduced a functional checkout process Save Points What Did We Learn? Go/No Go Monday Tuesday Wednesday Thursday Friday Saturday Sunday 24-Jan Monday 25-Jan R12.2 Code, Patches Check Out System shutdown R12.2.4/Online Patching 3rd Party, M/W • Graceful shutdown adds 12 hours (flushing data through system) • Issues need to be recreated. Painful. • VCP first collection was slow • Issues with R12.2.driver in the first 6 months vs the last 6 months • Code migration, VCP collection, Sabrix install and Brazil localizations took 42 hrs. • System startup was not smooth VCP What were the outcomes? • Finally reduced it for 7 hrs. • Gained ability to run multiple simulations at same time. • VCP collections was not a bottleneck • Reduced the time for the last phase to 27 hrs. from 42 hrs. • Resulted in a very high user confidence Practice to optimize the plan and understand dependencies Outage at ~5 Days…additional time needed to ensure Quality
- 23. 23 9/15/2016 Key Learnings Where is your Plan B? We had a 300 line back out plan Plan for back out from every step of the upgrade Playbook for executive engagement in the worst case Have Plan B incase of delays. Understand steps where you could lose time and where you could gain time Use the time gained wisely. Try to stay ahead and be ready for contingencies. Have playbooks for concurrent activities going out of sequence. Practice your back out plans and Plan B’s to make sure they can work. Your Back out plans should cover the ecosystem not just the ERP. Have a Plan B for every possible thing you can think about.
- 24. 24 9/15/2016 Key Learnings System Performance Comprehensive load testing Verified performance between 11i & R12 up to the second API and Interface performance is important too. Watch out for first collection run for ASCP. …… But… Watch out for performance post-upgrade Allow time for post upgrade steps to improve performance Simulate post upgrade performance to take corrective actions Productions system will be faster than your test environments Gather Stats and Table re-orgs will help. Ask your DBA’s to plan for those pre and post Upgrade Plan for post-upgrade performance. Don’t make it an after thought !
- 25. 25 9/15/2016 Key Learnings Post Go-live support Engage Production support teams during UAT Pair the project SME’s with the support SME’s Be ready for a high call volume Have regional contact points by defect types for level 1 support If you are changing java versions, be ready for login Issues. Over communicate about go-live . Communicate not only during the project but even after go-live. Be ready with swat teams especially for finance. All the best for your upgrade & hope that our learnings will make you smarter.
- 26. 26 9/15/2016
- 27. 27 9/15/2016 Operating Mechanism Purpose / Objective Participants Frequency Senior Leadership & GMT Review Inform corporate officers of project status, issues, and help needed Senior Leadership Team, aligned GMT members, selected team members, Global functional & IT leads, Gilles, Jason Monthly Program Leadership Forum Strategic review, develop recovery plans, make course corrections Program management, Global functions, Gilles, Region Leaders, Jason Monthly Program Review Review scorecard status at end of week; weekly status report preparation Program management, Global functions, Gilles, Jason Weekly Global functional Review Manage track progress, issue escalation & mitigation, update on program information Global functions, track leads, region integration leader, break/fix manager Weekly Region Leadership Review Manage region progress, issue escalation & mitigation, update on program information Region leader, Gilles, Brian, Optional: Functional leads Weekly Training & Improvement Forum Add / update training & improvement lists Functional Leads, Gilles, Brian, Architecture leader Bi-Weekly IT Technical Integration Session Manage IT Platform progress, issue escalation & mitigation, dependencies IT Technical Team Weekly Defect Management Review Defect assignment, status, critical issues by module Defect manager, IT track leads, BA COE architects, BI COE, GAMS Weekly Project Plan Review Progress vs plan update on activities Program management, Gilles, Jason Weekly Operating Mechanisms