The experiences of migrating a large scale, high performance healthcare network
- 1. The experiences of migrating a large scale, high performance healthcare network Larry Williams Corporate Manager, Partners HealthCare
- 2. In the next half hour… Partners Healthcare System overview Caché platform architecture & metrics The need to migrate Phased migration approach Benchmark testing and results Discoveries and production enhancements Post database migration results Future milestones
- 3. Partners Healthcare System Founded in 1994 Brigham & Women’s Hospital Massachusetts General Hospital Now includes: Community physician network (1200 + 3500 MD’s) PCHi 3 community hospitals 2 rehab hospitals 3 specialty institutions Enterprise-wide Information Systems 1100 employees Annual budget FY05 approximately $160 million
- 4. Anchor Hospitals & Airport BWH MGH Logan Airport 10 km 6 km
- 5. Acute Care Hospitals MGH BWH Newton- Wellesley Community Physician Practices
- 6. Partners Domain Devices Internet 12,000 Printers 32,000 Desktops Firewall ~30,000 other devices 1,450 Servers Closely Managed Assumed Managed
- 7. Current Mixed-Mode Production Architecture
- 8. Enterprise Integration Over 30% are to and from Caché database Change from prior year Daily Average Est. Annual Transactions # of Interfaces 196 170 192 167 37% 4,659,035 1,330,962,017 2007 40% 3,399,211 1,240,712,044 2006 45% 2,431,917 887,649,802 2005 1,673,515 610,833,080 2004
- 10. Annual Database Growth Rate
- 11. Database Utilization Average Database References per day in Billions
- 12. The Need to Migrate - Availability Monthly Downtime Current State Business need
- 13. Additional Business Requirements Increase availability and reliability Decrease database risk from 5 single points of failure More robust hardware and OS Many less servers and OS instances to manage Clustering and automated failover Reduce monthly maintenance needs, updates once or twice per year -------------------------------------------------------- Improve Performance 64 bit OS, more memory for cache Caché 5.0.20 to Caché 2008.1, significantly improved ECP performance Increase Scalability 91 Terabytes available on EMC SAN DMX3 On-demand addition of processor cores
- 14. Caché Migration Decision Making Process Only considered first tier vendors and support (IBM, HP) HP assumed much more risk with Professional Services Existing HP business yields more leverage & visibility with regional office More headroom in HP configuration Price was not a distinguishing factor
- 15. Phased migration approach Proof of Concept (benchmark testing) Completed 10/15/07 Phase 1 – Database tier Completed 4/14/08 Phase 2 – Application tier Big Bang migration 12/14/08 (includes Cache 2008.1 UNIX platform upgrade) Phase 3 – Disaster Recovery Now implemented as part of Phase 2, leveraging HP Blade servers
- 16. UNIX DataBase Tier Benchmark Environment
- 17. UNIX Application Tier Benchmark Environment
- 18. Benchmark Load Testing Results Goals Simulate current Production user counts & transaction loads Verify support for load increases up to 3x Benchmark Environment Isolated LAN, new DMX3 SAN 20 new Windows blade servers (10 app servers, 10 script ‘players’) Scripts for 8 apps (represent heaviest use, Web/Telnet/VB apps) 2 batch jobs (screensaver simulation, NullGen LMR functions) Conclusions Able to simulate production load, 1.5x and 3x load 2 HP rx8640 can handle growth projections 0.66 0.15 0.32 LMR avg Caché app time (in sec.) 40,000 40,000 11,806 LMR transactions (5 min. period) 135,000 30,000 35,000 Database Global Refs / sec. Benchmark full script load Benchmark “paced” script load Production peak (8/21, 11:20 am) Metric
- 19. Design and Configuration Considerations Database configuration simulation testing 1 to 5 Caché database instances were assessed 1 vs. 5 ECP channels per Caché instance were assessed Number of active cores were accessed (4 active, 2 reserved) Results and unexpected discoveries Identify 5 Caché database instance as optimal design configuration Maintain same data distribution across 5 DB instances Journal synch bottleneck the biggest issue High Transaction Journal deamon maintains ECP durability to guarantee transaction (1 per Caché instance) Determine 1 ECP channel per instance optimal Additional channels did not improve throughput, still have only 1 Journal Deamon
- 20. Benchmark Discoveries Led to Production Improvements (the rules have changed) References to Undefined globals using $Data and $Get These commands require network round trip Use of $increment Each call to $I requires network round trip Excessive use of Cache locks Forces more than 1 round trip Use of large strings Strings that require more than 3900–4000 bytes to represent the string value are big strings and never cached on the ECP client. Lesson Learned - Each trip to the database server results in overhead caused by a Journal Synch. Increasing the Journal Synch rate causes bottlenecks in the ECP channel which increase the risk of long transactions.
- 21. Post UNIX DB migration - 50% Reduction in Outlier Transaction
- 22. Monthly Average Caché Web Transaction Time
- 23. Unprecedented Growth - The LMR
- 25. Parallel Migration Testing and Certification Effort DEV VC/m DEV QA PROD QA auto copy on promote promote check-out & check-in Developer test Analyst cert Production release promote auto copy on check-in Caché Environment 5.0 (WIN) 2008.1 (UNIX)
- 26. Application Models Old New Browser client Web server Cache Cache VB client .Net server Cache Cache .Net client Browser client Web server Cache Web Services Browser client .Net client Scalability/Connection pooling, robustness/error handling, Vism Managed Provider Vism.ocx (MServices) Managed Provider Cache Web Services WebLink
- 27. The experiences of migrating a large scale, high performance healthcare network Larry Williams Corporate Manager, Partners HealthCare