Transitioning from Traditional DW to Apache® Spark™ in Operating Room Predictive Modeling
20 likes6,449 views
The document discusses the transition from traditional data warehousing to Apache Spark in predictive modeling for surgical operating room scheduling at Beth Israel Deaconess Medical Center. It highlights the importance of predictability in scheduling to optimize resource usage and reduce wait times for patients. Key insights include the need for cloud infrastructure, the advantages of Spark for data processing, and the importance of collaboration among data professionals.
Transitioning from Traditional DW to Apache® Spark™ in Operating Room Predictive Modeling
- 1. Transitioning from Traditional DW to Spark in OR Predictive Modeling Ayad Shammoutand Denny Lee October 21st,2015
- 2. About Ayad Shammout • Director of BusinessIntelligence,Beth IsraelDeaconess Medical Center • Helped build BusinessIntelligence, highlyavailable / disaster recoveryinfrastructure for BIDMC 2
- 3. About Denny Lee • TechnologyEvangelist, Databricks • Former Sr. Director of Data SciencesEng, Concur • Helped bring Hadoop onto Windows and Azure 3
- 4. We are Databricks, the company behind Spark Founded by the creators of Apache Spark in 2013 Share of Spark code contributed by Databricks in 2014 75% 4 Data Value Created Databricks on top of Spark to make big data simple.
- 5. Why is Operating Room Scheduling Predictive Modeling Important?
- 6. 6 $15-$20 / minute for a basic surgicalprocedure Time is an OR's most valuable resource Lack of OR availability means loss of patient OR efficiency differs depending on the OR staffing and allocation (8, 10, 13, or 16h), not the workload (i.e. cases)
- 7. 7 “You are notgoing to getthe elephantto shrink or change itssize. You need to face the fact that the elephantis8 OR tall and 11hrwide” Steven Shafer, MD
- 8. 8 Operating Room Better utilization = Better profit margins Reduce support and maintenance costs Medical Staff Better utilization = Better profit margins Better medical staff efficiencies= Better outcomes Patients Shorter wait times and lesscancellations Better medical staff efficiencies= Better outcomes
- 9. Develop Predictive Model • Develop a predictive model that would identify available OR time 15 business days in advance. • Allowus to confirm wait list cases two weeks in advance, instead of when the blocks normally release four days out. 9
- 10. Forecast OR Schedule • Case load 15 businessdays in advance • Book more cases weeks in advance to prevent under- utilization • Reduce staff overtime and idle time 10
- 11. Background • Threesurgical groups • GYN, urology, generalsurgery,colorectal, surgical oncology • Eyes, plastics, ENT • Orthopedics, podiatry • Currentlybuilt using SQL ServerData Mining 11
- 12. Using Traditional Data Warehousing Techniques
- 13. OR DW SSAS Data Mining Data Sources OR Reports Traditional Data Warehousing & Data Mining OR Predictive Model Process mining model every3 hours OR Prediction DB Data inserts every 3 hours Predictionresults
- 14. 14 Original Design • Multiple data sourcespushing data into SQL Server and SQL ServerAnalysis ServerData Mining • Hand built 225 different DM modules (5 days, 15 businessdays ahead, 3 differentgroups) • Pipeline processhad to run225 times / day (3 pools x 75 modules)
- 15. 15 Regression Calculations SSAS Data Mining T-SQL Code Intercept R2 Mean Adjusted R2 Coefficients Standard Deviation Variance Standard Error
- 16. Taking advantage of Spark’s DW Capabilities and MLlib
- 17. OR DWData Sources OR Reports OR Predictive Model in Spark Data inserts every 3 hours
- 18. 18 demoOR Block Scheduling Extract History data and run linear regression with SGD with multiple variables
- 19. 19
- 21. 21
- 22. 22
- 23. 23
- 24. 24
- 25. 25
- 26. 26
- 27. 27 OR Schedule Report (example)
- 28. 28 Why the model is working • Can coordinate waitlist schedulinglogistics with physicians and patients within two weeks of the surgery • Plan staff schedulingand resourcessothere are less last-minute staffing issuesfor nursing and anesthesia • Utilization metrics are showing us where we can maximize our elective surgicalscheduleand level demand
- 29. Key Learnings when Migrating from Traditional DW to Spark
- 30. 30 Transitioning to the Cloud Beth Israel DeaconessMedical Center is increasingly moving to cloud infrastructure services with the hopesof closing itsdata center when the hospital's lease is up in the next five years. CIO John Halamka says he's decommissioning HP and Dell servers as he movesmore of hiscompute workloads to Amazon Web Services, where he's currently using 30 virtual machines to test and develop new applications. "It is no longer cost effective to deal with server hosting ourselves because our challenge isn't real estate, it's power and cooling," he says.
- 31. 31 Transitioning to the Cloud • Need time for engineers,analysts, and data scientists to learn how to build for the cloud • Build for security right from start – processheavy, a lot of documentation, audits / reviews • Differentiating data engineersand engineers(REST APIs, services, elasticity, etc.)
- 32. 32 Transitioning to Spark • No more stored proceduresor indexes • Good for Spark SQL, services design • Prototype, prototype, prototype • Leverage existing languagesand skill sets • Leverage the MOOCsand other Spark training • Break down the silos of data engineers,engineers,data scientists, and analysts
- 33. 33 Transitioning DW to Spark • Understand Partitioning,Broadcast Joins, and Parquet • Not all Hive functionsare available in Spark (99%of the time that is okay) due to Hive context • Don’t limit yourselfto build star-schemas / snowflake schemas • Expand outside of traditional DW: machine learning,streaming
- 34. Thank you. For more information, please contact [email protected] [email protected]