Downloaded 31 times
More Related Content
Similar to Regulating AI & ML in Medicine
![Rheumatic Fever CASE PRESENTATION [Autosaved].pptx](https://cdn.slidesharecdn.com/ss_thumbnails/casepresentationautosaved-251123182512-9d9b0da4-thumbnail.jpg?width=640&height=640&fit=bounds)
Regulating AI & ML in Medicine
- 1. REGULATING AI &ML IN MEDICINE Shyamal Patel, PhD Digital Medicine & Translational Imaging Pfizer, Inc. An overview of the “Proposed Regulatory Framework for Modifications to Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD)” https://www.fda.gov/media/122535/download 5/19/19 1
- 2. AI, machine learningand deep learning Artificial Intelligence Machine Learning Deep Learning 1950s Today1980s ApplicationBreadth Automated Driving Speech Recognition RoboticsObject Recognition Bioinformatics Recommender Systems Spam Detection Fraud Detection Weather Forecasting Algorithmic Trading Sentiment Analysis Medical Diagnosis Health Monitoring Computer Board Games Machine Translation Knowledge Representation Perception Reasoning Interactive Programs Expert Systems 25/19/19
- 3. AI/ML around us Source:https://beebom.com/examples-of-artificial-intelligence/ 35/19/19
- 4. Medicine is becomingincreasingly data driven Chaussabel, D., & Pulendran, B. (2015). A vision and a prescription for big data-enabled medicine. Nature Publishing Group, 16(5), 435–439. http://doi.org/10.1038/ni.3151 45/19/19
- 5. AI/ML in medicine Sources:SkinVision, PathAI, CheXNet assist radiologists in detection of pneumonia from chest X-rays at a level exceeding practicing radiologists assist pathologists in making diagnoses and identify patients that benefit from novel therapies assist consumers in performing self-checks of skin health and provide indications for risk of skin cancer 55/19/19 Skin Health Pathology Reads Radiology Workflow
- 6. Classical programming VSAI/ML • learn efficiently from large, heterogeneous datasets • learn complex non-linear relationships • adapt to the unique needs of each individual • update continuously or when more data becomes available Advantages of AI/ML Classical Programming Data + Rules = Answers AI/ML Data + Answers = Rules 65/19/19
- 7. When would anAI/ML-based solution require premarket submission for an algorithm change? Sensor data Images … Diagnosis Intervention … Input AI/ML algorithm Output 75/19/19
- 8. Risk categorization ofSoftware as a Medical Device (SaMD) SaMD When software is intended to be used for one or more medical purposes without being part of a hardware medical device AI/ML-based SaMD When algorithms are intended to diagnose, treat, cure, mitigate or prevent diseases 85/19/19
- 9. Types of AI/ML-basedSaMD modifications • Changing the types of input signals • Changing compatibility with different devices Modifications related to inputs • Retraining with new data • Changes in AI/ML architecture or parameters Modifications related to performance • Changing the significance of information • Changing the intended patient population Modifications related to intended use i ii iii 95/19/19 Sensor data Images … Diagnosis Intervention …
- 10. Proposed Total ProductLife-Cycle (TPLC) approach 105/19/19
- 11. 1. Quality systemsand Good Machine Learning Practices (GMLP) • Acquire data in a consistent, clinically relevant and generalizable manner • Maintain proper separation between training, tuning and testing datasets • Maintain appropriate level of clarity of output and algorithm to end user • Define performance metrics and success criteria in advance • Select AI/ML technique after careful analytical consideration of factors like simplicity, interpretability, accuracy, robustness, speed and scalability 115/19/19
- 12. 2. Modification plansubmission during initial premarket review Scope of modifications SaMD Pre-specifications (SPS) Manage & control risks Algorithm Change Protocol (ACP) SPS Inputs Performance Intended Use ACP Data Management Retraining Performance Evaluation Update Procedures 125/19/19
- 13. 3.Approach for modificationswith an established SPS andACP 135/19/19
- 14. 4. Real-world performancemonitoring and transparency Monitoring Understand how product is being used in the real-world, identify opportunities for improvement and respond proactively to safety or usability concerns Transparency Provide periodic updates to the stakeholders (e.g. FDA, collaborators, clinicians, patients) on implementation of updates (as per SPS and ACP) and related changes in performance metrics Monitoring/repor ting type and frequency based on risk of the device number and type of modifications maturity of algorithms 145/19/19
- 15. Example 1: Intensivecare unit SPS Modify algorithm to ensure consistent performance across sub-population Reduce false alarm rates while maintaining or increasing sensitivity ACP Methods for database generation, reference standard labeling, and comparative analysis Specification of performance requirements and statistical analysis plan Detect onset of physiological instability Risk category III: ‘drive clinical management’ in a ‘critical healthcare situation or condition’ 155/19/19 ECG, Blood Pressure, Pulse Oximetry
- 16. Example 1: Intensivecare unit Modification Scenario 1A Algorithm modified in accordance with ACP to achieve lower false- alarm rate while maintaining sensitivity on an independent validation dataset Update algorithm and labeling in accordance with modified SaMD performance and communicate to users 165/19/19 Detect onset of physiological instability Risk category III: ‘drive clinical management’ in a ‘critical healthcare situation or condition’ ECG, Blood Pressure, Pulse Oximetry
- 17. Example 1: Intensivecare unit Modification Scenario 1B Algorithm re-trained on additional data can now predict onset of physiological instability 15 minute in advance while maintaining same sensitivity and false-alarm rate Update algorithm, labeling and intended use to indicate change in alarm condition 175/19/19 Detect onset of physiological instability Risk category III: ‘drive clinical management’ in a ‘critical healthcare situation or condition’ ECG, Blood Pressure, Pulse Oximetry
- 18. Example 2: Skinlesion mobile medical app SPS Improve sensitivity and specificity of skin lesion characterization by using real-world data Extend usage with other smartphones with similar image acquisition capabilities and monitor performance ACP Methods for database generation, reference standard labeling, and comparative analysis Acceptance criteria for image acquisition systems, design of validation study and plan for real-world performance Physical characteristics of skin lesion Risk category II: ‘drive clinical management’ in a ‘serious healthcare situation or condition’ 185/19/19 Smartphone camera image
- 19. Example 2: Skinlesion mobile medical app Modification Scenario 2A Algorithm actively learning on real-world data achieved improved sensitivity and specificity in analytical validation (per ACP) Update algorithm and labeling in accordance with modified SaMD performance and communicate to users 195/19/19 Physical characteristics of skin lesion Risk category II: ‘drive clinical management’ in a ‘serious healthcare situation or condition’ Smartphone camera image
- 20. Example 2: Skinlesion mobile medical app Modification Scenario 2B Analytical validation (per ACP) demonstrated two additional smartphones with similar image acquisition systems achieved performance consistent with initial system Update labeling to reflect new app compatibility and communicate to users 205/19/19 Physical characteristics of skin lesion Risk category II: ‘drive clinical management’ in a ‘serious healthcare situation or condition’ Smartphone camera image
- 21. Example 2: Skinlesion mobile medical app Modification Scenario 2C New algorithm analyses the physical characteristics of skin lesions and provides recommendation to the user based on assessment of malignancy Distribute a new app that will now be patient-facing (instead of the dermatologist) 215/19/19 Physical characteristics of skin lesion Risk category II: ‘drive clinical management’ in a ‘serious healthcare situation or condition’ Smartphone camera image
- 22. Example 3: X-rayfeeding tube misplacement SPS Improve accuracy of incorrect tube placement by using real-world data Extend notifications to nursing staff based on achieving pre-specified performance ACP Methods for real-world data collection, reference standard, performance, and comparative analysis Analytical validation of performance improvement and clinical validation for high confidence cases Detect feeding tube placement error Risk category II: ‘drive clinical management’ in a ‘serious healthcare situation or condition’ 225/19/19 Chest X-ray
- 23. Example 3: X-rayfeeding tube misplacement Modification Scenario 3A Algorithm re-trained and re-validated on real-world data achieved higher accuracy and clinical validation of high confidence cases Update algorithm with new version and add notifications to nursing staff in parallel with radiologists for high confidence cases 235/19/19 Detect feeding tube placement error Risk category II: ‘drive clinical management’ in a ‘serious healthcare situation or condition’ Chest X-ray
- 24. Example 3: X-rayfeeding tube misplacement Modification Scenario 3B New algorithm trained and validated to identify pneumonia using development and validation process similar to SPS and ACP with some adaptations. Market new algorithm for pneumonia detection from chest x-ray 245/19/19 Detect feeding tube placement error Risk category II: ‘drive clinical management’ in a ‘serious healthcare situation or condition’ Chest X-ray
Editor's Notes
- #3 ----- Meeting Notes (5/13/19 15:38) ----- Wont spend too much time on this.
- #4 Massive amount of data Personalization Continuous learning
- #8 ----- Meeting Notes (5/13/19 15:38) ----- Put a snapshot of the pdf instead of the text
- #11 TPLC approach enables the evaluation and monitoring of a software product from its premarket development to post-market performance, along with continued demonstration of the organization’s excellence ----- Meeting Notes (5/13/19 15:38) ----- pre-cert barrier to innovation?
- #12 Reasonable assurance of the high quality of software development, testing, and performance monitoring
- #16 ----- Meeting Notes (5/13/19 15:38) ----- Provide images as input ...
- #17 Type “I” modification Additional FDA review not required
- #18 Type “iii” modification FDA may review new SPS and ACP for new information about algorithm modification before providing approval to make the change.
- #20 Type “i” modification Additional FDA review not required
- #21 Type “ii” modification Additional FDA review not required
- #22 Type “iii” modification inconsistent with SPS and ACP Introduces many new, unconsidered risks not yet mitigated FDA may require a new pre-market submission or application and updated SPS and ACP ----- Meeting Notes (5/13/19 15:38) ----- Put a
- #24 Type “iii” modification Changes consistent with SPS and ACP Additional FDA review not required ----- Meeting Notes (5/13/19 15:38) ----- Put a color box
- #25 Type “iii” modification Changes consistent with SPS and ACP but change in healthcare situation and condition as well as significance of information … new intended use FDA may require new premarket submission or application and updated SPS and ACP