Downloaded 22 times
Uploaded byLevi Shapiro
mHealth Israel_Human Factors Engineering BOOTCAMP_Tressa J. Daniels_AAMI
The document outlines a Human Factors Engineering Bootcamp presentation by Tressa J. Daniels, focusing on the role of human factors in medical device development. It discusses regulatory standards, methodologies, and best practices for incorporating user needs and safety into product design, including the importance of validation testing and user interface specifications. The emphasis is on ensuring that devices are safe, effective, and usable in real-world conditions through systematic research and evaluation.
More Related Content
Similar to mHealth Israel_Human Factors Engineering BOOTCAMP_Tressa J. Daniels_AAMI
mHealth Israel_Human Factors Engineering BOOTCAMP_Tressa J. Daniels_AAMI
- 1. Human Factors Engineering Bootcamp April13, 2020 Tressa J. Daniels, M.S. Global Director Human Factors Engineering T. Daniels Do Not Copy or Distribute
- 2. Disclaimer The opinions expressedin this presentation and on the following slides are solely those of the presenter and do not represent those of Teleflex Inc. or AAMI. Presentations are intended for educational purposes only and do not replace independent professional judgement. Teleflex Inc. and AAMI do not endorse or approve, and assumes no responsibility for the content, accuracy or completeness of information presented.
- 3. Agenda • Introduction (5minutes) • Human Factors Engineering in the Product Development Process (15 minutes) • Regulatory Standards and Expectations regarding Human Factors (20 minutes) • Human Factors Validation Testing (20 minutes) • What leads to failure (5 minutes) • What to do if failure occurs (5 minutes) • Submitting a Human Factors Report to a Regulatory Body (5 minutes) • Human Factors Best Practices (5 minutes) • Concluding Remarks (5 minutes)
- 4. Definition “The scientific disciplineconcerning understanding of interactions among humans and other elements of a system, and applying theory, principles, data, and other methods to design of user interfaces in order to optimize human well-being and overall system performance.”* *As defined by the Human Factors and Ergonomics Society T. Daniels Do Not Copy or Distribute
- 5. Human Factors Engineers Research Research userand product problems Study Study workflows Observe Observe environments of use Define Define product and user needs Run Moderate usability testing to verify solutions meet user expectations Ensure Ensure safety and effectiveness are designed into the product T. Daniels Do Not Copy or Distribute
- 6. Industries that apply HumanFactors T. Daniels Do Not Copy or Distribute
- 7. Human Factors Engineering Unique to Medical Devices Globally regulatedindustry Balance between “Safe” & “Usable” Systems Risk assessment oriented T. Daniels Do Not Copy or Distribute
- 8. The Primary Objective Identification andcontrol of user interface design problems or flaws that could lead to… • Harm to user or patients • Compromised medical care T. Daniels Do Not Copy or Distribute
- 9. Human Factors inProduct Development Process: Activities & Deliverables T. Daniels Do Not Copy or Distribute
- 10. Human Factors Skillsets & Activities •Formative Research - Identifying user’s needs and wants • Ethnographic research • Usability Inspection & Heuristic Evaluations • Formative Usability Testing • Tasks Analysis • Interaction Design • Workflows, wire frames • Online, mobile, embedded graphic design • Industrial Design • Product Color, shape and form • Physical interface design • Ergonomic design • Human Factors Validation - Testing and validating safety and efficiency • HF/UE Documentation T. Daniels Do Not Copy or Distribute
- 11. Process T. Daniels Do NotCopy or Distribute
- 12. In the beginning:HF Focus is on Research CONTEXTUAL INQUIRY BEHAVIORAL OBSERVATION JOB SHADOWING SITE VISITS T. Daniels Do Not Copy or Distribute
- 13. In the middle:HF Focus is on Design Feedback Prototypes Low fidelity: Paper High fidelity: Clickable screens or foam core mockups Formative Evaluations T. Daniels Do Not Copy or Distribute
- 14. Feedback gathered on allproduct UI • IFU • Packaging • Labeling • Hardware • Software T. Daniels Do Not Copy or Distribute
- 15. Design changes implementedbased on feedback T. Daniels Do Not Copy or Distribute
- 16. At the end:HF Validates Production Units Sample Size: 15 users per user group Final production equivalent product All aspects of the UI Focus on critical tasks Relies on uFMEA or Hazards Analysis If design changes are needed after validation – those changes must be validated T. Daniels Do Not Copy or Distribute
- 17. Documentation created during the PDP •HF/UE Project Plan • Exploratory Research Plan • Exploratory Research Report • Use Specification • Task Analysis • uFMEA • User Interface Specification • Formative Usability Test Plan (per test) • Formative Usability Test Report (per test) • Human Factors Validation Test Plan • Human Factors Validation Test Report • HF/UE Project Report • HF/UE File T. Daniels Do Not Copy or Distribute
- 18. Regulatory Standards Related toHuman Factors T. Daniels Do Not Copy or Distribute
- 19. Medical products are regulated by these standards& guidances IEC-62366 ANSI/AAMI HE75 FDA Human Factors in MD Design PMA Submissions 510K Submissions T. Daniels Do Not Copy or Distribute
- 20. US and OUSThe IEC report provides guidance on how to implement a usability engineering process. It contains requirements for safety and human factors. By following the IEC, AAMI and FDA guidelines the likelihood of a successful PMA and/or 510k submission increases greatly. Code of Federal Regulations, Part 820 Title 21. T. Daniels Do Not Copy or Distribute
- 21. AAMI: TIR 59: IntegratingHuman Factors into Design Controls
- 22. Regulatory Expectations Use Specification Identify Intended Users DevelopUser Profiles Define User Groups Understand Known Use Errors Utilize Risk Management Tools Plan for Formatives Execute Design Validation T. Daniels Do Not Copy or Distribute
- 23. Initiate Use Specification • Usespecification is refined over time starting with Intended use. • Level of detail is increased through user research • A Use specification might contain the following: • Identify user groups which are going to be approached for interviews • Identify use environment which is going to be inspected • Identify medical indications which are needed to be further explored • These elements aid in identifying the known and foreseeable hazards and hazardous situations related to the user interface. • Understanding these elements is necessary to develop an adequate usability evaluation plan T. Daniels Do Not Copy or Distribute
- 24. Methodologies for development of Use Spec •Contextual Inquiry and Observation • Researcher observes users while performing User Tasks in intended use environment and discusses the details with the user • Interviews and Surveys • Location agnostic discussion with users as to knowledge, perception and opinions as they relate to the medical device and its use. This can be stand-alone or supplement the above-mentioned observation • Expert reviews • An expert might cite strengths and weaknesses of a device or heuristic analysis might be performed where a number of experts independently identify and prioritize improvements • Advisory Panel reviews • 6-12 members who collaborate with the design team on design options. Such an advisory panel might be leveraged continuously throughout the design process • Competitive/Predicate Device Usability Testing • Performing base-line usability testing to identify strengths and weaknesses of existing products T. Daniels Do Not Copy or Distribute
- 25. Identify Intended Users • Example PrimaryUsers: • Patients • Physicians • Nurses • Technicians • Therapists • Pharmacists • Emergency Response Personnel • Example Non-Primary Users: • Assemblers • Installers • Trainers • Transporters • Engineers • Repair Personnel • Recyclers • Sterile Processors • Admin Personnel T. Daniels Do Not Copy or Distribute
- 26. Develop User Profiles • Commoncharacteristics of Users • Examples include: • Occupation • Demographics • Knowledge and Skills (education, experience level, language, literacy) • Limitations (vision, hearing, cognitive, impairments) • Performance Shaping Factors (learning style, preferences, tendencies) • Work responsibilities (Tasks pertinent to medical device) • It may be helpful to develop a “Persona” or a fictitious user who represents a user profile T. Daniels Do Not Copy or Distribute
- 27. Define User Groups • Whereusers share characteristics, users can be grouped to inform usability • Examples include: • Age: Child (>2 yrs to 12 yrs), adolescent (>12 yrs to 21 yrs), Adult (>21 yrs) • Occupation: Physician, Nurse, Therapist, Technician, Patient, Installer • Prior experience using similar medical devices: New user (none), inexperienced (<6 mo), experienced user (>6 mo) • Level of training • Education T. Daniels Do Not Copy or Distribute
- 28. Anticipate Environment of Use • Characteristicsof the anticipated use environment should be defined to inform usability. Examples include: • Physical environment (gloves, eye protection, heavy clothing) • Lighting • Noise level • Professional interactions and responsibilities • Additional equipment present • Furnishings • Climate • Distractions T. Daniels Do Not Copy or Distribute
- 29. Understand Known Use Errors • ComplaintData • MAUDE • EUDAMED • Literature Review • Social Media T. Daniels Do Not Copy or Distribute
- 30. Finalize Use Specification • IEC62366-1 requires the following: • Intended Medical Indication • Intended Patient Population • Intended Part of the Body or Type of Tissue Interacted With • Intended User Profiles • Intended Use Environment • Operating Principle • The following are not required but are recommended: • User Tasks (Use Cases, User Stories) • User needs derived from User Tasks T. Daniels Do Not Copy or Distribute
- 31. User Interface Specification • The UserInterface Specification consists of: • The Use Specification • The known or foreseeable Use Errors • The selected Hazard-Related Use Scenarios • The User Interface Specification should be developed ahead of the User Interface, but then evolve with the ensuing design • User interface requirements should be developed to encompass Instructions for Use and other Accompanying Documentation • IEC 62366-1 requires that the manufacturer determine whether Accompanying Documentation or Training are required for safe use of the medical device T. Daniels Do Not Copy or Distribute
- 32. Iterative Formative Usability Testing • IEC Recommends2 to 3 Formative evaluations to be conducted at a minimum. They further note that multiple small-scale evaluations can be more productive than fewer larger scale evaluations. • Formative Evaluations can vary in formality and methodology but should begin to resemble the planned Validation as it approaches. • Both Formative and Summative Evaluations typically involve one or more Usability Tests. • IEC 62366-1 requires that the Usability Test Protocol include the following: • Participants in the Usability Test to be representative of each intended User Group • Test environment and other use conditions, to be representative of the Intended Use Environment • The Accompanying Documentation to be provided during the Usability Test, if any • The Training to be provided during the Usability Test, if any, and the minimum elapsed time between training and the beginning of the Usability Test T. Daniels Do Not Copy or Distribute
- 33. Human Factors ValidationTest T. Daniels Do Not Copy or Distribute
- 34. Design Validation Defined Validation is theprocess of making sure that you have objective evidence that user needs and intended uses are met. It is usually done by tests, inspections, and in some cases analysis. However, the target of the validation is to make sure the user needs are met in a medical device that consistently provides the intended medical benefit in actual-use conditions. https://www.asme.org/topics-resources/content/validation-verification-for-medical-devices T. Daniels Do Not Copy or Distribute
- 35. Design Verification Defined • According tothe FDA, design verification is “confirmation by examination and provision of objective evidence that specified requirements have been fulfilled.” • Keep in mind that while it will involve testing, there are other acceptable verification activities. • They can include tests, inspections, and analyses (for more on this, check out FDA Design Control Guidance • While Validation looks at User Needs, Verification looks at Product Requirements & Design Specs. T. Daniels Do Not Copy or Distribute
- 36. Design Validation Guidance • CFR 820.30 •Design validation. Each manufacturer shall establish and maintain procedures for validating the device design. Design validation shall be performed under defined operating conditions on initial production units, lots, or batches, or their equivalents. Design validation shall ensure that devices conform to defined user needs and intended uses and shall include testing of production units under actual or simulated use conditions. Design validation shall include software validation and risk analysis, where appropriate. The results of the design validation, including identification of the design, method(s), the date, and the individual(s) performing the validation, shall be documented in the DHF. • The FDA Design Controls Guidance • VALIDATION METHODS. Many medical devices do not require clinical trials. • However, all devices require clinical evaluation and should be tested in the actual or simulated use environment as a part of validation. This testing should involve devices which are manufactured using the same methods and procedures expected to be used for ongoing production. While testing is always a part of validation, additional validation methods are often used in conjunction with testing, including analysis and inspection methods, compilation of relevant scientific literature, provision of historical evidence that similar designs and/or materials are clinically safe, and full clinical investigations or clinical trials. T. Daniels Do Not Copy or Distribute
- 37. Human factors Validation Defined • Human FactorsValidation is a type of Design Validation • Demonstrates that devices can be used by the intended users, under expected use conditions, without serious use errors or problems. • Test participants represent the intended (actual) users of the device. • All critical tasks are performed during the test. • The device user interface represents the final design (production equivalent). • The test conditions are sufficiently realistic to represent actual conditions of use. T. Daniels Do Not Copy or Distribute
- 38. IEC Guidance Clause5.9 • The MANUFACTURER shall perform a SUMMATIVE EVALUATION of each HAZARD-RELATED USE SCENARIO on the final or production equivalent USER INTERFACE. • The data from the SUMMATIVE EVALUATION shall be analyzed to identify the potential consequences of all USE ERRORS that occurred. If the consequences can be linked to a HAZARDOUS SITUATION, the root cause of each USE ERROR shall be determined. T. Daniels Do Not Copy or Distribute
- 39. FDA Guidance section3.7 • Testing conducted at the end of the device development process to assess user interactions with a device user interface to identify use errors that would or could result in serious harm to the patient or user. • Human factors validation testing is also used to assess the effectiveness of risk management measures. • Human factors validation testing represents one portion of design validation. T. Daniels Do Not Copy or Distribute
- 40. Simulated HFVT Simulated-use shouldbe sufficiently realistic so that the results of the testing are generalizable to actual use. It is not acceptable to use of the “think aloud” technique. Mirrors participant’s real-life interaction with labeling. Testing carried out with final finished combination product. (can contain placebo) Simulation methods can vary (e.g., manikin, injection pads, placebo, etc.) T. Daniels Do Not Copy or Distribute
- 41. Test Participants Represent thepopulation of intended users. Minimum number of participants is 15 for each distinct user population; (e.g. pediatric, adult, healthcare providers and lay users) For FDA Reside in the United States Exceptions to this are considered on a case-by-case basis. Does not include your employees. T. Daniels Do Not Copy or Distribute
- 42. Task & Use Scenarios •Tasks that logically occur in sequence > Use Scenarios • Organize to represent a natural workflow. • Prior to testing, define user performance that represents success for each task. Not just a percentage of completion! T. Daniels Do Not Copy or Distribute
- 43. Select Hazard- Related Use Scenarios • Choosetasks based on severity • Critical tasks that have a low frequency of occurrence require careful consideration and should be included in the testing. • the manufacturer must determine which Hazard-Related Use Scenarios shall be evaluated • The purpose of this determination is to ensure that the Validation includes all Use Scenarios needed to demonstrate Safety related to the User Interface of the Medical Device • IEC 62366-1 provides the following three options: • Include all Hazard-Related Use Scenarios • Include a subset of Hazard-Related Use Scenarios based on Severity of Harm • Include a subset of Hazard-Related Use Scenarios based on Severity of Harm and circumstances specific to device and Manufacturer • HF Validation must also show that Users are able to accomplish the intended purpose of the Medical Device as described in the Use Specification T. Daniels Do Not Copy or Distribute
- 44. Instructions for Use • Thelabeling must represent the final designs. • Validation test can indirectly assess the IFU, but only in the context of use. • Stating that you mitigated the risks by “modifying the IFU” is not acceptable, unless you provide additional test data demonstrating that the modified elements were effective in reducing the risks to acceptable levels T. Daniels Do Not Copy or Distribute
- 45. IFU Knowledge Tasks • Assesscomprehension • Read-then-do • Read-then- paraphrase T. Daniels Do Not Copy or Distribute
- 46. Participant Training • Should approximatethe training that actual users would receive. • Minimum 1-hour training decay must follow training • Longer time may be appropriate if potential source of use-related risk. • Stating that you mitigated the risks by “providing additional training” is not acceptable unless you provide additional data that demonstrates that it would be effective in reducing the risks to acceptable levels. T. Daniels Do Not Copy or Distribute
- 47. Objective Data Collection • Observational data- observations of participants’ performance of all the critical use • Knowledge task data - comprehension of interface components • DFU, quick start guide, labeling on the device itself, and training. T. Daniels Do Not Copy or Distribute
- 48. Subjective Data Collection • Post SessionInterview data from Participants • Their feedback considering the overall device focused on each critical task or use scenario. • Their identification of any use difficulties, confusions or errors that were experienced during the test. • Their assessment of root cause for any observed or participant reported use difficulties, confusions or errors • Do not use Likert scale ratings T. Daniels Do Not Copy or Distribute
- 49. Actual Use Testing •Definition: Use of final finished product in a real (not simulated) environment of use. • It is rare that actual-use testing is determined to be necessary to ensure safe use of the proposed device. T. Daniels Do Not Copy or Distribute
- 50. Analyzing the Data T.Daniels Do Not Copy or Distribute
- 51. Why we doHuman Factors Validation Tests T. Daniels Do Not Copy or Distribute
- 52. REGULATION • FDA HFDevice Guidance: • “CDRH recommends that manufacturers consider human factors testing for medical devices as a part of a robust design control subsystem. • CDRH believes that for those devices where an analysis of risk indicates that users performing tasks incorrectly or failing to perform tasks could result in serious harm, manufacturers should submit human factors data in premarket submissions (i.e., PMA, 510(k)).” T. Daniels Do Not Copy or Distribute
- 53. We should Responsible for safeand effective products Controls risk Reduces support calls Reduces complaints Reduces recalls Competitive advantage Brand loyalty T. Daniels Do Not Copy or Distribute
- 54. What is thedifference between a Usability Test & a Human Factors Validation Test? T. Daniels Do Not Copy or Distribute
- 55. Formative Methods • Analytical ► TaskAnalysis ► Heuristic Evaluations ► Expert Reviews • Empirical ► Interviews & Focus Groups ► Contextual Inquiry ► Formative Studies (Cognitive Walkthroughs, & Usability Testing) T. Daniels Do Not Copy or Distribute
- 56. Goals of FormativeEvaluations COLLECT FEEDBACK UNDERSTAND PAIN POINTS UPDATE DESIGN ITERATIVE TEST/EVALUATION T. Daniels Do Not Copy or Distribute
- 57. Formative vs. HF Validation •Formative – expect to iterate on the design to “form” it • Formative studies can be relatively short and conducted with a small number of representative users. • Conducted early in the design cycle, as few as 8-10 participants can reveal 90 percent of existing design flaws in the UI that can be modified to eliminate errors and use problems. • As the UI design matures, formative studies may be employed as trial run tests before conducting the final validation test, which saves significant time and cost by preventing the need to repeat the validation study due to residual problems in the interface or the study methodology itself. T. Daniels Do Not Copy or Distribute
- 58. Formative vs. HF Validation (cont.) •Validation – prove safe and effective use • is not meant to be an exploratory effort seeking inputs on design features but should serve as a ‘final’ demonstration of use safety for the device. Therefore, participants are not interrupted with questions or corrected in their performance. • Participants engage in use scenarios chosen to represent sequences of typical interaction with the device. These scenarios should cover all tasks that have been categorized as containing high risk in the use risk analysis. • Training and device familiarization should be provided and represent realistic conditions. This could be done by conducting a training session prior to testing with an appropriate intervening interval to represent potential memory and learning decay. • Device instructions for use should be available to the participant, but with no requirement to read the instructions prior to performing the use scenarios, unless the participant chooses to do so on their own. T. Daniels Do Not Copy or Distribute
- 59. Formative vs. HF Validation (cont.) •Validation – prove safe and effective use • User performance on each scenario task (or step) should be observed and categorized with respect to failure, success, and success with difficulties such as hesitation, self correcting of actions, and potential confusion. These instances, along with failures, should be noted for further post-test analysis. • Should include a post-test dialogue with the study team in order to determine the root cause of any failures and difficulties including both specific questions about unexpected or incorrect actions, or general questions about task difficulty during the test. • Results of the validation test should support an overall conclusion regarding use safety for the device. This conclusion should not be based on meeting pre-defined quantitative goals (i.e. 95 per cent success rate across tasks and participants), but on whether there is a remaining pattern of use-related problems that are directly attributable to the UI or accompanying documentation (labelling and instructions, etc.). T. Daniels Do Not Copy or Distribute
- 60. Order of Events T.Daniels Do Not Copy or Distribute
- 61. Where is HFVT within theProcess • End • Production Equivalent System ► Training (IFU, ILT, Computer Based) ► Product ► Labels/Labeling ► Hardware ► Software T. Daniels Do Not Copy or Distribute
- 62. What Leads toFailure T. Daniels Do Not Copy or Distribute
- 63. Use Error • Asituation where device use was different than intended, but not due to malfunction of the device • Frequently, use errors are due to poorly designed devices • Well-designed medical devices eliminate or minimize the possibility of a user using a device incorrectly Easy to use correctly = fewer medical use errors = safer device https://www.fda.gov/medical-devices/human-factors-and- medical-devices/postmarket-information-device-surveillance-and- reporting-processes T. Daniels Do Not Copy or Distribute
- 64. Common Mistakes • Testing anincomplete sample size (fewer than 15 users per user group) • Lack of evidence or due diligence of developing an appropriate HF protocol • Not relying on your use related risk analysis to drive test scenarios • Lack of consideration of KUE’s for a predicate product • Inability to prove an effective mitigation • Preparing to make it a formative test if validation fails T. Daniels Do Not Copy or Distribute
- 65. Reasons for validation Failure •Failure to conduct formative evaluations • Validating untested final product and study design • Failure to focus on strong mitigations • Red flags identified before study initiation • Timeline priority affecting decisions T. Daniels Do Not Copy or Distribute
- 66. Responding to Validation Failure •Pause the study • Redesign your protocol • Repeat validation • Convert validation to formative • Identify opportunities for improvement • Initiate redesign • Assess study results • Determine the validation failures • Initiate redesign • Focused validation T. Daniels Do Not Copy or Distribute
- 67. Design Changes T. Daniels DoNot Copy or Distribute
- 68. Making Design Changes after HFVT • Considerationsfor Design Changes after HF Validation • ► Clinical study reveals design flaws • ► Post-market surveillance • ► Respond to post-market use-related safety reports, complaints or problems • ► Meet needs of an expanded indication or user population T. Daniels Do Not Copy or Distribute
- 69. Making Design Changes after HFVT • Conductupdated use-related risk analysis • ► Does the design change alter the user interface in any way (e.g., audible, tactile, color recognition, user instructions, etc.)? • ► Does the design change alter an existing critical task or add a new critical task? • ► Does the design change alter the expected users or their knowledge base? T. Daniels Do Not Copy or Distribute
- 70. Submitting an HFVTReport T. Daniels Do Not Copy or Distribute
- 71. Submitting an HFVT Report Submissions toFDA pre-market approval process generally require validation test results Typical Manufacturer Deficiencies -Not using the preliminary use risk analysis to justify testing protocols -Reporting just the success rates without making the overall use-safety case -Reporting preference results -Failure to include the right user groups -Attributing test result failures to human error without further justification -Failure to test the IFU when indication it is a risk control T. Daniels Do Not Copy or Distribute
- 72. Concluding Remarks T. Daniels DoNot Copy or Distribute
- 73. Best Practices Early research: •Ethnography •Customerconfirmation •Build user needs based on data Iterative formative testing during design and development – with all intended user groups Ensure a minimum of 15 users per distinct user group Your employees should not serve as test participants Be sure to test mitigations that map to IFU comprehension Clearly list critical tasks and the mitigation you are testing – linked to a user need Clearly define performance success/failure (acceptance criteria) Clearly describe how you will identify root cause(s) of use errors, difficulties, and close calls If design changes are made to any part of a user facing portion of the product you must do a focused validation test T. Daniels Do Not Copy or Distribute
- 74. Thank You! Tressa J.Daniels [email protected] https://www.linkedin.com/in/tressadaniels/ T. Daniels Do Not Copy or Distribute