When
Perfect Algorithms
Meet
Imperfect Healthcare Systems
Joyce Lee, MD, MPH
Robert Kelch Professor of Pediatrics
doctorasdesigner.com
@joyclee on Twitter
Lenovo (Grant Funding)
Disclosures
@joyclee
www.doctorasdesigner.com
medium.com/@joyclee
Professor, Robert Kelch Professor of Pediatrics
Clinical/Health Outcomes Research
(Obesity/Diabetes)
Learning Health Systems
(Clinical Informatics, Quality Improvement, Participatory Design)
Applications of Emerging Technologies in Healthcare
(Ethnography, Analysis of Diabetes Online Maker Communities)
@joyclee
@joyclee
Agenda
Why Diabetes Data Matters
Diabetes Data Barriers
Patient Innovation with Data
Why
Diabetes Data
Matters
@joyclee
@joyclee
It’s an infinite amount of math
over a lifetime
It’s an infinite amount of math
over a lifetime
@joyclee
Blood Glucose
Meters
Continuous Glucose
Monitoring (CGM)
Systems
Insulin Pumps
@joyclee
Glucose levels change all the time
Puberty Growth Activity
Glucose levels impact health outcomes
Management is truly patient-driven
Provider Time
managing diabetes
(1 hour in a year?)Patient Time Spent
Managing Diabetes
Data Nirvana for Cars
@joyclee
“The Autonomous Car”
The Singularity!
@joyclee
@joyclee
“The Artificial Pancreas”
Continuous
Glucose Monitor
Insulin
Pump
Mobile Phone
with Algorithm
Data Nirvana for Diabetes
The Singularity!
Goodbye Pediatric Endocrinologist!
You have been eliminated!
@joyclee
The Singularity!
…not
really
@joyclee
@joyclee
“The Artificial Pancreas”
“Closing the Loop"
Medtronic 670G DIY In Development
What are the
Diabetes Data
Barriers?
@joyclee
No data
Trapped data
Infrequent data
Isolated data
Confusing data
Biased data
Judgmental data
Suspicious data @joyclee
No data
@joyclee
We still give out mostly analog tools
”It was unacceptable to me in 2002, when my son was
diagnosed, to be given needles, an insulin vial, and a piece
of paper.”
- Jeff Brewer, Bigfoot CEO @joyclee
Blood Glucose Meters (100%)
Insulin Pumps (50%)
Continuous Glucose Monitoring (CGM) Systems (15%)
Connected Pens (<1%)
AP Systems (<1%)
Device Adoption Rates (Type 1 Diabetes)
@joyclee
Diabetes tools are very expensive
$1411 Starter Kit
$596 Transmitter (every 3 months)
$349/month for Sensors
$70-80 Starter Kit
$120-150/month for Sensors
@joyclee
Insurance companies don’t like to pay
for things like blood glucose test
strips, connected pens, and CGMs
Burnout +
Stigma +
Burden +
Frustration
= No BG data
@joyclee
Trapped data
@joyclee
Email to fax:
xxxxxxx-diabetes-fax@med.umich.edu
@joyclee
@joyclee
@joyclee
People Prefer Mobile to Desktop
@joyclee
Infrequent data
@joyclee
Downloads and diabetes decision-
making only happens at clinic
3-4 visits per year with 15
minutes per visit spent on
data
Meter/pump/CGM is
collected at the clinic visit
and a data download is
exported as a PDF and
scanned into the media tab
of the chart
@joyclee
Mental Health
Issues
20,028 calls in 2017
@joyclee
School Forms
Supplies
Prior Authorization
Insurance
Child Protection
Services
DME
Pharmacy
Patient data review outside of clinic
is reactive, not proactive
Reporting blood
glucose numbers
over the phone
PDF attached
to a Portal Message
(page limits)
Fax (email-fax)
48 hour turnaround time
Isolated data
@joyclee
@joyclee
We get upset when the Color Printer
is not working in clinic
@joyclee
Confusing data
@joyclee
Technology is about culture
change
”We’re living through this time right now where
technology is a Trojan Horse for change. We say
technology, but we mean innovation. We say
interoperability and open data, but we mean
culture change.”
-Susannah Fox
@joyclee
Onboarding Patients to the
Patient Portal and Diabetes Data
Platforms is no one’s job
Not the Health IT Specialists
Not the Medical Assistants
Not the Educators
Not the Doctors
@joyclee
There is no formal patient education
focused on how to download data,
how to interpret it, and how to use it
to adjust insulin doses
“Am I supposed to still keep a logbook?”
“I don’t feel comfortable making dose adjustments
without first consulting with the CDE or
endocrinologist. I mean, it’s his life, you know?”
@joyclee
“Routine Downloaders” = Downloaded data at least once between
routine clinic visits every 3 months which was four or more times
in the past year (40%)
“Routine Reviewers’’ = Reviewed the data at least ‘‘most of the
time’’ he/she downloaded (27%)
Lower A1c for Routine Reviewers: 7.8% vs. 8.6% (p=0.001)
Diabetes Data Platform Use
Wong et al, 2015
@joyclee
There is a lack of standardization for
data visualizations
@joyclee
@joyclee
There is a lack of standardization for
data visualizations
Biased data
@joyclee
@joyclee
Are they covering all carbs?
Are they carb counting carefully?
Are they inputting all the carbs / BG values into the pump?
Are they bolusing before or after meals?
Can I trust the data?
Insulin lasts 2-3 hours
There are A Lot of Unmeasured Variables
Judgmental data
@joyclee
Guilt
Loss of Control
Alienation
Obsession
Data is not always empowering
Isolation
@joyclee
CSCW 2017, Kaziunas, Ackerman, Lindtner, Lee
Suspicious data @joyclee
Patients and clinicians don’t trust a
black box
“I don’t know what
it’s doing so how
can I trust it?”
“I wouldn’t give up
my DIY AP”
@joyclee
And that doesn’t
even include linking this vital
diabetes data to the
clinical EHR data!
@joyclee
“Garbage data”
Lack of User-Centered Design for the
EHR
No culture of human-centered design in Health IT
Design without a strategic understanding of what
metrics are needed to improve care
Build without enough input from users
Deploy without iteration and testing
Physician Resentment/Anger
@joyclee
Clinical EHR: A combination of
Microsoft Word and Pinterest
@joyclee
Clinicians are inputting
data in unstructured format
in the notes
Data is being lost and/or underutilized
Patient paper questionnaires and
the diabetes data are
scanned to PDF
@joyclee
No data
Trapped data
Isolated data
Infrequent data
Confusing data
Suspicious data
Garbage data
PDFMediaTab
How do we address
these Diabetes Data
Barriers?
@joyclee
@joyclee
PDFMediaTab
@joyclee
MediaTab
@joycleeKumar RB et al J Am Med Inform Assoc. 2016 May;23(3):532-7.
@joyclee
MediaTab
Patients, caregivers, clinicians and
researchers work together to choose
care based on best evidence; together
they drive discovery as natural
outgrowth of patient care; and ensure
innovation, quality, safety and value, all
in real-time.
- C3N Project
@joyclee
“
Design for Users
Quality
Improvement
Implementation,
Sustainability,
Outcomes
Human-centered
Design
Data Outcomes
Health IT
@joyclee
Aim: To decrease the % of
the population with
HbA1c ≥ 9% and increase
the % of the population
with ≥ 0.5% HbA1c
interval improvement
Preference driven
treatment and
effective self-
management
Enhanced registry population
management & Pre-visit planning
Peer/community support
Education/training to support
technology use and patient
viewing and problem solving with
blood glucose data between visits
Interventions/toolkits for
addressing barriers to adherence
Efficient use of
technology and data
to support care
Access to care and
regular follow-up
Screening for depressionPsychosocial
Support
Shared decision making
Partnership between
engaged patients
and the care team
Effective use of EHR by diabetes
team for population management
• % of pts. testing ≥4 times/day
or using CGM (6/7 days/week)
• % of pts. giving 3 or more
short-acting boluses/day
• % of pts reviewing data
between visits
• % pts setting, documenting,
and reviewing goals
• % completed pre-visit planning
• % with ≥ 4 visits per year
• % of pts with annual
CDE/RD/SW visit
• % of pts on case mgmt.
pathway
• % pts screened for depression
Developing a Clear Measurable Aim and
a Theory of Change Care Process Measures
@joyclee
Local Infrastructure to support an LHS
Team (Director, Associate Director, Patient Partner/Advisor, Project
Manager/Analyst)
Patient Engagement (Patient Advisor/Advisory Board, Website/Newsletter)
QI interventions (Depression Screening, High-risk Patient Recall, Portal
Onboarding, Data Engagement Curriculum)
Improving Data/Technology Systems
@joyclee
@joyclee
@joyclee
@joyclee
@joyclee
“No one is going to use that tool
if you can’t BOLD the text!”
Rogue commas
Tool for Data Input
Insulin sensitivity
12AM 90
2:30 AM 110
4 AM 230*
10 AM 160
Patient Instructions
Insulin sensitivity 12AM
12AM 90
2:30 AM 110
4 AM 23, 0*
10 AM 160
@joyclee
MediaTab
Outcomes that Matter
Tools for Structured Data Collection
Patient Reported Outcomes
(Portal Questionnaires)
Clinical Interface Redesign
Tools for Population Management
How might our
machine learning
colleagues help?
@joyclee
Understand how, when, and what data
is desired by different stakeholders to
solve the right problems
Think ecosystems not apps
@joyclee
Population Level Monitoring
Prioritization of high-risk
patients
Tools for Seamless
Data Acquisition
Data Pattern Review
Documentation
Communication
Watch patient experts solve problems
@joyclee
Who’s the real
medical expert?
@joyclee
@joyclee
@joyclee
@joyclee
21. France
22. Netherlands
23. New Zealand
24. Mexico
25. Croatia
26. South Africa
27. Israel
28. Japan
29. Switzerland
30. Hungary
31. Belarus
29,000+
Int’l
11. Australia
12. Portugal
13. Denmark
14. Germany/Austria
15. Turkey
16. Russia
17. Romania
18. Korea
19. Ireland
20. Brasil
55,000+
worldwide
1. Italy
2. UK
3. Spain
4. Sweden
5. Bulgaria
6. Norway
7. Poland
8. Czech+Slovakia
9. Canada
10. Finland
Software Innovations
@joyclee
Hardware Innovations
@joyclee
How do you get your CGM in the Cloud?
@joyclee
“The DIY Artificial Pancreas”
Loop/Loopkit OpenAPS AndroidAPS
@joyclee
Interoperability and choice matter
Reduce the mental burden
Proactive not reactive
Be transparent
What should I consider and why?
Give access to all of the data
@joyclee
The machine is not an end. An
airplane is not an end: it is a tool.
Tools are created to allow you to
reach greater goals, and machines
should not distract from this pursuit.
In fact, you should barely notice that
they’re there.
@joyclee
“
Jacob Dwyer, Ashley Garrity, Valeria Gavrila, Emily Hirschfeld
Dorene Markel, Ram Menon, Amy Ohmer, Lilia Verchichina, Michelle Wichorek,
Pediatric Diabetes Team, The Nightscout Foundation, T1D Exchange
Acknowledgements
@joyclee
Joyce Lee, MD, MPH
doctorasdesigner.com
joyclee@med.umich.edu
http://www.nightscout.info/
https://www.nightscoutfoundation.org/contribute-data/
AHRQ, R21HS023865 and PCORI Engagement Award (1442-UMich)

When Perfect Algorithms Meet Imperfect Healthcare Systems: My talk at the Machine Learning for Healthcare Meeting 2018

  • 1.
    When Perfect Algorithms Meet Imperfect HealthcareSystems Joyce Lee, MD, MPH Robert Kelch Professor of Pediatrics doctorasdesigner.com @joyclee on Twitter
  • 2.
  • 3.
    www.doctorasdesigner.com medium.com/@joyclee Professor, Robert KelchProfessor of Pediatrics Clinical/Health Outcomes Research (Obesity/Diabetes) Learning Health Systems (Clinical Informatics, Quality Improvement, Participatory Design) Applications of Emerging Technologies in Healthcare (Ethnography, Analysis of Diabetes Online Maker Communities) @joyclee
  • 4.
    @joyclee Agenda Why Diabetes DataMatters Diabetes Data Barriers Patient Innovation with Data
  • 5.
  • 6.
    @joyclee It’s an infiniteamount of math over a lifetime
  • 7.
    It’s an infiniteamount of math over a lifetime @joyclee
  • 9.
  • 10.
    @joyclee Glucose levels changeall the time Puberty Growth Activity
  • 11.
    Glucose levels impacthealth outcomes
  • 12.
    Management is trulypatient-driven Provider Time managing diabetes (1 hour in a year?)Patient Time Spent Managing Diabetes
  • 13.
    Data Nirvana forCars @joyclee “The Autonomous Car”
  • 14.
  • 15.
    @joyclee “The Artificial Pancreas” Continuous GlucoseMonitor Insulin Pump Mobile Phone with Algorithm Data Nirvana for Diabetes
  • 16.
    The Singularity! Goodbye PediatricEndocrinologist! You have been eliminated! @joyclee
  • 17.
  • 18.
    @joyclee “The Artificial Pancreas” “Closingthe Loop" Medtronic 670G DIY In Development
  • 19.
    What are the DiabetesData Barriers? @joyclee
  • 20.
    No data Trapped data Infrequentdata Isolated data Confusing data Biased data Judgmental data Suspicious data @joyclee
  • 21.
  • 22.
    We still giveout mostly analog tools ”It was unacceptable to me in 2002, when my son was diagnosed, to be given needles, an insulin vial, and a piece of paper.” - Jeff Brewer, Bigfoot CEO @joyclee
  • 23.
    Blood Glucose Meters(100%) Insulin Pumps (50%) Continuous Glucose Monitoring (CGM) Systems (15%) Connected Pens (<1%) AP Systems (<1%) Device Adoption Rates (Type 1 Diabetes) @joyclee
  • 24.
    Diabetes tools arevery expensive $1411 Starter Kit $596 Transmitter (every 3 months) $349/month for Sensors $70-80 Starter Kit $120-150/month for Sensors @joyclee
  • 25.
    Insurance companies don’tlike to pay for things like blood glucose test strips, connected pens, and CGMs
  • 26.
    Burnout + Stigma + Burden+ Frustration = No BG data @joyclee
  • 27.
  • 28.
  • 29.
  • 30.
  • 31.
    People Prefer Mobileto Desktop @joyclee
  • 32.
  • 33.
    Downloads and diabetesdecision- making only happens at clinic 3-4 visits per year with 15 minutes per visit spent on data Meter/pump/CGM is collected at the clinic visit and a data download is exported as a PDF and scanned into the media tab of the chart @joyclee
  • 34.
    Mental Health Issues 20,028 callsin 2017 @joyclee School Forms Supplies Prior Authorization Insurance Child Protection Services DME Pharmacy
  • 35.
    Patient data reviewoutside of clinic is reactive, not proactive Reporting blood glucose numbers over the phone PDF attached to a Portal Message (page limits) Fax (email-fax) 48 hour turnaround time
  • 36.
  • 37.
  • 38.
    We get upsetwhen the Color Printer is not working in clinic @joyclee
  • 39.
  • 40.
    Technology is aboutculture change ”We’re living through this time right now where technology is a Trojan Horse for change. We say technology, but we mean innovation. We say interoperability and open data, but we mean culture change.” -Susannah Fox @joyclee
  • 41.
    Onboarding Patients tothe Patient Portal and Diabetes Data Platforms is no one’s job Not the Health IT Specialists Not the Medical Assistants Not the Educators Not the Doctors @joyclee
  • 42.
    There is noformal patient education focused on how to download data, how to interpret it, and how to use it to adjust insulin doses “Am I supposed to still keep a logbook?” “I don’t feel comfortable making dose adjustments without first consulting with the CDE or endocrinologist. I mean, it’s his life, you know?” @joyclee
  • 43.
    “Routine Downloaders” =Downloaded data at least once between routine clinic visits every 3 months which was four or more times in the past year (40%) “Routine Reviewers’’ = Reviewed the data at least ‘‘most of the time’’ he/she downloaded (27%) Lower A1c for Routine Reviewers: 7.8% vs. 8.6% (p=0.001) Diabetes Data Platform Use Wong et al, 2015 @joyclee
  • 44.
    There is alack of standardization for data visualizations @joyclee
  • 45.
    @joyclee There is alack of standardization for data visualizations
  • 46.
  • 47.
    @joyclee Are they coveringall carbs? Are they carb counting carefully? Are they inputting all the carbs / BG values into the pump? Are they bolusing before or after meals? Can I trust the data? Insulin lasts 2-3 hours There are A Lot of Unmeasured Variables
  • 48.
  • 49.
    Guilt Loss of Control Alienation Obsession Datais not always empowering Isolation @joyclee CSCW 2017, Kaziunas, Ackerman, Lindtner, Lee
  • 50.
  • 51.
    Patients and cliniciansdon’t trust a black box “I don’t know what it’s doing so how can I trust it?” “I wouldn’t give up my DIY AP” @joyclee
  • 52.
    And that doesn’t eveninclude linking this vital diabetes data to the clinical EHR data! @joyclee “Garbage data”
  • 53.
    Lack of User-CenteredDesign for the EHR No culture of human-centered design in Health IT Design without a strategic understanding of what metrics are needed to improve care Build without enough input from users Deploy without iteration and testing Physician Resentment/Anger @joyclee
  • 54.
    Clinical EHR: Acombination of Microsoft Word and Pinterest @joyclee Clinicians are inputting data in unstructured format in the notes Data is being lost and/or underutilized Patient paper questionnaires and the diabetes data are scanned to PDF
  • 55.
    @joyclee No data Trapped data Isolateddata Infrequent data Confusing data Suspicious data Garbage data PDFMediaTab
  • 56.
    How do weaddress these Diabetes Data Barriers? @joyclee
  • 57.
  • 58.
  • 59.
    @joycleeKumar RB etal J Am Med Inform Assoc. 2016 May;23(3):532-7.
  • 60.
  • 61.
    Patients, caregivers, cliniciansand researchers work together to choose care based on best evidence; together they drive discovery as natural outgrowth of patient care; and ensure innovation, quality, safety and value, all in real-time. - C3N Project @joyclee “
  • 62.
  • 63.
    Aim: To decreasethe % of the population with HbA1c ≥ 9% and increase the % of the population with ≥ 0.5% HbA1c interval improvement Preference driven treatment and effective self- management Enhanced registry population management & Pre-visit planning Peer/community support Education/training to support technology use and patient viewing and problem solving with blood glucose data between visits Interventions/toolkits for addressing barriers to adherence Efficient use of technology and data to support care Access to care and regular follow-up Screening for depressionPsychosocial Support Shared decision making Partnership between engaged patients and the care team Effective use of EHR by diabetes team for population management • % of pts. testing ≥4 times/day or using CGM (6/7 days/week) • % of pts. giving 3 or more short-acting boluses/day • % of pts reviewing data between visits • % pts setting, documenting, and reviewing goals • % completed pre-visit planning • % with ≥ 4 visits per year • % of pts with annual CDE/RD/SW visit • % of pts on case mgmt. pathway • % pts screened for depression Developing a Clear Measurable Aim and a Theory of Change Care Process Measures @joyclee
  • 64.
    Local Infrastructure tosupport an LHS Team (Director, Associate Director, Patient Partner/Advisor, Project Manager/Analyst) Patient Engagement (Patient Advisor/Advisory Board, Website/Newsletter) QI interventions (Depression Screening, High-risk Patient Recall, Portal Onboarding, Data Engagement Curriculum) Improving Data/Technology Systems @joyclee
  • 65.
  • 66.
  • 67.
  • 68.
    @joyclee “No one isgoing to use that tool if you can’t BOLD the text!” Rogue commas Tool for Data Input Insulin sensitivity 12AM 90 2:30 AM 110 4 AM 230* 10 AM 160 Patient Instructions Insulin sensitivity 12AM 12AM 90 2:30 AM 110 4 AM 23, 0* 10 AM 160
  • 69.
    @joyclee MediaTab Outcomes that Matter Toolsfor Structured Data Collection Patient Reported Outcomes (Portal Questionnaires) Clinical Interface Redesign Tools for Population Management
  • 70.
    How might our machinelearning colleagues help? @joyclee
  • 71.
    Understand how, when,and what data is desired by different stakeholders to solve the right problems Think ecosystems not apps @joyclee
  • 72.
    Population Level Monitoring Prioritizationof high-risk patients Tools for Seamless Data Acquisition Data Pattern Review Documentation Communication
  • 73.
    Watch patient expertssolve problems @joyclee
  • 74.
    Who’s the real medicalexpert? @joyclee
  • 76.
  • 77.
  • 78.
  • 79.
    21. France 22. Netherlands 23.New Zealand 24. Mexico 25. Croatia 26. South Africa 27. Israel 28. Japan 29. Switzerland 30. Hungary 31. Belarus 29,000+ Int’l 11. Australia 12. Portugal 13. Denmark 14. Germany/Austria 15. Turkey 16. Russia 17. Romania 18. Korea 19. Ireland 20. Brasil 55,000+ worldwide 1. Italy 2. UK 3. Spain 4. Sweden 5. Bulgaria 6. Norway 7. Poland 8. Czech+Slovakia 9. Canada 10. Finland
  • 80.
  • 81.
  • 82.
    How do youget your CGM in the Cloud? @joyclee
  • 83.
    “The DIY ArtificialPancreas” Loop/Loopkit OpenAPS AndroidAPS @joyclee
  • 84.
    Interoperability and choicematter Reduce the mental burden Proactive not reactive Be transparent What should I consider and why? Give access to all of the data @joyclee
  • 85.
    The machine isnot an end. An airplane is not an end: it is a tool. Tools are created to allow you to reach greater goals, and machines should not distract from this pursuit. In fact, you should barely notice that they’re there. @joyclee “
  • 86.
    Jacob Dwyer, AshleyGarrity, Valeria Gavrila, Emily Hirschfeld Dorene Markel, Ram Menon, Amy Ohmer, Lilia Verchichina, Michelle Wichorek, Pediatric Diabetes Team, The Nightscout Foundation, T1D Exchange Acknowledgements @joyclee Joyce Lee, MD, MPH doctorasdesigner.com [email protected] http://www.nightscout.info/ https://www.nightscoutfoundation.org/contribute-data/ AHRQ, R21HS023865 and PCORI Engagement Award (1442-UMich)