Uncover Untold Stories in Your Data: A Deep Dive on Data Profiling
- 1. T24 Special Topics 5/5/16 15:00 Uncover Untold Stories in Your Data: A Deep Dive on Data Profiling Presented by: Catherine Cruz Agosto Shauna Ayers Availity, LLC Brought to you by: 350 Corporate Way, Suite 400, Orange Park, FL 32073 888-‐-‐-‐268-‐-‐-‐8770 ·∙·∙ 904-‐-‐-‐278-‐-‐-‐0524 -‐ [email protected] -‐ http://www.stareast.techwell.com/
- 2. Catherine Cruz Agosto Availity, LLC Catherine Cruz Agosto found that her software engineering experience at Baxter Healthcare and Boeing-‐subsidiary Insitu provided an excellent foundation for finding more effective and user-‐friendly approaches to complex technical problems. Catherine has developed more efficient and innovative data quality testing solutions at healthcare intermediary Availity, expanding their automated data quality testing processes to accommodate diverse and dissimilar data sources, facilitating analysis, testing, and controls for data integration, analytics, and healthcare data reporting. Shauna Ayers Availity, LLC Shauna Ayers has been untangling the Gordian knots of IT systems for more than seventeen years, analyzing data systems and testing both software and data quality in the manufacturing, medical device, and healthcare industries. Shauna found her passion in developing creative solutions for the analysis and testing of sensitive and highly-regulated data sets at industry leaders such as Blue Cross Blue Shield of Florida (now Florida Blue), Vistakon (a subsidiary of Johnson & Johnson), and Availity.
- 3. Uncovering the Untold Stories of Your Data A Deep Dive on Data Profiling By Catherine Cruz Agosto and Shauna Ayers
- 4. Overview • Define Data Profiling • Importance of Profiling • Profiling in action – Tools? – Lifecycle and need – Classification • Conclusion
- 5. Define Data Profiling • Myths – You need to buy a profiling tool – Profiling tools do all the work to get the data metrics – Profiling is one-time activity • Definitions – Data profiling: The ongoing process of examining data from one or multiple sources (i.e. databases, files, etc.) and collecting meaningful metrics and statistics to gain a greater understanding of the data and its quality in its appropriate context.
- 6. Importance of Profiling • Communicate Health and Status more effectively • Get people of different roles on the same page • Identify Revenue Opportunities • Identifying Operational Gaps and Risks
- 7. Profiling in action: Tools? • Tools: Uses – Visibility – Monitoring – Trends • Tools: To buy or not to buy – Reminder: You do not need a fancy tool in order to profile – Common tools: IDQ, Datamartist, Microsoft Data Profiling Task • All tools have some sort of limitations • Can get expensive – Creating your own tools • Seems more time consuming up-front • More control/ less limitations compared to pre-bought In the end, what matters is how you use the data.
- 8. Profiling in action: Lifecycle and Need • Dimensions of Profiling – Properties of Data (Quality) – Movement of Data (Flow) – Usage of Data (Business Need)
- 9. Profiling in action: Lifecycle and Need continued • Profiling for planned changes – I.E. New projects, enhancements • Profiling for unplanned changes – Changes within the data itself – Monitoring
- 10. Profiling in action: Classification • Datasets • Velocity and Functional Flow • Properties • Relationships • Tolerances • Business Value
- 11. Conclusion • Profiling is dynamic. • Technology can improve profiling. • Profiling includes qualitative analysis that cannot be done solely via machine. • Profiling can help improve both strategic and operational outcomes… if you do it right.