![SQL
Distinct Values: SQL
A SQL query will return all
records, including:
• Departments you cannot see
• Template records
SELECT
DISTINCT ObjectName
FROM Objects
SELECT
ObjectName, COUNT(*)
FROM Objects
GROUP BY ObjectName
[HAVING COUNT(*) = 1]
[HAVING COUNT(*) > 1]](https://image.slidesharecdn.com/dataquality2020-200619172811/85/How-Clean-is-your-Database-Data-Scrubbing-for-all-Skill-Sets-21-320.jpg)
![SQL
My Stored Procedure
Stored Procedure
• @ColumnID = Identifies the field
• Get the ColumnID from the
Data Dictionary
• @PK = Primary key for the record
• @NewValue = the value you want
• @LoginID = your username
EXECUTE [dbo].[MLM_UpdateFieldValue]
@ColumnID = 1243, @PK = 273469,
@NewValue = ‘Gift of John Doe’,
@LoginID = ‘cpetrovay’;
EXECUTE [dbo].[MLM_UpdateFieldValue]
@ColumnID = 1228, @PK = 273469,
@NewValue = ‘Loaned Object’,
@LoginID = ‘cpetrovay’;](https://image.slidesharecdn.com/dataquality2020-200619172811/85/How-Clean-is-your-Database-Data-Scrubbing-for-all-Skill-Sets-48-320.jpg)
![SQL
My Stored Procedure
Process:
• Truncates new value if too long
• Looks up authority key values
• Updates only when value changes
• Tracks change in Audit Trail
Available:
• github.com/cpetrovay/TMS_UpdateField_SP/
EXECUTE [dbo].[MLM_UpdateFieldValue]
@ColumnID = 1243, @PK = 273469,
@NewValue = ‘Gift of John Doe’,
@LoginID = ‘cpetrovay’;
EXECUTE [dbo].[MLM_UpdateFieldValue]
@ColumnID = 1228, @PK = 273469,
@NewValue = ‘Loaned Object’,
@LoginID = ‘cpetrovay’;](https://image.slidesharecdn.com/dataquality2020-200619172811/85/How-Clean-is-your-Database-Data-Scrubbing-for-all-Skill-Sets-49-320.jpg)
How Clean is your Database? Data Scrubbing for all Skill Sets
- 1. Guest Presentation HOW CLEAN IS YOUR DATABASE? DATA SCRUBBING FOR ALL SKILL SETS (2020 EDITION) CHAD PETROVAY, TMS ADMINISTRATOR THE MORGAN LIBRARY & MUSEUM
- 2. Data Quality Data quality is a measure of the condition of data based on factors such as accuracy, completeness, consistency, reliability and whether it’s up to date.
- 3. ? What fields do you have that contain data quality issues?
- 4. ? What resources do you have to scrub your data?
- 5. What is your personal skill level? Power User Uses the TMS UI; has expanded rights, but not full rights. SQL Expert Wait? TMS has a UI? Nah, I’ll just script it in the database. Administrator Full rights in TMS and access to DB Config.
- 6. ? What is your data worth?
- 7. The costs of poor data quality are between 20% and 35% of the operating revenue of the average organization LARRY ENGLISH
- 8. PREVENTION “An ounce of prevention is worth a pound of cure” -Benjamin Franklin
- 9. Institution Value your Data • Understand the costs • Make data-driven decisions • Be a champion for accurate data
- 10. Institution Standards • Establish the rules for data entry • Conceptualize terms and authority values Prevents • Data entry errors • Formatting errors • Inconsistency • Creativity
- 11. Institution Training • Makes the system approachable • Improves user efficiency Prevents • Data entry errors • Unmanaged data silos (Excel)
- 12. Power User Spell Check Uses the Spelling and Grammar engine in Microsoft Office. Prevents: • Typographical errors • Misspellings • Punctuation errors • Grammatical errors
- 13. Power User Function Keys Reduces keystrokes when entering repeated text. Prevents: • Typographical errors • Misspellings • Punctuation errors • Grammatical errors • Formatting errors
- 14. System Admin Customize Field Labels • Clarify field usage • Makes system more intuitive • Align field labels with your institutional lingo Prevents • Confusion
- 15. System Admin Customize Field Labels In Database Configuration 1. Manage » Tables/Columns 2. Find the table 3. Find the column (i.e. field) 4. Right-click » Edit 5. Change Local Column Name
- 16. System Admin Security Groups • If your institution does not use a field, then restrict access • Restrict control of authority values to select power users • Text Types & Term Types Prevents • Populating obsolete fields • Creativity
- 17. DATA PROFILING “Mistakes are the portals of discovery.” -James Joyce
- 18. System Admin Usage Report In TMS Module 1. Maintenance » Authorities » Others 2. Usage Report 3. After report generates: • Browse • Print • Edit as RTF • Save As RTF
- 19. System Admin Frequency Report In Database Configuration 1. Manage » Tables/Columns 2. Find the table 3. Find the column (i.e. field) 4. Right-click » Frequency 5. Save TXT file
- 20. Power User Crystal Reports In TMS Module 1. Report » Reports 2. Find report by name 3. Click Run When creating the report: 1. Add formula “reporttype” 2. "NOTLINKED""NOTLINKED"
- 21. SQL Distinct Values: SQL A SQL query will return all records, including: • Departments you cannot see • Template records SELECT DISTINCT ObjectName FROM Objects SELECT ObjectName, COUNT(*) FROM Objects GROUP BY ObjectName [HAVING COUNT(*) = 1] [HAVING COUNT(*) > 1]
- 22. Power User Distinct Values: Excel Pivot Table Is the field in a List View? Can you export your result set? 1. Export into Excel 2. Copy column into a new sheet 3. Create column “Count” 4. Fill “Count” with 1 5. Create a Pivot Table Tutorial • bit.ly/3d4M8Ou
- 23. Power User OpenRefine Install OpenRefine • Download at www.openrefine.org • Extract archive • Execute openrefine.exe • Opens in your web browser Requires Java • www.java.com/en/download/
- 24. Power User OpenRefine: Facets • A Facet shows a value distribution • Filter records • Batch change • Facets • Word Facet • Text-Length Facet • Null / Empty String / Blank Facets
- 25. Power User OpenRefine: Duplicates • Facets • Duplicates Facet • Facet by Star • Facet by Flag • Export to Excel
- 26. Power User DataCleaner Install Community Edition • Download at www.datacleaner.org • Extract archive • Execute DataCleaner.exe Requires Java • www.java.com/en/download/
- 27. Power User DataCleaner: DataStore If your server uses NT Authentication: • Add SQL user to the database Create a datastore in DataCleaner: 1. Select Microsoft SQL Server 2. Supply details • Hostname = Server name • Database = TMS • Username & Password
- 28. Power User DataCleaner: New Job • Navigation pane • Datastore elements • Library of actions • Canvas
- 29. Power User DataCleaner: Building Job • Drag database elements and components onto the canvas • Best to drag columns instead of full tables/views • Use filter to exclude NULLs and empty strings
- 30. Power User DataCleaner: Results • String Analysis • Row Count • Null/Blank Count • All upper/lower count • Char/Word count • Max/Min/Avg char count • Max/Min/Avg space count • Max/Min word count • Click arrow for details
- 31. Power User DataCleaner: Results • Value Distribution • Total count • Distinct count • List of distinct values (except uniques) • Graphical rank-size of distinct values • Click arrow for details
- 32. Power User DataCleaner: Results • Pattern Finder • A = Uppercase letter • a = Lowercase letter • # = Number • ? = AlphaNumeric • Graphical rank-size of distinct patterns • Click arrow for details
- 33. Power User Data Quality Services (DQS) • Knowledge Base • Projects • Cleansing • Matching • Bundled with SQL Server • Enterprise Edition • Developer Edition • Only works with local databases
- 34. PLANNING “To achieve great things, two things are needed: a plan, and not quite enough time.” –Leonard Bernstein
- 35. Institution Human Capital Human capital is essential for any data scrubbing project. • Colleagues • Interns • Volunteers
- 36. Power User Project Management • Record projects • Plan future projects • Track progress • Provide metrics for administration
- 37. Power User Cheat Sheets • Training tool • Project specific • Simplifies access to standards
- 38. DATA SCRUBBING “Cleaning and organizing is a practice not a project.” -Meagan Francis
- 39. The Three Modes Human Middleware Direct human contact with UI Usually Record-by-Record Labor intensive Automation Requires additional tools/services/platforms Steeper learning curve Artificial Intelligence SQL Script Change one or more records through the back-end Requires intimate knowledge of database structure
- 40. Human Middleware Finding Records by Pattern • Query using wildcards: • single character (?) • multi-character (*) • Wrap sequences with double quotes Format TMS Search (646) 733-2239 “(???) ???-????” 646.733.2239 *???.???.????* +44 (0)207379 8188 +* (510) 652-8950 ext 223 “* ext*” Chad M. “* ?.” Cheryl & Edward *&* Cheryl and Edward “* and *”
- 41. Human Middleware Search and Replace (String) In Objects Module 1. Maintenance » Database » Search and Replace 2. Select Module/Table/Column 3. Provide search and replace terms 4. Review results • Replace All • Replace • Skip
- 42. System Admin Search and Replace (Thesaurus) In Database Configuration 1. Edit » Search and Replace » Linked Thesaurus Terms 2. Click Zoom button (…) to find source term 3. Click Zoom button (…) to find target term 4. Click OK and confirm
- 43. Human Middleware Merge Constituents Utility In Plugins folder 1. Search for duplicate constituents 2. Select candidates from the suggestions 3. Click Next Feature Idea: Constituent Packages!!
- 44. Human Middleware Merge Constituents Utility 4. Set Target record • Right-click » Merge to this 5. Edit data in the columns of the grid 6. Go section by section and select the data to keep 7. Ready to merge? • File » Merge 8. Save an XML file
- 45. SQL Updating with SQL • Know the system • Test the SQL script in a sandbox environment first • Backup your database before running SQL script • Consider converting frequently used scripts into Stored Procedures • Gallery Systems may not be able to provide support
- 46. SQL Finding Records with Patterns • Use a LIKE Statement • Query using wildcards: • single character (_) • multi-character (%) Format TMS Search (646) 733-2239 ( _ _ _ ) _ _ _ - _ _ _ _ 646.733.2239 % _ _ _ . _ _ _ . _ _ _ _ % +44 (0)207379 8188 +% (510) 652-8950 ext. 223 % ext% Chad M. % _. Cheryl & Edward %&% Cheryl and Edward % and %
- 47. SQL Excel Trick If you have data in Excel 1. Create a SQL script using a CONCATENATE formula 2. Copy the formula down the column 3. Select and copy the column 4. Paste the content in SSMS 5. Execute
- 48. SQL My Stored Procedure Stored Procedure • @ColumnID = Identifies the field • Get the ColumnID from the Data Dictionary • @PK = Primary key for the record • @NewValue = the value you want • @LoginID = your username EXECUTE [dbo].[MLM_UpdateFieldValue] @ColumnID = 1243, @PK = 273469, @NewValue = ‘Gift of John Doe’, @LoginID = ‘cpetrovay’; EXECUTE [dbo].[MLM_UpdateFieldValue] @ColumnID = 1228, @PK = 273469, @NewValue = ‘Loaned Object’, @LoginID = ‘cpetrovay’;
- 49. SQL My Stored Procedure Process: • Truncates new value if too long • Looks up authority key values • Updates only when value changes • Tracks change in Audit Trail Available: • github.com/cpetrovay/TMS_UpdateField_SP/ EXECUTE [dbo].[MLM_UpdateFieldValue] @ColumnID = 1243, @PK = 273469, @NewValue = ‘Gift of John Doe’, @LoginID = ‘cpetrovay’; EXECUTE [dbo].[MLM_UpdateFieldValue] @ColumnID = 1228, @PK = 273469, @NewValue = ‘Loaned Object’, @LoginID = ‘cpetrovay’;
- 50. MONITORING “Without a systematic way to start and keep data clean, bad data will happen.” -Donato Diorio
- 51. Human Middleware Saved Queries • Save time by saving your periodic review queries. • User has to initiate query
- 52. Human Middleware Audit Trail Report • Review changes to the database • Identify where to provide additional training • User has to run the report
- 53. Automation TMS Alerts • Notifies user when predefined criteria is met • User has to regularly access TMS • Setup by SQL Expert
- 54. Automation Database Mail • Sends email to user when predefined criteria is met • Requires configuration in SQL Server • Setup by SQL Expert
- 55. Automation SSRS Subscription • Sends email to user when predefined criteria is met • Requires configuration in SSRS Server • Setup by Report Writer
- 56. FINAL THOUGHTS “Data that is loved tends to survive.” –Kurt Bollacker
- 57. “While few things in life are guaranteed, it is safe to say that not addressing data quality issues this year means you’ll be facing the same issues next year, likely on a larger scale.” -BO CRADER (sgENGAGE)
- 58. Data scrubbing goes on as long as it has to. THE 7TH RULE OF THE DATA SCRUB
- 59. CHAD PETROVAY TMS ADMINISTRATOR, THE MORGAN LIBRARY & MUSEUM [email protected] Q&A