How to Optimize GIS Workflows
1 like4,582 views
The document provides an overview of new transformers in FME that help optimize GIS workflows by simplifying attribute and data validation tasks. The AttributeManager allows consolidated handling of attribute tasks like creation, renaming, copying, and validation. The AttributeValidator performs validation tests on attributes and outputs validation messages. The FeatureWriter enables writing features in workflows to avoid chaining workspaces and support post-processing like notifications and automation.
![Attribute Validation Output
Output is:
➢ Summary Attribute:
_fme_validation_message
➢ List of all messages:
_fme_validation_message_list{}
Attribute 'CodePrefix' with value 'ABE' fails check for Matches Regular Expression
'[ABCD]{3}'
{0} Attribute 'CodePrefix' with value 'ABE' fails check for Matches Regular Expression
'[ABCD]{3}'
{1} Attribute 'num_measures' with value '12' fails check for in Range '[0,10]'
{2} Attribute 'CodePrefix' with value 'ABE' fails check for in 'ABC,ABD,TXU,TXV'](https://image.slidesharecdn.com/howtooptimizegisworkflows-160309195442/85/How-to-Optimize-GIS-Workflows-16-320.jpg)
How to Optimize GIS Workflows
- 1. How to Optimize GIS Workflows With Mark Stoakes & Steve MacCabe
- 3. Connect Virtually Any Format with FME
- 4. FME & GIS can do amazing things Here is a taste of what’s possible. Manage your attributes Validate your attributes Distribute a Validation Report Automate the process
- 6. Attribute Management Why another attribute handling transformer? ➢ Because there are already too many … plus we needed a couple other operations Goal: consolidate the attribute management transformers Doomed transformers: AttributeCreator AttributeCopier AttributeRenamer ExpressionEvaluator StringConcatenator AttributeKeeper AttributeRemover Plus… …many string operations: case change trim/pad substring/regex One transformer to rule all your attributes
- 7. ● Create attributes ● Rename attributes ● Copy attributes ● Remove attributes ● Set attribute values ● Re-order attributes ● Concatenate attributes (Text Editor) ● Arithmetic expressions (Arithmetic Editor) ● Set conditional values (Conditional Value) ● Reference earlier attributes AttributeManager
- 8. AttributeManager creates an opportunity for more focused and simplified attribute management. #ThanksFME2016
- 9. Let’s Go - Demo! AttributeManger: Lets you organize your attributes in your workflows
- 11. Attribute Validation – why? If you distribute poor quality data then you've amplified the data issue. If you distribute good quality data you've amplified the benefits. Data validation is key to many FME workflows…
- 12. Attribute Validation – Standards Bunch of ISO standards – ISO19xxx Mostly replaced by - ISO 19157 Also CAD & Data Model Standards
- 13. Attribute Validation – FME Tools Attribute Validation – Key Tools ● Tester and TestFilter ● AttributeCreator (conditional tests) ● AttributeClassifier ● AttributeRangeFilter & AttributeFilter ● NullAttributeMapper ● DuplicateRemover ● StringSearcher ● StatisticsCalculator
- 14. AttributeValidator... Goals: Simplify validation workflows (Reduce number of transformers) Be more confident about the data you’re loading into a destination dataset ...say goodbye to bad data #ThanksFME2016
- 15. Attribute Validation Some of the validation tests available: ➢ Attribute type (integer, float, char, xml, json, etc.) ➢ In – either a list or range – good for domain validation ➢ Regular Expressions ➢ Unique ➢ Not Null ➢ …and more…
- 16. Attribute Validation Output Output is: ➢ Summary Attribute: _fme_validation_message ➢ List of all messages: _fme_validation_message_list{} Attribute 'CodePrefix' with value 'ABE' fails check for Matches Regular Expression '[ABCD]{3}' {0} Attribute 'CodePrefix' with value 'ABE' fails check for Matches Regular Expression '[ABCD]{3}' {1} Attribute 'num_measures' with value '12' fails check for in Range '[0,10]' {2} Attribute 'CodePrefix' with value 'ABE' fails check for in 'ABC,ABD,TXU,TXV'
- 17. Let’s Go - Demo! AttributeValidator: Lets you scrutinize your attributes and simplify your data validation workflows
- 18. Q&A Chat your questions to us
- 19. Writing Features ...a different approach
- 20. Traditional vs. state-of-the-art... Write data with a writer Write data with a FeatureWriter
- 21. Write the conclusion to your data process Write data in the middle of workflows to avoid chaining workspaces. Write the summary of your data output. #ThanksFME2016 Use the FeatureWriter to allow post-processing freedom. #FreeYourData
- 22. Some Easy Wins: • Create summary reports – Email an attachment • Copy data after writing – FTP results – Load into Dropbox
- 23. Automation Workflows • Quality checking • Database operations • Database validation • FME Server Notification
- 24. Transformation ...transform your data A new pattern for database workflows Transform your database workflows SQLCreator ...drop constraints FeatureReader ...read your data Transformation ...transform your data FeatureWriter ...write to database SQLExecutor ...enable constraints SQLExecutor ...validate/index/unique
- 25. A new pattern for FME Server notification workflows FeatureReader & FeatureWriter simplies you workspace
- 26. Let’s Go - Demo! FeatureWriter: Lets you post- process your output data
- 27. Automation
- 28. FME Server Overview FME Engine workhorse and runs FME Workspaces Automate Deliver Data Download Data Love FME ! ScaleReal Time Data
- 29. Let FME Server do the work Examples of things we can automate: ● Directory watcher ● Run the workspace through a notification ● Data Delivery ○ Dropbox ○ email Let’s Go - Demo!
- 30. Summary
- 31. Take advantage of our new transformers.
- 32. Free Stuff There are too many to list so be sure to check out www.safe.com for more. Knowledge Articles ● AttributeManager ● AttributeValidator Blog Posts ● FeatureWriter ● AttributeManager Free FME Training ● FME Desktop Basic: March 8-9 ● FME Server: March 15-16 ● FME Desktop Advanced: March 22-23
- 33. 60+ CITIES April - June
- 34. Poll Question Which of today's tasks do you see yourself performing on your GIS data?
- 35. Q&A Chat your questions into us
- 36. Thank you! Get a free trial of FME at www.safe.com
- 37. Resources: Attribute Validation Original PR: 8945 Knowledge Center: Validate your Data's Attributes with the AttributeValidator Transformer Webinar 2015: Data Validation Victories: Tips for Better Data Quality: Data Validation & Quality Assurance WWTour:FME & Data Validation: http://www.slideshare.net/SafeSoftware/fme-data-validation WWTour 2014: Validation and Quality Control with FME & FME Server and also at SlideShare Validating your data before loading it into a database: video by Robyn FME 2014 UC - Colonial Pipeline: CAD Data Validation using FME Automated Quality Control: ISo 9000:2000 Quality Measures ISO 19157/19131 Metria 2014 Berk Bayer 2014 Application of FME to Maintain Quality Control (QC) over Standardized Spatial Data FME WWTour 2013 – Introducing Geometry Validator
- 38. Resources: AttributeManager Knowledge Center: Control Your Data's Attributes with the AttributeManager Transformer