Process Optimisation at the Ordnance Survey Great Britain
1 like897 views
The document outlines the optimization process of FME at the Ordnance Survey in Great Britain, aimed at improving efficiency, performance, and maintenance documentation through the use of dynamic schemas and custom transformers. It details the evolution from older versions of FME to a more streamlined approach, resulting in significant reductions in workspace complexity and runtime. The project ultimately facilitated smoother data management and supported the future development of geospatial data infrastructure.
Process Optimisation at the Ordnance Survey Great Britain
- 1. FME Process Optimisation at the Ordnance Survey Great Britain An exercise in best practice [email protected] Principal Consultant FME Certified Professional & Trainer 1Spatial
- 2. Agenda • The Ordnance Survey • The Object Editor Adapter • Project Goals • FME as a Test Harness • Deploying ‘Best Practice’ in FME • Tips and tricks • Project Benefits 2
- 3. Ordnance Survey - Then 3 • The original ‘Board of Ordnance’, MOD of the day • Started survey work in the late 1700s • Comprehensively mapped the south coast to be able to effectively repel invasion • http://goo.gl/02tVFZ Lands End, Mount's Bay, Cornwall *
- 4. Ordnance Survey - Now 4 • Self financing part of the UK government • £120-million-a-year civilian organisation • At the forefront of the digital economy • Remit covers Great Britain (England, Scotland and Wales) • 243,241 sq km / 93,956 sq miles • Integrated data capture for field surveyors, HQ-based editors, and external suppliers *
- 5. Geospatial Data Management System (GDMS) • Seamless data maintenance solution • Intelligent and integrated maintenance data model • Allows separation of product data • Includes automated data validation • Platform for new product development • 650+ data editors • 500+ million features • 4,000 data maintenance jobs per day *
- 7. GDMS – Production Architecture • Includes best of breed COTS components from • 1Spatial • BAE Systems • ESRI • Intergraph • Microsoft • Oracle • Safe Software • Snowflake Software • Scalable infrastructure • Disconnected editing thanks to the ‘Object Editor Adapter’ *
- 8. Object Editor Adapter 8 • Transforms data into a structure usable in the Object Editor map application • FME underpins the transformation • Schema is manipulated into ‘understandable’ structure from the database GML OEA (FME) PGDB *
- 9. Project Goals 9 1. Document process and make maintenance practical • Schema changes pending • Component needed to be better understood 2. Upgrade process to FME 2013 (look beyond 2013) • Built with FME 2009 • Utilise recent FME enhancements 3. Improve efficiency & performance • Remove redundancy
- 10. “If it ain’t broke…” 10 • Huge amount of change in FME since 2009 • Attribute management particular interest • OS needed to unpick the workspace to be able to update it anyway • Workspace opening time was slow, impacting batch run time and FME Server – 3 minutes – c.60mb (most *.fmw don’t exceed a few 100kb) • Quite a few transformers…
- 11. Transformers, transformers everywhere 11 • 943 – Custom Transformer(s) = 255 – AttributeRenamer = 211 – Tester = 150 – AttributeRemover = 87 – AttributeCreator = 51 – ListExploder = 72 – CaseChanger = 21 – ListElementCounter = 18 – AttributeSplitter = 12 – AttributeSetter = 10 – ListIndexer = 9 – AttributeCopier = 8
- 12. What is a ‘Test Harness’? 12 In software testing, a test harness or automated test framework is a collection of software and test data configured to test a program unit by running it under varying conditions and monitoring its behaviour and outputs. Wikipedia
- 13. Using FME as a ‘Test Harness’ 13 • The process had to change but the data must not! • FME to check the old process against the new • Controlled via a spreadsheet, a WorkspaceRunner and Published Parameters GML PGDB OLD PGDB NEW Detect Change Failures: Spreadsheets and Geometry *
- 14. 14
- 15. Dynamic Schema 15 • Introduced around FME 2009 – Existing process didn’t use it • Enable ‘Dynamic’ schema mode – Whatever schema is read, gets written – Decouple schema from format • http://blog.safe.com/2010/02/fmeevangelist54/
- 16. Dynamic Schema 16 • 96 input layers can be reduced to 1! – Magic Transformer = FeatureTypeExtractor – Workspace infinitely less daunting
- 17. Reduce Transformer Duplication 17 • Helped by Dynamic Schema Mode • Try not to duplicate operations
- 18. Reduce Transformer Duplication 18 • ‘Like’ operations carried out on all features – Filter features off to unique processes when needed • 20% of data had a ‘Level’ – 21 CaseChangers used to apply UPPERCASE – Replaced with 1 StringCaseChanger • 69 Custom Transformers with embedded Python used to format Date attributes – All features passed through a StringReplacer to remove hyphens • 2013-01-01 to 20130101
- 20. Custom Transformers 20 • What? • A container for many regular transformers • Allows a common set of tasks to be reused in 1 or many workspaces • When exported becomes an *.FMX file for sharing
- 21. • The FME Store was introduced in 2012 – 2014 FME Store mature with 130+ transformers (and other resources) • http://fmestore.safe.com • http://fmepedia.safe.com/articles/Samples_and_Demos/Custom-Transformer- KMLDiagrammer • It’s increasingly likely your workspaces will contain Store resources • FME now auto connects & downloads any resources required Custom Transformers 21
- 22. • Support for versioning • Ports are not disconnected when renamed – http://blog.safe.com/2013/12/fmeevangelist121/ • Embed/link all instances of a given transformer • Manage input and output schema on Custom Transformers Custom Transformers 2014 22
- 23. • 14 unique custom transformers used 255 times • PGDB had constraints, many tables did not allow null values – Custom transformers tested and removed null attributes – Thanks to Generic mode expose constrained attributes and pass through a PythonCaller to remove the null attributes • FME 2014 Use Custom Transformers Effectively 23
- 24. Schema Mapping 24 • Attribute names are hardcoded into 211 AttributeRenamers in the workspace – Changes to target schema means attribute hunting! – Biggest challenge for ongoing maintenance • 1x SchemaMapper replaces all AttributeRenamers – Non-FME user can alter the process by editing AttributeMapping.csv
- 25. Documentation 25 • Always document your Workspace • The OEA process was documented – But didn’t include detailed explanations ‘why’ • You should be able to read a Workspace and understand it – Highlight dependencies and use XML notation for naming
- 26. • In 2014 canvas annotations taken to the next level Documentation – Self contained 26
- 28. Benefits 28 Before After Improvement Time to open (from local disk) 2mins 12secs 31secs 1min 41secs File size 56mb 13.8mb 75% reduction Lines of machine written code 450,332 111,334 75% reduction Number of transformers 943 336 64% fewer Sample run time 1hr 7mins 48mins 19mins • Current, documented and ‘owned’ by the Ordnance Survey • Robust production extraction routine • Simpler to maintain and change – Some schema changes possible without opening FME!
- 29. 29 [email protected] Principal Consultant FME Certified Professional & Trainer 1Spatial