Download as PDF, PPTX
Uploaded byGeorge Percivall
Geospatial Temporal Open Standards for Big Data from Space (BiDS2014)
The document discusses the Open Geospatial Consortium (OGC) standards relevant to big data and geospatial-temporal information, particularly in the context of Earth observation (EO) systems. It outlines various service architectures, discovery services, and protocols under configuration control, as well as the implications for interoperability and big data analytics. Additionally, it highlights the performance testing of OGC services in cloud environments, with a focus on web mapping and processing services.
More Related Content
Similar to Geospatial Temporal Open Standards for Big Data from Space (BiDS2014)
![Governance, Deployment & Methodologies for Agentic Automation [2/3]](https://cdn.slidesharecdn.com/ss_thumbnails/agenticcommunityseries-day2-251112135038-a386476d-thumbnail.jpg?width=640&height=640&fit=bounds)
Geospatial Temporal Open Standards for Big Data from Space (BiDS2014)
- 1. ® OGC BigData and Open Geospatial-Temporal Information Standards George Percivall OGC Chief Engineer 2014 Conference on Big Data from Space (BiDS '14) Jointly organised by ESA, SatCen, JRC 12-14 November 2014, ESRIN, Frascati, Italy © 2014 Open Geospatial Consortium
- 2. OGC ® OGCServices Architecture • Introduction to OGC Standards • …Service Interfaces and Encodings standards • …Grouped into semantically similar sets
- 3. OGC ® OGCServices Architecture WMS WFS WMTS Simple Features Access Access Services Geospatial Feature Data Geospatial Browse/Maps WCS Geospatial Coverage Data
- 4. OGC ® OGCServices Architecture WMS WFS WMTS Simple Features Access Access Services Geospatial Feature Data Geospatial Browse/Maps WCS Geospatial Coverage Data SPS SensorML O&M SOS Sensors Puck Sensor Web Enablement Discover Task Access
- 5. OGC ® OGCServices Architecture Processing Services TJS WPS WCPS OpenMI Other Data WMS WFS WMTS Simple Features Access Access Services Geospatial Feature Data Geospatial Browse/Maps WCS Geospatial Coverage Data SPS SensorML O&M SOS Sensors Puck Sensor Web Enablement Discover Task Access
- 6. OGC ® OGCServices Architecture Processing Services TJS WPS WCPS OpenMI Other Data Discovery Services Geospatially Enabled Metadata CSW OpenSearch Geo ebRIM WMS WFS WMTS Simple Features Access Access Services Geospatial Feature Data Geospatial Browse/Maps WCS Geospatial Coverage Data SPS SensorML O&M SOS Sensors Puck Sensor Web Enablement Discover Task Access
- 7. Other Services Workflow,Alerts OGC ® OGC Services Architecture Processing Services TJS WPS WCPS OpenMI Other Data Discovery Services Geospatially Enabled Metadata CSW OpenSearch Geo ebRIM WMS WFS WMTS Simple Features Access Access Services Geospatial Feature Data Geospatial Browse/Maps WCS Geospatial Coverage Data SPS SensorML O&M SOS Sensors Puck Sensor Web Enablement Discover Task Access
- 8. OGC Services Architecture NetCDF GMLJP2 IndoorGML GeoSparql Other Services Workflow, Alerts OGC ® SLD WaterML SE GeoPackage Data Models WMC and Encodings FE GML GeoXACML KML CityGML OpenGeoSMS Processing Services TJS WPS WCPS OpenMI Other Data Discovery Services Geospatially Enabled Metadata CSW OpenSearch Geo ebRIM WMS WFS WMTS Simple Features Access Access Services Geospatial Feature Data Geospatial Browse/Maps WCS Geospatial Coverage Data SPS SensorML O&M SOS Sensors Puck Sensor Web Enablement Discover Task Access
- 9. OGC Services Architecture Visualization / Decision Tools and Applications NetCDF GMLJP2 IndoorGML GeoSparql Other Services Workflow, Alerts OGC ® GeoAPI OpenLS SLD WaterML SE GeoPackage Data Models WMC and Encodings FE GML GeoXACML KML CityGML OpenGeoSMS Processing Services TJS WPS WCPS OpenMI Other Data Discovery Services Geospatially Enabled Metadata CSW OpenSearch Geo ebRIM WMS WFS WMTS Simple Features Access Access Services Geospatial Feature Data Geospatial Browse/Maps WCS Geospatial Coverage Data SPS SensorML O&M SOS Sensors Puck Sensor Web Enablement Discover Task Access
- 10. ESA HMA andOGC • Foreword by Mark Reichardt, OGC CEO and President – Comprehensive guide to the use of international standards to enable a European EO system of systems. – “Indispensable resource for organisations with EO assets that wish to connect to and build a local to global system of systems” • OGC Working Groups of major importance to HMA – Service Architecture – Sensor Web Enablement – Catalogue – Metadata – Coverages – Compliance OGC ® Thomas Usländer (Fraunhofer IOSB) Yves Coene (Spacebel s.a.) Pier Giorgio Marchetti (ESA) © 2014 Open Geospatial Consortium 10
- 11. ESA HMA Protocolsunder Configuration Control http://wiki.services.eoportal.org/tiki-index.php?page=HMA+Configuration+Management+Table Service Standard Status Collection Metadata OGC 11-035r1, EO Collection and Service Discovery using the ebRIM Application Profile of CSW 2.0, version OGC ® © 2014 Open Geospatial Consortium 11 1.0, 26/03/2013. OGC Best Practice Collection& Service Discovery OGC 13-084, I15 (ISO19115 Metadata) Extension Package of CSW ebRIM profile, version 1.0, 28/04/2014 OGC Implementation Specification OGC 10-032r8, OpenSearch Geo and Time Extensions, version 1.0.0, 14/04/2014. OGC Implementation Standard OGC 13-026r5, OpenSearch Extension for Earth Observation, DRAFT 1.0.0, 29/10/2014 OGC SWG Internal document Earth Observation Product Metadata OGC 06-080r4, GML Application Schema for EO Products, Version 1.0.0, 25/02/2010 OGC Profile OGC 10-157r3, Earth Observation Metadata Profile of Observations and Measurements,Version 1.0, 12/06/2012 OGC Implementation Standard Product Discovery OGC 06-131r6, EO Products Extension Package for ebRIM Profile of CSW 2.0, Version 1.0.0, 10/02/2010 OGC Implementation Standard OGC 10-032r8, OpenSearch Geo and Time Extensions, version 1.0.0, 14/04/2014. OGC Implementation Standard OGC 13-026r5, OpenSearch Extension for Earth Observation, DRAFT 1.0.0, 29/10/2014 OGC SWG Internal document Order OGC 06-141r6. Ordering Services Framework for Earth Observation Products Interface Standard, Version 1.0.0, 09/01/2012 OGC Implementation Specification OGC 13-042, RESTful Encoding of Ordering Services Framework for Earth Observation Products, version 1.0, 28/04/2014. OGC Best Practice Feasibility Analysis OGC 10-135, Sensor Planning Service Interface Standard 2.0 Earth Observation Satellite Tasking Extension, version 2.0, 28/03/2011 OGC Implementation Standard OGC 13-039, OpenSearch Extension for Earth Observation Satellite Tasking, OGC Best Practice OGC 14-012r1, RESTful encoding of Sensor Planning Service for Earth Observation Satellite Tasking, version OGC Best Practice 1.0.0, 17/07/2014. Web Map Service (WMS) OGC 07-063r1 OpenGIS WMS EO Application Profile Version 0.3.3, 05/11/2009 OGC Best Practice Web Coverage Service (WCS) OGC 09-110r4 WCS 2.0 Interface Standard - Core, Version 1.0.0, 10/07/2012. OGC 09-146r2 GML 3.2.1 Application Schema - Coverages (1.0.1) OGC 09-147r3 WCS 2.0 Interface Standard - KVP Protocol Binding Extension (1.0.1) OGC 09-149r1 WCS 2.0 Interface Standard - XML/SOAP Protocol Binding Extension (1.0) OGC 09-148r1 WCS 2.0 Interface Standard - XML/POST Protocol Binding Extension (1.0) OGC 09-153 WCS 2.0 Overview: Core and Extensions (Best Practice) OGC 10-164, OGC GML 3.2.1 AS – Coverages – CF NetCDF Coverage Encoding Extension, version 2.0.0 OGC 12-108, OGC GML 3.2.1 AS – Coverages – GML JPEG2000 Coverage Encoding Extension, version 0.0.4. OGC Interface Standard OGC 12-100r1 OGC GML 3.2.1 Application Schema – Coverages – GeoTIFF Coverage Encoding Profile, v.1.0 OGC Interface Standard OGC 10-140r1 OGC Web Coverage Service 2.0 Interface Standard - Earth Observation Application Profile OGC Interface Standard v.1.0.0, 26/02/2014. Web Processing Service (WPS) No standards under configuration control Download Service OGC 13-043, Download service for EO Products, Version 1.0, 31/01/2014. OGC Best Practice Identity OGC 07-118r9, User Management Interfaces for Earth Observation Services, Version 1.1, 28/04/2014. OGC Best Practice Management !
- 12. OGC ® OGCBig Data Working Group • Open forum on how to use OGC standards for Big Data interoperability, access, and especially analytics – Encourage collaborative development among participants representing many organizations and communities, – Ensure appropriate liaisons to other Big Data relevant working groups, both inside and outside OGC. • Big Data agenda – September 2014 – GeoWave Introduction and Overview (NGA) – Well Known Text for Accumulo (NGA) – Spatio-Temporal Data Cube Analytics (Jacobs Univ.) – Elasticity for WMS Hosted in a Cloud (Amazon) – NOAA Big Data Partnership RFI (NOAA) http://www.opengeospatial.org/projects/groups/bigdatadwg © 2014 Open Geospatial Consortium 12
- 13. Discrete Global GridSystems OGC ® Slide Source: Matt Purss, Geoscience Australia ANZLIC National Nested Grid*** SCENZ-Grid by CSRIO for NZ Earth System Spatial Grid by China Snyder Grid e.g. PYXIS *** See BIDS 2014 paper: Iterating Petabyte-scale Earth Observation processes in the Australian Geoscience Data Cube, Simon Oliver (Geoscience Australia)
- 14. Discrete Global Grid System (DGGS) Standards Working Group (SWG) • Develop common criteria that will define conformant DGGSs – Considering Goodchild criteria • Develop conceptual standard to facilitate data fusion between DGGSs using OGC Standards – to make them interoperable – with conventional and other DGGS data – to standardize operations on them • Engage stakeholders to encourage new use cases and adoption of interoperability through DGGSs http://www.opengeospatial.org/projects/groups/dggsswg OGC ®
- 15. OGC ® Bigdata in OGC Testbeds • OGC Interoperability Program – Prototyping unites users and industry in accelerating interoperability – 85 initiatives since 1999: Testbeds, Pilots, Experiments • OGC Testbeds with Big Data from space – OWS-8: Observation Fusion – OWS-9: OWS Innovations – OGC Testbed 10: Open Mobility and Cloud – OGC Testbed 11: Urban Climate Resilience • Highlights discussed in the following categories – Maps; Coverages; Performance © 2014 Open Geospatial Consortium 15
- 16. OGC ® Mapsand Big Data • Web Map Tiling Service (WMTS) – WMTS is WMS tuned for high velocity maps – Serves map tiles with predefined content, extent, and resolution – TileMatrixSet: collection of tile matrices defined at different scales • OWS-9 evaluated diverse raster map tiling schemes – WMTS, Tile Map Service (OSGeo), MBTiles, TileCache (MetaCarta), and various others • Recommendations forwarded to OGC Standards Program. – Basis for “WMTS – Simple” based on limited TileMatrixSet – Revised WTMS standard nearing completion in SWG OWS-9 Map Tiling Methods Harmonization (Editor Joan Maso, OGC Doc 12-157) © 2014 Open Geospatial Consortium 16
- 17. WMTS Example: Severalsources, Same TileMatrix a.tile.openstreetmap.org mt1.google.com otile1.mqcdn.com/tiles/1.0.0/map www.ogc.uab.cat/SITiled/GeoEye_Haiti_M OGC ® Copyright © 2013 Open Geospatial Consortium tile.opencyclemap.org/cycle ecn.t0.tiles.virtualearth.net X=9798 Y=14664 X=9799 Y=14664 X=9800 Y=14664 X=9798 Y=14665 X=9799 Y=14665 X=9800 Y=14665
- 18. OGC ® OWS-8Coverages as Big Data • WCS 2.0 Earth Observation Application Profile (EO-AP) – Build on WCS 1.0 EO-AP developed in ESA’s HMA project. • …to include data models for: – 2-D Coverage with latitude and longitude axes, e.g., a hyperspectral satellite scene. – Subset of a 2-D satellite scene (either radar or multispectral) – Time series of co-registered 2-D satellite scenes – Stitched Mosaic as a 2-D horizontal for several co-referenced non-overlapping datasets – Dataset Series as a collection of coverages; • …supporting these data products: – MODIS – ENVISAT MERIS L3 demonstration products (http://earth.eo.esa.int/level3/) – ENVISAT ASAR Wide Swath – Limb-scanning instruments (e.g. Microwave Limb Sounder) typically produce vertical profiles, – Narrow-swath instruments (e.g., CALIPSO) produce vertical cross-sections. • …and these encodings: – GeoTIFF, netCDF / CF-netCDF, JPEG2000, HDF-EOS • With a WCS 2.0 EO-AP Compliance Test and Reference Implementation OWS-8 WCS 2.0 EO Application Profile ER (11-096) OWS-8 WCS 2.0 EO Application Profile Compliance Tests and Reference Implementation ER (11-095) OWS-8 ER Metadata Mapping between NASA ECS/HDF-EOS and WCS 2.0 (11-090) Copyright © 2011 Open Geospatial Consortium
- 19. OGC ® OWS-8Coverage Processing • WPS for EO data analysis of multiple coverages and other data • WCPS for advanced access to EO Coverages – multi-coverage fusion capabilities – queries taken from real-life examples WCPS and WPS Orchestration OWS-8 Geoprocessing of EO (Editor P. Baumann, OGC 11-116) © 2014 Open Geospatial Consortium 19
- 20. OWS-8 Big DataAnalytics on Motion Imagery OGC WPS for generating track features STANAG 4676 represented as GML features VMTI Implemented in OWS-8 OGC ® Display to Common Operational Picture © 2011 Open Geospatial Consortium, Inc. …and send to Mobile Client References: Ingo Simonis, IGSi OGC Doc 11-108 R. Cass, M. Simms, Compusult, OGC Doc 11-134
- 21. OGC ® VMTIto GeoSMS OWS-8 Mobile Client Implementation • Android app for OGC GeoSMS, WFS features and KML • Mobile client receives GeoSMS with URL for WFS track feature Videos of OWS-8 Moving Object Tracking http://www.opengeospatial.org/pub/www/ows8/tracking.html Copyright © 2012, Open Geospatial Consortium
- 22. OWS-9 Access toBig Data Coverages • OWS-9 WCS and OPeNDAP – DAP and WCS compliment each other. – Easy to use DAP to ‘reach into’ files and access their components – WCS provides specialized interface for geospatial data • OWS-9 WCS for LIDAR – Implementation of WCS 2.0 service that serves the LIDAR data in NITF format – service supports the JPEG 2000 output format along with GMLJP2 metadata description as well as the JPIP protocol to deliver the output JPEG2000 data. – Demonstrated WCS with JPIP streaming for GMLJP2. OGC ® OWS-9 Coverage Access ER (Editor P. Bauman, OGC 12-095) OWS-9 OWS Innovations WCS for LIDAR Engineering Report (Editor W. Han et.al., OGC 12-155) © 2014 Open Geospatial Consortium 22
- 23. OGC ® OGCTestbed 9: LiDAR Geoprocessing with OGC Standards OWS-9 OWS Innovations WCS for LIDAR Engineering Report (Editor W. Han et.al., OGC 12-155) © 2014 Open Geospatial Consortium 23
- 24. Performance of OGCServices in the Cloud • OGC service standards have been widely adopted and deployed across the world but little information has filtered out regarding the performance and scalability of products based on OGC standards. • Easy access to large IT computing resources such as the Amazon EC2 Cloud infrastructure provides an opportunity to use a flexible and low cost IT resource environment to investigate the performance and scalability of products based on OGC standards. • OGC Testbed 10 tested web mapping and other geo-processing OGC ® use cases as a way to characterize the performance of OGC data services deployed in Cloud infrastructures. © 2014 Open Geospatial Consortium 24
- 25. Concurrent user accessto OGC-compliant map servers using an Amazon EC2 with an S3 network attached storage configuration OGC ® OGC Testbed 10: WMTS Massively Available with Cloud Services © 2014 Open Geospatial Consortium
- 26. OGC ® OGCTestbed 10: Performance & Scalability of WMS/WMTS • Understand how OGC services perform on clouds through testing in OWS-10 Testbed • Response time with concurrent users for – WMS: linear – WMTS: flat to 1800 ! OGC Testbed 10 Performance of OGC Services in the Cloud ER (Editor Edric Keighan, OGC 14-028r1)
- 27. OGC ® OGCTestbed 10: SAR Interferometry with WPS on SBAS Cloud • Performance enhancements with Cloud deployment of SBAS (Small Baseline Subset) processing application using WPS and OpenSearch OGC Web Services © 2014 Open Geospatial Consortium 27 ! • Exploit 64 differential SAR scenes for the generation of time series showing ground displacements over a decade in geological sensitive areas. • Part of an ongoing effort from ESA, CNR-IREA and Terradue partners
- 28. As a resultof this testbed: User organizations able to request service providers (data provider, algorithm provider) to operate under a multi-tenant Cloud environment with specific performance constraints, in order to build and run scientific applications that deliver added-value OGC ® OGC Testbed 10: Moving Cloud Processing to Data 1. An OGC WPS Service “bought” on Terradue Cloud Marketplace 2. then deployed to Amazon UK Cloud 3. which computes using data from Interoute Cloud UK Experiments led by Terradue in OGC Testbed 10 OGC Testbed 10 Performance of OGC Services in the Cloud ER (Editor Edric Keighan, OGC 14-028r1) products.
- 29. OGC Testbed 11:Urban Climate Resilience • WFS-T Transactions • data representation • Desktop/mobile clients • Geosynchronization • REST and GeoJSON OGC ® • simulation models integration • high-resolution data • auxiliary data integration Copyright © 2014 Open Geospatial Consortium • What-if scenarios & models • Social/political data (TJS) • GeoPackage creation • Georeferenced grid harmon. • Streaming • Alerting • GeoPackage creation • GeoPackage synchronization • Image streaming tests OGC Testbed 11 UCR Thread Architect: Dr. Ingo Simonis, OGC Director Interoperability and Science
- 30. OGC ® OGCTestbed 11 Threads RFQ/CFP: http://www.opengeospatial.org/standards/requests/126 Geo4NIEM • NIEM V3.0 IEPD Geoprocessing Round-Trip • NIEM Enhancements Security Tagging • NIEM-GML Feature Processing • Security: Identification Authorization, Access • And more… © 2014, Open Geospatial Consortium 30 Urban Climate Resilience (UCR) • Climate/Big Data processing - WPS • Hi-Res Flood Model • Social Media • WFS-T with REST • GeoPackage, GeoSync • Georeferenceable Grid Harmonization • And more… Cross-Community Interoperability (CCI) • Aviation: NOTAM, AFX • REST and SOAP • JSON/GeoJSON • Semantic Broker: Social Media, Linked Data, GeoSPARQL • SPARQL for Symbology • Compliance • And more… Cross Thread Scenario: Displaced Population Due to Coastal Inundation
- 31. Relevant OGC EngineeringReports These engineering reports (ER) provide more detail on the topics addressed in OGC Testbeds listed above: • OWS-8 ER Metadata Mapping between NASA-EOS and WCS 2.0 (11-090) • OWS-8 WCS 2.0 EO Application Profile ER (11-096) • OWS-8 WCS 2.0 EO AP Compliance Tests and Reference Implementation ER (11-095) • OWS-8 Geoprocessing of EO (11-116) • OWS-9 Map Tiling Methods Harmonization (12-157) • OWS-9 Data Quality for Web Mapping ER (12-160) • OWS-9 Coverage Access ER (12-095) • OGC Testbed 10 Service Integration ER (14-013) • OGC Testbed 10 Performance of OGC Services in the Cloud ER (14-028r1) OGC ® © 2014 Open Geospatial Consortium 31
- 32. OGC ® ForDetails on OGC Standards… • OGC Standards – Freely available – www.opengeospatial.org/standards • OGC Reference Model (ORM) – Overview of OGC Standards Baseline – Resource for defining architectures for specific applications – www.opengeospatial.org/standards/orm George Percivall, gpercivall at opengeospatial.org