Drag and Drop Open Source GeoTools ETL with Apache NiFi
- 1. 1 Drag and Drop: Open Source GeoTools ETL with Apache NiFi Andrew Hulbert โ CCRI Constantin Stanca โ Hortonworks 5/16/2018
- 2. 2 Andrew Hulbert โ Principal Software Engineer @CCRI, GeoMesa Contributor: [email protected] Cpnstantin Stanca โ Solutions Engineer @Hortonworks, Data Transformation and Analytics SME: [email protected]
- 3. 3 Executive Summary โฌข Loading your GeoTools SimpleFeatures into your cloud or database never seems as easy as it should be. There's Twitter streams, FTP servers, inboxes, dropboxes, and all sorts of other data that you need to parse, convert to SimpleFeatures, and then ingest into your GeoTools datastore. โฌข GeoMesa and NiFi can provide a fully open source solution to ease the pain of ingesting data into ANY GeoTools data store. It's as easy as drag and drop! Literally! โฌข We'll show how real-time streaming data such as satellite AIS can be ingested and managed in real-time using ingest pipelines with your web browser without having to compile any code.
- 4. 4 Challenges โฌข Ancillary: โ Lots of data stores, tools and frameworks โ Moving data between systems (data tracking and data loss) โ convoluted data flow โ Configuration and Monitoring โฌข Big Data: โ Volume, Variety, Velocity โ More data stores, tools and frameworks โฌข Spatial Data Types
- 5. 5 Spatial Data Types Points Locations Events Instantaneous Positions Lines Road networks Voyages Trips Trajectories Polygons Administrative Regions Airspaces
- 7. 7 โTraditionalโ Geo-Spatial ETL OGC Simple Features
- 8. 8 โCloudโ Geo-Spatial ETL OGC Simple Features ?
- 9. 9 Solution: Apache NiFi and GeoMesa OGC Simple Features
- 10. 10 What is Apache NiFi? โฌข Powerful and scalable directed graphs of data routing, transformation, and system mediation logic. โฌข Web-based user interface โฌข Highly configurable โฌข Data Provenance โฌข Designed for extension โฌข Secure An open source project dedicated to making dataflow easy. It's as easy as drag and drop! Literally!
- 12. 12 Data in Motion (Cloud) Data in Motion (on-premises) Data at Rest (on-premises) Edge Data Data in Motion Edge Analytics Data at Rest (Cloud) Edge Data Data at Rest (on-premises) Closed Loop Analytics Machine Learning Deep Historical Analysis Data Flow Transformation with NiFi On-premises Cloud
- 14. 14 NiFi โ UI and Terms NiFi Terminology FlowFile Unit of data moving through the system Content + Attributes (key/value pairs) Connection Links between processors Queues that can be dynamically prioritized Processor Performs the work, can access FlowFiles Process Group Set of processors and their connections Receive data via input ports, send data via output ports โข Drag and drop processors to build a flow โข Start, stop, and configure components in real time โข View errors and corresponding error messages โข View statistics and health of data flow โข Create templates of common processor & connections
- 15. 15 NiFi โ Queue Prioritization โข Configure a prioritizer per connection โข Determine what is important for your data โ time based, arrival order, importance of a data set โข Funnel many connections down to a single connection to prioritize across data sets โข Develop your own prioritizer, if needed
- 16. 16 NiFi โ Provenance โข Tracks data at each point as it flows through the system โข Records, indexes, and makes events available for display โข Handles fan-in/fan-out, i.e. merging and splitting data โข View attributes and content at given points in time
- 17. 17 NiFi Demo
- 18. 18 SQL with NiFi Record Processors
- 24. 24 What is GeoMesa? A suite of tools for persisting, querying, analyzing, and streaming spatio-temporal data at scale
- 25. 25 GeoMesa NiFi โฌข GeoMesa-NiFi allows you to ingest data into GeoMesa straight from NiFi by leveraging custom processors. โฌข NiFi allows you to ingest data into GeoMesa from every source GeoMesa supports and more. Data SimpleFeatureType Schema GeoMesa NiFi Processors enabled datastores
- 26. 26 GeoMesa NiFi Processors โฌข PutGeoMesaAccumulo: Ingest data into a GeoMesa Accumulo datastore with a GeoMesa converter or from geoavro โฌข PutGeoMesaHBase: Ingest data into a GeoMesa HBase datastore with a GeoMesa converter or from geoavro โฌข PutGeoMesaFileSystem: Ingest data into a GeoMesa File System datastore with a GeoMesa converter or from geoavro โฌข PutGeoMesaKafka: Ingest data into a GeoMesa Kafka datastore with a GeoMesa converter or from geoavro โฌข PutGeoTools: Ingest data into an arbitrary GeoTools datastore using a GeoMesa converter or avro โฌข ConvertToGeoAvro: Use a GeoMesa converter to create geoavro
- 30. 30 Resources: โฌข GeoMesa Project: http://www.geomesa.org/ โฌข Geomesa-NiFi: https://github.com/geomesa/geomesa-nifi โฌข NiFi Project: https://nifi.apache.org/ โฌข NiFi Overview and Tutorials: https://hortonworks.com/apache/nifi/
- 31. 31 Thank you!