Self-Service Data Ingestion Using NiFi, StreamSets & Kafka
10 likes3,495 views
The document discusses self-service data ingestion technologies using Apache NiFi, StreamSets, and Kafka, focusing on their architecture, features, and challenges. It highlights data flow processing methods, the importance of data lineage, and real-time streaming capabilities, while addressing difficulties in data ingestion, such as infrastructure changes and data quality issues. Each technology is compared in terms of visual data flow modeling, ease of use, and integration with existing systems.
Self-Service Data Ingestion Using NiFi, StreamSets & Kafka
- 1. BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENF HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH Self-Service Data Ingestion Using NiFi, StreamSets & Kafka Guido Schmutz – 6.12.2017 @gschmutz guidoschmutz.wordpress.com
- 2. Guido Schmutz Working at Trivadis for more than 21 years Oracle ACE Director for Fusion Middleware and SOA Consultant, Trainer Software Architect for Java, Oracle, SOA and Big Data / Fast Data Head of Trivadis Architecture Board Technology Manager @ Trivadis More than 30 years of software development experience Contact: [email protected] Blog: http://guidoschmutz.wordpress.com Slideshare: http://www.slideshare.net/gschmutz Twitter: gschmutz
- 3. Agenda 1. Data Flow Processing 2. Apache NiFi 3. StreamSets Data Collector 4. Kafka Connect 5. Summary
- 5. Hadoop Clusterd Hadoop Cluster Big Data Cluster Traditional Big Data Architecture BI Tools Enterprise Data Warehouse Billing & Ordering CRM / Profile Marketing Campaigns File Import / SQL Import SQL Search / Explore Online & Mobile Apps Search NoSQL Parallel Batch Processing Distributed Filesystem • Machine Learning • Graph Algorithms • Natural Language Processing
- 6. Event Hub Event Hub Hadoop Clusterd Hadoop Cluster Big Data Cluster Event Hub – handle event stream data BI Tools Enterprise Data Warehouse Location Social Click stream Sensor Data Billing & Ordering CRM / Profile Marketing Campaigns Event Hub Call Center Weather Data Mobile Apps SQL Search / Explore Online & Mobile Apps Search Data Flow NoSQL Parallel Batch Processing Distributed Filesystem • Machine Learning • Graph Algorithms • Natural Language Processing
- 7. Hadoop Clusterd Hadoop Cluster Big Data Cluster Event Hub – taking Velocity into account Location Social Click stream Sensor Data Billing & Ordering CRM / Profile Marketing Campaigns Call Center Mobile Apps Batch Analytics Streaming Analytics Results Parallel Batch Processing Distributed Filesystem Stream Analytics NoSQL Reference / Models SQL Search Dashboard BI Tools Enterprise Data Warehouse Search / Explore Online & Mobile Apps File Import / SQL Import Weather Data Event Hub Event Hub Event Hub
- 8. Container Hadoop Clusterd Hadoop Cluster Big Data Cluster Event Hub – Asynchronous Microservice Architecture Location Social Click stream Sensor Data Billing & Ordering CRM / Profile Marketing Campaigns Call Center Mobile Apps Parallel Batch ProcessingDistributed Filesystem Microservice NoSQLRDBMS SQL Search BI Tools Enterprise Data Warehouse Search / Explore Online & Mobile Apps File Import / SQL Import Weather Data { } API Event Hub Event Hub Event Hub
- 9. Container Hadoop Clusterd Hadoop Cluster Big Data Cluster Location Social Click stream Sensor Data Billing & Ordering CRM / Profile Marketing Campaigns Call Center Mobile Apps Parallel Batch ProcessingDistributed Filesystem Microservice NoSQLRDBMS SQL Search BI Tools Enterprise Data Warehouse Search / Explore Online & Mobile Apps File Import / SQL Import Weather Data { } API Event Hub Event Hub Event Hub Integrate Sanitize / Normalize Deliver
- 10. IoT GW MQTT Broker Continuous Ingestion - DataFlow Pipelines DB Source Big Data Log Stream Processing IoT Sensor Event Hub Topic Topic REST Topic IoT GW CDC GW Connect CDC DB Source Log CDC Native IoT Sensor IoT Sensor 12 Dataflow GW Topic Topic Queue Messaging GW Topic Dataflow GW Dataflow Topic REST 12 File Source Log Log Log Social Native
- 11. DataFlow Pipeline • Flow-based ”programming” • Ingest Data from various sources • Extract – Transform – Load • High-Throughput, straight-through data flows • Data Lineage • Batch- or Stream-Processing • Visual coding with flow editor • Event Stream Processing (ESP) but not Complex Event Processing (CEP) Source: Confluent
- 12. SQL Polling Change Data Capture (CDC) File Polling (File Tailing) File Stream (Appender) Continuous Ingestion – Integrating data sources Sensor Stream
- 13. Ingestion with/without Transformation? Zero Transformation • No transformation, plain ingest, no schema validation • Keep the original format – Text, CSV, … • Allows to store data that may have errors in the schema Format Transformation • Prefer name of Format Translation • Simply change the format • Change format from Text to Avro • Does schema validation Enrichment Transformation • Add new data to the message • Do not change existing values • Convert a value from one system to another and add it to the message Value Transformation • Replaces values in the message • Convert a value from one system to another and change the value in-place • Destroys the raw data!
- 14. Why is Data Ingestion Difficult? Physical and Logical Infrastructure changes rapidly Key Challenges: Infrastructure Automation Edge Deployment Infrastructure Drift Data Structures and formats evolve and change unexpectedly Key Challenges: Consumption Readiness Corruption and Loss Structure Drift Data semantics change with evolving applications Key Challenges Timely Intervention System Consistency Semantic Drift Source: Streamsets
- 15. Challenges for Ingesting Sensor Data • Multitude of sensors • Real-Time Streaming • Multiple Firmware versions • Bad Data from damaged sensors • Regulatory Constraints • Data Quality Source: Cloudera
- 17. Apache NiFi
- 18. Apache NiFi • Originated at NSA as Niagarafiles – developed behind closed doors for 8 years • Open sourced December 2014, Apache Top Level Project July 2015 • Look-and-Feel modernized in 2016 • Opaque, “file-oriented” payload • Distributed system of processors with centralized control • Based on flow-based programming concepts • Data Provenance and Data Lineage • Web-based user interface
- 19. Processors for Source and Sink • ConsumeXXXX (AMQP, EWS, IMAP, JMS, Kafka, MQTT, POP3, …) • DeleteXXXX (DynamoDB, Elasticsearch, HDFS, RethinkDB, S3, SQS, ...) • FetchXXXX (AzureBlobStorage, ElasticSearch, File, FTP, HBase, HDFS, S3 ...) • ExecuteXXXX (FlumeSink, FlumeSource, Script, SQL, ...) • GetXXXX (AzureEventHub, Couchbase, DynamoDB, File, FTP, HBase, HDFS, HTTP, Ignite, JMSQueue, JMSTopic, Kafka, Mongo, Solr, Splunk, SQS, TCP, ...) • ListenXXXX (HTTP, RELP, SMTP, Syslog, TCP, UDP, WebSocket, ...) • PublishXXXX (Kafka, MQTT) • PutXXXX (AzureBlobStorage, AzureEventHub, CassandraQL, CloudWatchMetric, Couchbase, DynamoDB, Elasticsearch, Email, FTP, File, Hbase, HDFS, HiveQL, Kudu, Lambda, Mongo, Parquet, Slack, SQL, TCP, ....) • QueryXXXX (Cassandra, DatabaseTable, DNS, Elasticserach)
- 20. Processors for Processing • ConvertXxxxToYyyy • ConvertRecord • EnforceOrder • EncryptContent • ExtractXXXX (AvroMetdata, EmailAttachments, Grok, HL7Attributes, ImageMetadata, ...) • GeoEnrichIP • JoltTransformJSON • MergeContent • ReplaceText • ResizeImage • SplitXXXX (Avro, Content, JSON, Record, Xml, ...) • TailFile • TransformXML • UpdateAttribute
- 21. Demo Case Truck-2 truck/nn/ position Truck-1 Truck-3 truck position raw truck/nn/ positionTruck-4 Truck-5 Raw Data Store MQTT to Kafka Kafka to Raw {"truckid":"57","driverid":"15","routeid":"1927624662 ","eventtype":"Normal","latitude":"38.65","longitude": "-90.21","correlationId":"4412891759760421296"} Port: 1883 Port: 1884
- 22. Demo: Dataflow for MQTT to Kafka
- 25. Demo: Masking Field with ReplaceText Processor
- 26. StreamSets
- 27. StreamSets Data Collector • Founded by ex-Cloudera, Informatica employees • Continuous open source, intent-driven, big data ingest • Visible, record-oriented approach fixes combinatorial explosion • Batch or stream processing • Standalone, Spark cluster, MapReduce cluster • IDE for pipeline development by ‘civilians’ • Relatively new - first public release September 2015 • So far, vast majority of commits are from StreamSets staff
- 28. StreamSets Origins Source: https://streamsets.com/connectors An origin stage represents the source for the pipeline. You can use a single origin stage in a pipeline Origins on the right are available out of the box API for writing custom origins
- 29. StreamSets Processors A processor stage represents a type of data processing that you want to perform use as many processors in a pipeline as you need Programming languages supported • Java • JavaScript • Jython • Groovy • Java Expression Language (EL) Spark Some of processors available out-of-the- box: • Expression Evaluator • Field Flattener • Field Hasher • Field Masker • Field Merger • Field Order • Field Splitter • Field Zip • Groovy Evaluator • JDBC Lookup • JSON Parser • Spark Evaluator • …
- 30. StreamSets Destinations A destination stage represents the target for a pipeline. You can use one or more destinations in a pipeline Destinations on the right are available out of the box API for writing custom origins Source: https://streamsets.com/connectors
- 31. Demo Case Truck-2 truck/nn/ position Truck-1 Truck-3 truck position raw truck/nn/ positionTruck-4 Truck-5 Raw Data Store MQTT-1 to Kafka Kafka to Raw {"truckid":"57","driverid":"15","routeid":"1927624662 ","eventtype":"Normal","latitude":"38.65","longitude": "-90.21","correlationId":"4412891759760421296"} MQTT-2 to Kafka Edge Port: 1883 Port: 1884
- 32. Demo: Dataflow for MQTT to Kafka
- 34. Demo: Kafka Sink
- 35. Demo: Dataflow for MQTT to Kafka
- 37. Demo: Sending Message to Kafka in Avro
- 38. StreamSets Dataflow Performance Manager • Map dataflows to topologies, manage releases & track changes • Measure KPIs and establish baselines for data availability and accuracy • Master dataflow operations through Data SLAs Source: https://streamsets.com/connectors
- 39. Kafka Connect
- 40. Kafka Connect - Overview Source Connector Sink Connector
- 41. Kafka Connect – Single Message Transforms (SMT) Simple Transformations for a single message Defined as part of Kafka Connect • some useful transforms provided out-of-the-box • Easily implement your own Optionally deploy 1+ transforms with each connector • Modify messages produced by source connector • Modify messages sent to sink connectors Makes it much easier to mix and match connectors Some of currently available transforms: • InsertField • ReplaceField • MaskField • ValueToKey • ExtractField • TimestampRouter • RegexRouter • SetSchemaMetaData • Flatten • TimestampConverter
- 42. Kafka Connect – Many Connectors 60+ since first release (0.9+) 20+ from Confluent and Partners Source: http://www.confluent.io/product/connectors Confluent supported Connectors Certified Connectors Community Connectors
- 43. Demo Case Truck-2 truck/nn/ position Truck-1 Truck-3 truck position raw truck/nn/ positionTruck-4 Truck-5 Raw Data Store MQTT-1 to Kafka Kafka to Raw {"truckid":"57","driverid":"15","routeid":"1927624662 ","eventtype":"Normal","latitude":"38.65","longitude": "-90.21","correlationId":"4412891759760421296"} MQTT-2 to Kafka Port: 1883 Port: 1884
- 44. Demo: Dataflow for MQTT to Kafka #!/bin/bash curl -X "POST" "http://192.168.69.138:8083/connectors" -H "Content-Type: application/json" -d $'{ "name": "mqtt-source", "config": { "connector.class": "com.datamountaineer.streamreactor.connect.mqtt.source.MqttSourceConnector", "connect.mqtt.connection.timeout": "1000", "tasks.max": "1", "connect.mqtt.kcql": "INSERT INTO truck_position SELECT * FROM truck/+/position", "name": "MqttSourceConnector", "connect.mqtt.service.quality": "0", "connect.mqtt.client.id": "tm-mqtt-connect-01", "connect.mqtt.converter.throw.on.error": "true", "connect.mqtt.hosts": "tcp://mosquitto-1:1883" } }'
- 45. Summary
- 46. Summary Apache NiFi • visual dataflow modelling • very powerful – “with power comes responsibility” • special package for Edge computing • data lineage and data provenance • supports for backpressure • no transport mechanism (DEV/TST/PROD) • custom processors • supported by Hortonworks StreamSets • visual dataflow modelling • very powerful – “with power comes responsibility” • special package for Edge computing • data lineage and data provenance • no transport mechanism • custom sources, sinks, processors • supported by StreamSets Kafka Connect • declarative style data flows • simplicity - “simple things done simple” • very well integrated with Kafka – comes with Kafka • Single Message Transforms (SMT) • use Kafka Streams for complex data flows • custom connectors • supported by Confluent
- 47. Technology on its own won't help you. You need to know how to use it properly.