Hadoop data ingestion
Download as PPTX, PDF5 likes13,401 views
Vinod Nayal presented on options for ingesting data into Hadoop, including batch loading from relational databases using Sqoop or vendor-specific tools. Data from files can be FTP'd to edge nodes and loaded using ETL tools like Informatica or Talend. Real-time data can be ingested using Flume for transport with light enrichment or Storm with Kafka for a queue to enable low-latency continuous ingestion with more in-flight processing. The choice between Flume and Storm depends on the amount of required in-flight processing.
Hadoop data ingestion
- 1. Hadoop Data Ingestion Presented by Vinod Nayal
- 2. Data Ingestion Options SQOOP RDBMS Files coming in batch SFTP ETL TOOLS Real time KAFKA FLUME STORM NATIVE BIG DATA CONNECTORS Hadoop Staging
- 3. Data Ingestion Options Batch Load from RDBMS : Sqoop : RDBMS can support multiple parallel connections . millions of rows can be imported in a reasonable timeframe which can be scaled. Most vendors these days have a loader/connector product that delivers better performance and more security when compared to Sqoop, For ex Oracle has OraOop or at Oracle Big Data Connectors Data from files : FTP the data to edge nodes and then load the data using the ETL tool. ETL tools like informatica /talend can be integrated . With 40 -50 Mbps speed and 5 machines 1 TB can be imported in 1 hr . Compressing the data will result in better time frame . Files can also be consolidated at source to fit into hadoop optimal size . Real time Data ingestion : Flume is good at transport and some light enrichment Storm +queue (kafka) : Good for low-latency continuous ingestion.With storm we can do major processing to data while ingesting .Flume vs. Storm decision should depend largely on the amount of processing needed in-flight. With storm we can do event processing like fraud detection and pattern matching as data is flowing