Stream Processing with Pipelines and Stored Procedures
Download as PDF, PPTX1 like905 views
The document outlines a method for stream processing using pipelines and stored procedures. It details the creation of a pipeline for loading data from various sources into a database table and demonstrates how to implement a procedure to split data into multiple tables. Additionally, it discusses topics such as updating statistics, managing complex logic, and automating the deletion of obsolete data.
![CREATE PIPELINE AS LOAD DATA
[KAFKA | S3 | FS]
INTO TABLE tbl](https://image.slidesharecdn.com/meetup122f112f17tewei-171213175633/85/Stream-Processing-with-Pipelines-and-Stored-Procedures-2-320.jpg)
![CREATE PIPELINE AS LOAD DATA
[KAFKA | S3 | FS]
INTO PROCEDURE split](https://image.slidesharecdn.com/meetup122f112f17tewei-171213175633/85/Stream-Processing-with-Pipelines-and-Stored-Procedures-5-320.jpg)
More Related Content
![Real Time Operational Analytics with Microsoft Sql Server 2016 [Liviu Ieran]](https://cdn.slidesharecdn.com/ss_thumbnails/realtimeoperationalanalyticswithmicrosoftsqlserver2016liviuieran-161102082504-thumbnail.jpg?width=560&fit=bounds)
Stream Processing with Pipelines and Stored Procedures
- 1. Stream Processing with Pipelines and Stored Procedures Tewei Luo Software Engineer
- 2. CREATE PIPELINE AS LOAD DATA [KAFKA | S3 | FS] INTO TABLE tbl
- 3. Stream into more tables?
- 4. Stream into more tables? Example: - 2 fields in source - 2 tables, each takes one column
- 5. CREATE PIPELINE AS LOAD DATA [KAFKA | S3 | FS] INTO PROCEDURE split
- 6. CREATE PROCEDURE split(batch Query(c1 int, c2 text)) AS BEGIN INSERT INTO tb1 SELECT c1 from batch; INSERT INTO tb2 SELECT c2 from batch; … END
- 7. 7 DEMO
- 8. - Split stream to multiple tables - Update statistics - Complex logic using existing tables - Delete obsolete data automatically
- 9. 9 Questions?