Continuous SQL with Apache Streaming (FLaNK and FLiP)
1 like794 views
The document discusses StreamNative's solutions for real-time data processing using Apache Pulsar and other Apache projects. It outlines the capabilities of Apache Pulsar, including cloud-native messaging, pub-sub systems, geo-replication, and scalability. The text also highlights the use of Apache Flink for advanced stream processing and provides links to community resources and upcoming events.
Continuous SQL with Apache Streaming (FLaNK and FLiP)
- 1. Continuous SQL with Apache Streaming Timothy Spann Developer Advocate
- 2. streamnative.io StreamNative Solution Application Messaging Data Pipelines Real-time Contextual Analytics Tiered Storage APP Layer Computing Layer Storage Layer StreamNative Platform IaaS Layer Micro Service Notification Dashboard Risk Control Auditing Payment ETL
- 3. streamnative.io Speaker Bio DZone Zone Leader and Big Data MVB; @PaasDev https://github.com/tspannhw https://www.datainmotion.dev/ https://github.com/tspannhw/SpeakerProfile https://dev.to/tspannhw https://sessionize.com/tspann/ https://www.slideshare.net/bunkertor Developer Advocate
- 4. streamnative.io FLaNK and FLiP Stacks ● Apache Flink ● Apache NiFi ● Apache Kafka ● Apache Flink ● Apache Pulsar ● StreamNative's Flink Connector for Pulsar ● Apache +++ Apache projects are the way for all streaming use cases.
- 5. streamnative.io I Can Haz Data? Today’s Data. REST and Websocket JSON {"symbol":"CLDR", "uuid":"10640832-f139-4b82-8780-e3ad37b3d0 ce", "ts":1618529574078, "dt":1612098900000, "datetime":"2021/01/31 08:15:00", "open":"12.24500", "close":"12.25500", "high":"12.25500", "volume":"12353", "low":"12.24500"}
- 6. streamnative.io End to End Streaming Demo Pipeline Enterprise sources Weather Errors Aggregates Alerts Stocks ETL Analytics Streaming SQL Clickstream Market data Machine logs Social
- 7. streamnative.io All Data - Anytime - Anywhere - Multi-Cloud - Multi-Protocol Multi- inges t Multi- inges t Multi-ingest Merge Priority
- 8. streamnative.io Apache Pulsar is Cloud-Native Messaging and Event-Streaming Platform
- 9. streamnative.io Apache Pulsar Overview Enable Geo-Replicated Messaging ● Pub-Sub ● Geo-Replication ● Pulsar Functions ● Horizontal Scalability ● Multi-tenancy ● Tiered Persistent Storage ● Pulsar Connectors ● REST API ● CLI ● Many clients available ● Four Different Subscription Types ● Multi-Protocol Support ○ MQTT ○ AMQP ○ JMS ○ Kafka ○ ...
- 10. streamnative.io What are the Benefits of Pulsar? Data Durability Scalability Geo-Replication Multi-Tenancy Unified Messaging Model
- 11. streamnative.io A Unified Messaging Platform Message Queuing Data Streaming
- 12. streamnative.io Upcoming - Flink + Pulsar (FLiP) https://flink.apache.org/2019/05/03/pulsar-flink.html https://github.com/streamnative/pulsar-flink https://streamnative.io/en/blog/release/2021-04-20-flink-sql- on-streamnative-cloud
- 13. streamnative.io Apache Flink Apache Flink is a distributed stream processing system. It is capable of providing high throughput, near real-time processing of streams from Pulsar. It is ideal for ambitious Stream Processing compared to Pulsar’s model of lightweight Stream Processing. End-to-end exactly-once stream processing https://streamnative.io/en/blog/release/2021-06-14-exactly-once-semantics-with-transactions-in-pulsar
- 14. streamnative.io Connect with the Community & Stay Up-To-Date ● Join the Pulsar Slack channel - Apache-Pulsar.slack.com ● Follow @streamnativeio and @apache_pulsar on Twitter ● Subscribe to Monthly Pulsar Newsletter for major news, events, project updates, and resources in the Pulsar community
- 15. streamnative.io ● https://github.com/tspannhw/StreamingSQLExamples ● https://github.com/streamnative/pulsar-flink ● https://www.linkedin.com/pulse/2021-schedule-tim-spann/ ● https://github.com/tspannhw/SpeakerProfile/blob/main/2021/talks/20210729_HailHydrate!FromStream toLake_TimSpann.pdf ● https://streamnative.io/en/blog/release/2021-04-20-flink-sql-on-streamnative-cloud ● https://docs.streamnative.io/cloud/stable/compute/flink-sql Deeper Content @PaasDev https://www.pulsardeveloper.com/ timothyspann
- 16. streamnative.io Pulsar Summit Europe October 6, 2021 Pulsar Summit Asia November 20-21, 2021 Contact us at [email protected] to become a sponsor or partner
- 17. streamnative.io