Flink Forward Berlin 2018: Timo Walther - "Flink SQL in Action"
8 likes1,482 views
This document discusses Flink SQL and its capabilities for stream and batch processing. Flink SQL provides a unified SQL interface that allows the same queries to run over both bounded batch data and unbounded streaming data. It achieves this through dynamic tables that are continuously updated as streaming data arrives. The document outlines Flink SQL's features, how queries are translated and executed, and examples of how SQL can be used to build data pipelines, real-time analytics, and live dashboards. It also introduces the new SQL client for interacting with Flink without writing code.
![© 2018 data Artisans4
FLINK’S POWERFUL ABSTRACTIONS
Process Function (events, state, time)
DataStream API (streams, windows)
SQL / Table API (dynamic tables)
Stream- & Batch
Data Processing
High-level
Analytics API
Stateful Event-
Driven Applications
val stats = stream
.keyBy("sensor")
.timeWindow(Time.seconds(5))
.sum((a, b) -> a.add(b))
def processElement(event: MyEvent, ctx: Context, out: Collector[Result]) = {
// work with event and state
(event, state.value) match { … }
out.collect(…) // emit events
state.update(…) // modify state
// schedule a timer callback
ctx.timerService.registerEventTimeTimer(event.timestamp + 500)
}
Layered abstractions to
navigate simple to complex use cases](https://image.slidesharecdn.com/ffb18-flinksqlinaction-180919155643/85/Flink-Forward-Berlin-2018-Timo-Walther-Flink-SQL-in-Action-4-320.jpg)
![© 2018 data Artisans14
SQL FEATURE SET IN FLINK 1.6.0
• SELECT FROMWHERE
• GROUP BY / HAVING
‒ Non-windowed,TUMBLE, HOP, SESSION windows
• JOIN / IN
‒ Windowed INNER, LEFT / RIGHT / FULL OUTER JOIN
‒ Non-windowed INNER, LEFT / RIGHT / FULL OUTER JOIN
• [streaming only] OVER /WINDOW
‒ UNBOUNDED / BOUNDED PRECEDING
• [batch only] UNION / INTERSECT / EXCEPT / ORDER BY](https://image.slidesharecdn.com/ffb18-flinksqlinaction-180919155643/85/Flink-Forward-Berlin-2018-Timo-Walther-Flink-SQL-in-Action-14-320.jpg)
![© 2018 data Artisans16
• Streaming enrichment joins (Temporal joins) [FLINK-9712]
• Support for complex event processing (CEP) [FLINK-6935]
‒ MATCH_RECOGNIZE
• More connectors and formats [FLINK-8535]
UPCOMING SQL FEATURES
SELECT
SUM(o.amount * r.rate) AS amount
FROM
Orders AS o,
LATERAL TABLE (Rates(o.rowtime)) AS r
WHERE r.currency = o.currency;](https://image.slidesharecdn.com/ffb18-flinksqlinaction-180919155643/85/Flink-Forward-Berlin-2018-Timo-Walther-Flink-SQL-in-Action-16-320.jpg)
Flink Forward Berlin 2018: Timo Walther - "Flink SQL in Action"
- 1. TIMO WALTHER, SOFTWARE ENGINEER FLINK FORWARD, BERLIN SEPTEMBER 5, 2018 FLINK SQL IN ACTION
- 2. © 2018 data Artisans2 ABOUT DATA ARTISANS Original creators of Apache Flink® Open Source Apache Flink + dAApplication Manager + dA Streaming Ledger
- 3. © 2018 data Artisans3 BIG APACHE FLINK SQL USERS
- 4. © 2018 data Artisans4 FLINK’S POWERFUL ABSTRACTIONS Process Function (events, state, time) DataStream API (streams, windows) SQL / Table API (dynamic tables) Stream- & Batch Data Processing High-level Analytics API Stateful Event- Driven Applications val stats = stream .keyBy("sensor") .timeWindow(Time.seconds(5)) .sum((a, b) -> a.add(b)) def processElement(event: MyEvent, ctx: Context, out: Collector[Result]) = { // work with event and state (event, state.value) match { … } out.collect(…) // emit events state.update(…) // modify state // schedule a timer callback ctx.timerService.registerEventTimeTimer(event.timestamp + 500) } Layered abstractions to navigate simple to complex use cases
- 5. © 2018 data Artisans5 APACHE FLINK’S RELATIONAL APIS Unified APIs for batch & streaming data A query specifies exactly the same result regardless whether its input is static batch data or streaming data. tableEnvironment .scan("clicks") .groupBy('user) .select('user, 'url.count as 'cnt) SELECT user, COUNT(url) AS cnt FROM clicks GROUP BY user LINQ-style Table APIANSI SQL
- 6. © 2018 data Artisans6 QUERY TRANSLATION tableEnvironment .scan("clicks") .groupBy('user) .select('user, 'url.count as 'cnt) SELECT user, COUNT(url) AS cnt FROM clicks GROUP BY user Input data is bounded (batch) Input data is unbounded (streaming)DataSet Rules DataSet PlanDataSet DataStreamDataStream Plan DataStream Rules Calcite Catalog Calcite Logical Plan Calcite Optimizer Calcite Parser & Validator Table API SQL API DataSet External Tables DataStream Table API Validator
- 7. © 2018 data Artisans7 WHAT IF “CLICKS” IS A FILE? Clicks user cTime url Mary 12:00:00 https://… Bob 12:00:00 https://… Mary 12:00:02 https://… Liz 12:00:03 https://… user cnt Mary 2 Bob 1 Liz 1 SELECT user, COUNT(url) as cnt FROM clicks GROUP BY user Input data is read at once Result is produced at once
- 8. © 2018 data Artisans8 WHAT IF “CLICKS” IS A STREAM? user cTime url user cnt SELECT user, COUNT(url) as cnt FROM clicks GROUP BY user Clicks Mary 12:00:00 https://… Bob 12:00:00 https://… Mary 12:00:02 https://… Liz 12:00:03 https://… Bob 1 Liz 1 Mary 1Mary 2 Input data is continuously read Result is continuously updated The result is the same!
- 9. © 2018 data Artisans9 • Usability ‒ ANSI SQL syntax: No custom “StreamSQL” syntax. ‒ ANSI SQL semantics: No stream-specific results. • Portability ‒ Run the same query on bounded and unbounded data ‒ Run the same query on recorded and real-time data • How can we achieve SQL semantics on streams? now bounded query unbounded query past future bounded query start of the stream unbounded query WHY IS STREAM-BATCH UNIFICATION IMPORTANT?
- 10. © 2018 data Artisans10 • Materialized views (MV) are similar to regular views, but persisted to disk or memory ‒Used to speed-up analytical queries ‒MVs need to be updated when the base tables change • MV maintenance is very similar to SQL on streams ‒Base table updates are a stream of DML statements ‒MV definition query is evaluated on that stream ‒MV is query result and continuously updated DATABASE SYSTEMS RUN QUERIES ON STREAMS
- 11. © 2018 data Artisans11 CONTINUOUS QUERIES IN FLINK • Core concept is a “DynamicTable” ‒Dynamic tables are changing over time • Queries on dynamic tables ‒produce new dynamic tables (which are updated based on input) ‒do not terminate • Stream ↔ Dynamic table conversions 11
- 12. © 2018 data Artisans12 STREAM ↔ DYNAMIC TABLE CONVERSIONS • Append Conversions ‒Records are only inserted (appended) • Upsert Conversions ‒Records are upserted/deleted ‒Records have a (composite) unique key • Retract Conversions ‒Records are inserted/deleted SELECT user, url FROM clicks WHERE url LIKE '%xyz.com' SELECT user, COUNT(url) FROM clicks GROUP BY user
- 13. SQL FEATURES
- 14. © 2018 data Artisans14 SQL FEATURE SET IN FLINK 1.6.0 • SELECT FROMWHERE • GROUP BY / HAVING ‒ Non-windowed,TUMBLE, HOP, SESSION windows • JOIN / IN ‒ Windowed INNER, LEFT / RIGHT / FULL OUTER JOIN ‒ Non-windowed INNER, LEFT / RIGHT / FULL OUTER JOIN • [streaming only] OVER /WINDOW ‒ UNBOUNDED / BOUNDED PRECEDING • [batch only] UNION / INTERSECT / EXCEPT / ORDER BY
- 15. © 2018 data Artisans15 • Support for POJOs, maps, arrays, and other nested types • Large set of built-in functions (150+) ‒ LIKE, EXTRACT, TIMESTAMPADD, FROM_BASE64, MD5, STDDEV_POP, AVG, … • Support for custom UDFs (scalar, table, aggregate) SQL FEATURE SET IN FLINK 1.6.0 See also: https://ci.apache.org/projects/flink/flink-docs-master/dev/table/functions.html https://ci.apache.org/projects/flink/flink-docs-master/dev/table/udfs.html
- 16. © 2018 data Artisans16 • Streaming enrichment joins (Temporal joins) [FLINK-9712] • Support for complex event processing (CEP) [FLINK-6935] ‒ MATCH_RECOGNIZE • More connectors and formats [FLINK-8535] UPCOMING SQL FEATURES SELECT SUM(o.amount * r.rate) AS amount FROM Orders AS o, LATERAL TABLE (Rates(o.rowtime)) AS r WHERE r.currency = o.currency;
- 17. © 2018 data Artisans17 WHAT CAN I BUILD WITH THIS? • Data Pipelines ‒ Transform, aggregate, and move events in real-time • Low-latency ETL ‒ Convert and write streams to file systems, DBMS, K-V stores, indexes, … ‒ Ingest appearing files to produce streams • Stream & Batch Analytics ‒ Run analytical queries over bounded and unbounded data ‒ Query and compare historic and real-time data • Power Live Dashboards ‒ Compute and update data to visualize in real-time
- 18. SQL CLIENT BETA
- 19. © 2018 data Artisans19 • Newest member of the Flink SQL family (since Flink 1.5) INTRODUCTION TO SQL CLIENT
- 20. © 2018 data Artisans20 • Goal: Flink without a single line of code ‒ only SQL andYAML ‒ "drag&drop" SQL JAR files for connectors and formats • Build on top of Flink'sTable & SQL API • Useful for prototyping & submission INTRODUCTION TO SQL CLIENT
- 21. © 2018 data Artisans21 SQL CLIENT CONFIGURATION See also: https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sqlClient.html
- 22. © 2018 data Artisans22 PLAY AROUND WITH FLINK SQL SQLClient Results CLI Submit Query SELECT user, COUNT(url) AS cnt FROM clicks GROUP BY user Gateway Database / HDFS Event Log Query StateResults Submit Job Catalog Optimizer Result Server Initialized by: conf/sql-client-defaults.yaml Initialized by: --environment my-config.yaml Modified by DDL commands within session. Changelog orTable
- 23. © 2018 data Artisans23 SUBMIT DETACHED QUERIES SQLClient Target Information CLI Submit Query INSERT INTO dashboard SELECT user, COUNT(url) AS cnt FROM clicks GROUP BY user Gateway Database / HDFS Event Log Query State Submit Job Catalog Optimizer Result Server Initialized by: conf/sql-client-defaults.yaml Initialized by: --environment my-config.yaml Modified by DDL commands within session. Cluster ID & Job ID
- 25. © 2018 data Artisans27 SUMMARY • Unification of stream and batch is important. • Flink’s SQL solves many streaming and batch use cases. • Runs in production at Alibaba, Uber, and others. • The community is working on improving user interfaces. • Get involved, discuss, and contribute!
- 26. THANK YOU! @twalthr @dataArtisans @ApacheFlink WE ARE HIRING data-artisans.com/careers Available on O’Reilly Early Release!
Editor's Notes
- #2: Today I wanna show you the power and simplicity And why and how you can leverage it for your use cases my name is … committer, pmc, and part of the flink team since the early days I’m working as a software engineer
- #3: • data Artisans was founded by the original creators of Apache Flink • We provide dA Platform, manage a zoo of stream processing application - a complete stream processing infrastructure with open-source Apache Flink
- #4: If your company would like to be represented on the “Powered by Apache Flink SQL” page, email me.
- #5: Flink offers APIs for different levels of abstraction that all can be mixed and matched. On the lowest level, ProcessFunctions give precise control about state and time, i.e., when to process data. The intermediate level, the so-called DataStream API provides higher-level primitives such as for window processing Finally, on the top level the relational API, SQL and the Table API, are centered around the concept of dynamic tables. This is what I’m going to talk about next.
- #18: Now let me show an example to show the power and simplicity of SQL
- #22: Non-programmatic way of configuring Flink jobs Per-session and/or global configuration in YAML Configures: Tables from external systems Views defined in SQL User-defined functions Execution properties (e.g. result mode, execution mode) Deployment properties
- #25: a Flink SQL client container to submit queries and visualize their results, a Flink master and a Flink worker container to execute queries, an Apache Kafka container to produce input streams and consume result streams, an Apache Zookeeper container (required by Kafka), and an ElasticSearch container to maintain an external materialized table.
- #29: (Keep this slide up during the Q&A part of your talk. Having this up in the final 5-10 minutes of the session gives the audience something useful to look at.)