Modifying Your SQL Streaming Queries on the Fly: The Impossible Trinity
0 likes168 views
The document discusses the challenges of modifying SQL streaming queries in real-time analytics, highlighting the difficulties caused by changing queries, data schemas, and business logic. It presents the concept of the 'impossible trinity' where achieving efficiency, generality, and consistency simultaneously is not possible. The text proposes that while optimizations can enhance query recomputation, a balance must be struck between the three competing factors.
Modifying Your SQL Streaming Queries on the Fly: The Impossible Trinity
- 1. Modifying Your SQL Streaming Queries on the Fly: The impossible Trinity Yingjun Wu Founder, RisingWave Labs
- 2. Who Am I? 2 • Yingjun Wu • Founder @ RisingWave • Ex-AWS Redshift • Ex-IBM Research Almaden • PhD, National University of Singapore • Visiting PhD, Carnegie Mellon University
- 3. What Am I Doing? • Building RisingWave, an open-source SQL streaming database • Extremely easy • Extremely cost-efficient 3 Adopted in hundreds of enterprises and SMBs! Streaming analytics Real-time monitoring/alerting Real-time ETL Database replication …
- 4. SQL Stream Processing 4 Operational databases Messaging queues BI tools Stream processing systems
- 5. SQL Stream Processing 5 Operational databases Messaging queues BI tools Stream processing systems
- 6. SQL Stream Processing 6 • Continuously monitor crypto data
- 7. SQL Stream Processing 7 • Continuously monitor crypto data
- 8. SQL Stream Processing 8 • Continuously monitor crypto data
- 9. SQL Stream Processing 9 • Continuously monitor crypto data
- 10. Challenges • Queries may be modified from time to time! 10
- 11. Challenges • Queries may be modified from time to time! • Upstream data schema may change 11 Column name Column type product_id VARCHAR price NUMERIC open_24h NUMERIC volume_24h TIMESTAMP Column name Column type product_id VARCHAR price NUMERIC open_12h NUMERIC volume_24h TIMESTAMP Schema change!
- 12. Challenges • Queries may be modified from time to time! • Upstream data schema may change • Business logic may change 12 5 60 Logic change!
- 13. Challenges • Queries may be modified from time to time! • Upstream data schema may change • Business logic may change • Not a big problem for batch queries… 13 Query 1 Query 2
- 14. Challenges • Queries may be modified from time to time! • Upstream data schema may change • Business logic may change • Not a big problem for batch queries… • But a HUGE problem for streaming queries! 14 Continuous query processing!
- 15. State Management • Consider time window functions 15 i96 i2 i1 o2 o1 State … … SELECT … FROM TUMBLE(crypto_source, …, interval ‘5’ second); o6 Input progress 96
- 16. State Management • Consider time window functions 16 i96 i2 i1 o2 o1 State … … SELECT … FROM TUMBLE(crypto_source, …, interval ‘5’ second); o7 i98 i97 o6 Updated! Add new values and evict expired values Input progress 98
- 17. State Management • Consider time window functions 17 i96 i2 i1 o2 o1 State … … SELECT … FROM TUMBLE(crypto_source, …, interval ‘5’ second); o7 i98 i97 o6 o8 i99 Input progress 99 Updated! Add new values and evict expired values
- 18. State Management • Consider time window functions 18 i96 i2 i1 o2 o1 State … … SELECT … FROM TUMBLE(crypto_source, …, interval ‘5’ second); o6 Input progress 96 Let’s modify the SQL query!
- 19. State Management • Consider time window functions 19 i96 i2 i1 o2 o1 State … … SELECT … FROM TUMBLE(crypto_source, …, interval ‘5’ second); o6 SELECT … FROM TUMBLE(crypto_source, …, interval ‘60’ second); Input progress 96 Let’s modify the SQL query!
- 20. State Management • Consider time window functions 20 i96 i2 i1 o2 o1 State … … SELECT … FROM TUMBLE(crypto_source, …, interval ‘5’ second); o6 SELECT … FROM TUMBLE(crypto_source, …, interval ‘60’ second); State Input progress 96
- 21. State Management • Consider time window functions 21 i96 i2 i1 o2 o1 State … … SELECT … FROM TUMBLE(crypto_source, …, interval ‘5’ second); o6 SELECT … FROM TUMBLE(crypto_source, …, interval ‘60’ second); State Input progress 96
- 22. The Native Solution • Recompute! 22
- 23. The Native Solution • Recompute! 23 i96 i2 i1 o2 o1 State … … SELECT … FROM TUMBLE(crypto_source, …, interval ‘5’ second); o6 SELECT … FROM TUMBLE(crypto_source, …, interval ‘60’ second); Input progress 96
- 24. The Native Solution • Recompute! 24 i96 i2 i1 o2 o1 State … … SELECT … FROM TUMBLE(crypto_source, …, interval ‘5’ second); o6 SELECT … FROM TUMBLE(crypto_source, …, interval ‘60’ second); Input progress 96 Latest progress 96 Mark the progress
- 25. The Native Solution • Recompute! 25 i96 i2 i1 o2 o1 State … … SELECT … FROM TUMBLE(crypto_source, …, interval ‘5’ second); o6 SELECT … FROM TUMBLE(crypto_source, …, interval ‘60’ second); Input progress 96 0 Latest progress 96 Mark the progress Drop the entire state
- 26. The Native Solution • Recompute! 26 SELECT … FROM TUMBLE(crypto_source, …, interval ‘5’ second); SELECT … FROM TUMBLE(crypto_source, …, interval ‘60’ second); Input progress 0 Latest progress 96 State Mark the progress Drop the entire state Rebuild from scratch
- 27. The Native Solution • Recompute! 27 SELECT … FROM TUMBLE(crypto_source, …, interval ‘5’ second); SELECT … FROM TUMBLE(crypto_source, …, interval ‘60’ second); Input progress 2 Latest progress 96 State i2 i1 Updated!! Mark the progress Drop the entire state Rebuild from scratch
- 28. The Native Solution • Recompute! 28 SELECT … FROM TUMBLE(crypto_source, …, interval ‘5’ second); SELECT … FROM TUMBLE(crypto_source, …, interval ‘60’ second); Input progress 2 Latest progress 96 State i2 i1 Mark the progress Drop the entire state Rebuild from scratch Updated!!
- 29. The Native Solution • Recompute! 29 SELECT … FROM TUMBLE(crypto_source, …, interval ‘5’ second); SELECT … FROM TUMBLE(crypto_source, …, interval ‘60’ second); Input progress 96 Latest progress 96 State i96 i2 i1 … o2 o1 … o6 Mark the progress Drop the entire state Rebuild from scratch Updated!!
- 30. The Native Solution • Recompute! 30 SELECT … FROM TUMBLE(crypto_source, …, interval ‘5’ second); SELECT … FROM TUMBLE(crypto_source, …, interval ‘60’ second); Input progress 96 Latest progress 96 State i96 i2 i1 … o2 o1 … o6 Mark the progress Drop the entire state Rebuild from scratch Continue processing Updated!!
- 31. The Native Solution • Recompute! 31 SELECT … FROM TUMBLE(crypto_source, …, interval ‘5’ second); SELECT … FROM TUMBLE(crypto_source, …, interval ‘60’ second); Input progress 96 Latest progress 96 State Updated!! i96 i2 i1 … o2 o1 … o6 Mark the progress Drop the entire state Rebuild from scratch Continue processing
- 32. The Ideal Solution • Reuse! 32
- 33. The Ideal Solution • Reuse! 33 State Old query New query State i96 i2 i1 o2 o1 … … o6
- 34. The Ideal Solution • Reuse! 34 State Old query New query State Reuse? i96 i2 i1 o2 o1 … … o6
- 35. The Ideal Solution • Reuse! 35 State State i96 i2 i1 o2 o1 … … o6 Internal state Entry 1 Entry 2 … Entry 5 … Entry 10 Internal state Entry 1 Entry 2 … Entry 5 SELECT … FROM WHERE … ORDER BY … LIMIT 10; SELECT … FROM WHERE … ORDER BY … LIMIT 5;
- 36. The Ideal Solution • Reuse! 36 State State i96 i2 i1 o2 o1 … … o6 Internal state Entry 1 Entry 2 … Entry 5 … Entry 10 Internal state Entry 1 Entry 2 … Entry 5 Truncate! SELECT … FROM WHERE … ORDER BY … LIMIT 10; SELECT … FROM WHERE … ORDER BY … LIMIT 5; Reuse!
- 37. The Ideal Solution • Reuse! 37 State State i96 i2 i1 o2 o1 … … o6 Internal state Entry 1 Entry 2 … Entry 5 … Entry 10 Internal state Entry 1 Entry 2 … Entry 5 Truncate! Reuse! SELECT … FROM WHERE … ORDER BY … LIMIT 10; SELECT … FROM WHERE … ORDER BY … LIMIT 5;
- 38. The Ideal Solution • Reuse! 38 State i96 i2 i1 o2 o1 … … o6 Detect reusable patterns SELECT … FROM WHERE … ORDER BY … LIMIT 10; SELECT … FROM WHERE … ORDER BY … LIMIT 5;
- 39. The Ideal Solution • Reuse! 39 State i96 i2 i1 o2 o1 … … o6 Detect reusable patterns Pattern matching! SELECT … FROM WHERE … ORDER BY … LIMIT 10; SELECT … FROM WHERE … ORDER BY … LIMIT 5;
- 40. The Ideal Solution • Reuse! 40 SELECT … FROM WHERE … ORDER BY … LIMIT 10; SELECT … FROM WHERE … ORDER BY … LIMIT 5; TABLE FILTER WHERE … TABLE FILTER WHERE … TOP N TOP N ORDER BY … LIMIT = 10 ORDER BY … LIMIT = 5 Stateful Stateless Pattern matching! Detect reusable patterns
- 41. The Ideal Solution • Reuse! 41 State i96 i2 i1 o2 o1 … … o6 Detect reusable patterns Modify internal state Internal state Entry 1 Entry 2 … Entry 5 … Entry 10 Truncate! SELECT … FROM WHERE … ORDER BY … LIMIT 10; SELECT … FROM WHERE … ORDER BY … LIMIT 5;
- 42. The Ideal Solution • Reuse! 42 State SELECT … FROM WHERE … ORDER BY … LIMIT 10; SELECT … FROM WHERE … ORDER BY … LIMIT 5; i96 i2 i1 o2 o1 … … o6 Detect reusable patterns Modify internal state Continue processing Internal state Entry 1 Entry 2 … Entry 5
- 43. The Ideal Solution • Reuse! 43 i96 i2 i1 o2 o1 State … … SELECT … FROM TUMBLE(crypto_source, …, interval ‘5’ second); o6 SELECT … FROM TUMBLE(crypto_source, …, interval ‘60’ second); State
- 44. The Ideal Solution • Reuse! 44 i96 i2 i1 o2 o1 State … … SELECT … FROM TUMBLE(crypto_source, …, interval ‘5’ second); o6 SELECT … FROM TUMBLE(crypto_source, …, interval ‘60’ second); State
- 45. The Ideal Solution • Dozens of SQL clauses • Unlimited number of combinations… 45 WHERE LIMIT UNION HAVING DISTINCT ORDER BY JOIN GROUP BY ……
- 46. The Real World 46 Recompute! Reuse! General but inefficient Efficient but not general
- 47. The Real World 47 Recompute! Reuse! General but inefficient Efficient but not general
- 50. The Impossible Trinity 50 Efficiency Generality Consistency It’s impossible to achieve all three!
- 51. The Impossible Trinity 51 Efficiency Generality Consistency • Sacrifice efficiency! It’s impossible to achieve all three!
- 52. The Impossible Trinity 52 Efficiency Generality Consistency • Sacrifice generality! It’s impossible to achieve all three!
- 53. The Impossible Trinity 53 Efficiency Generality Consistency It’s impossible to achieve all three! • Sacrifice consistency!
- 54. The Impossible Trinity 54 Efficiency Generality Consistency Which one should we sacrifice?
- 55. The Impossible Trinity 55 Efficiency Generality Consistency Option Sacrifice efficiency Strategy Always recompute Pros General; easy to implement Cons Can take very long time to recompute
- 56. The Impossible Trinity 56 Efficiency Generality Consistency Option Sacrifice generality Strategy Try best to reuse Pros Efficient Cons Very difficult to implement; very limited
- 57. The Impossible Trinity 57 Efficiency Generality Consistency Option Sacrifice consistency Strategy Try to reuse whenever possible Pros General enough and also efficient Cons May compromise consistency
- 58. The Impossible Trinity 58 Efficiency Generality Consistency Option Sacrifice consistency Strategy Try to reuse whenever possible Pros General enough and also efficient Cons May compromise consistency
- 59. The Impossible Trinity 59 Pattern matching! SELECT … FROM WHERE … ORDER BY … LIMIT 10; SELECT … FROM WHERE … ORDER BY … LIMIT 5; TABLE FILTER WHERE … TABLE FILTER WHERE … TOP N TOP N ORDER BY … LIMIT = 10 ORDER BY … LIMIT = 5 Stateful Stateless Query plan “Context”
- 60. The Impossible Trinity 60 Pattern matching! SELECT … FROM WHERE … ORDER BY … LIMIT 10; SELECT … FROM WHERE … ORDER BY … LIMIT 5; TABLE FILTER WHERE … TABLE FILTER WHERE … TOP N TOP N ORDER BY … LIMIT = 10 ORDER BY … LIMIT = 5 Stateful Stateless Query plan “Context”
- 61. The Impossible Trinity 61 Pattern matching! Internal state Entry 1 Entry 2 … Entry 5 … Entry 10 SELECT … FROM WHERE … ORDER BY … LIMIT 10; SELECT … FROM WHERE … ORDER BY … LIMIT 5; TABLE FILTER TABLE FILTER TOP N TOP N Stateful Stateless Directly reuse without editing!
- 62. The Impossible Trinity 62 Pattern matching! TABLE TABLE TUMBLE TUMBLE Stateful SELECT … FROM TUMBLE(crypto_source, …, interval ‘5’ second); SELECT … FROM TUMBLE(crypto_source, …, interval ‘60’ second);
- 64. Our Implementation 64 • Principle: let the users to decide!
- 65. Our Implementation 65 • Principle: let the users to decide! Old query New query
- 66. Our Implementation 66 • Principle: let the users to decide! Old query New query Is there common pattern?
- 67. Our Implementation 67 • Principle: let the users to decide! Old query New query Is there common pattern? Recompute! Can we preserve consistency? Yes No
- 68. Our Implementation 68 • Principle: let the users to decide! Old query New query Is there common pattern? “Are you okay to sacrifice consistency?” Recompute! Can we preserve consistency? Reuse! Yes No Yes No
- 69. Our Implementation 69 • Principle: let the users to decide! Old query New query Is there common pattern? “Are you okay to sacrifice consistency?” Recompute! Can we preserve consistency? Reuse! Recompute! Reuse! (Inconsistent!) Yes No Yes No Yes No
- 70. Optimization • Best case: reuse states without compromising consistency • Worst case: recompute from scratch 70 Can we optimize the worst-case scenario?
- 71. Optimization 71 SELECT … FROM TUMBLE(crypto_source, …, interval ‘5’ second); SELECT … FROM TUMBLE(crypto_source, …, interval ‘60’ second); Input progress 96 Latest progress 96 State i96 i2 i1 … o2 o1 … o6 Drop the entire state Rebuild from scratch Continue processing • Recompute! Updated!!
- 72. Optimization 72 SELECT … FROM TUMBLE(crypto_source, …, interval ‘5’ second); SELECT … FROM TUMBLE(crypto_source, …, interval ‘60’ second); Input progress 96 Latest progress 96 State i96 i2 i1 … o2 o1 … o6 Drop the entire state Rebuild from scratch Continue processing • Recompute! Scale out!!! Updated!!
- 73. Optimization 73 • “S3 as primary storage” State State State States State State State Compute nodes Persistent storage States Checkpoint Cache Cache Cache “state as checkpoint” Traditional stream processing systems State State State
- 74. Optimization 74 • Scale out during the recomputation (backfilling) stage Cache Cache State State State State State State Compute nodes Persistent storage
- 75. Optimization 75 • Scale out during the recomputation (backfilling) stage Cache Cache Cache Cache Cache State State State State State State Cache Compute nodes Persistent storage Scale out to recompute
- 76. Optimization 76 • Scale in once recomputation is done Cache Cache State State State State State State Compute nodes Persistent storage Scale in after recomputation
- 78. More Challenges 78 Materialized view Materialized view Materialized view Materialized view • Stacked materialized views, or dependent streaming jobs!
- 79. More Challenges 79 Materialized view Materialized view Materialized view Materialized view • Stacked materialized views, or dependent streaming jobs! • What if an upstream materialized view gets changed?
- 80. More Challenges 80 Materialized view Materialized view Materialized view Materialized view • Stacked materialized views, or dependent streaming jobs! • What if an upstream materialized view gets changed?
- 81. Conclusion • Modifying SQL streaming queries on the fly can cause troubles • We could choose only two from generality, efficiency, and consistency • Optimizations can be applied to greatly accelerate query recomputation 81