Achieving Massive Concurrency & Sub-second Query Latency on Cloud Warehouses & Data Lakes with Kyligence Cloud
0 likes•119 views
The Data Orchestration Summit 2020 discussed leveraging Kyligence Cloud for achieving massive concurrency in cloud data warehouses and lakes through advanced data orchestration techniques. It emphasized the use of Apache Kylin, a distributed OLAP platform, for AI-assisted query optimization and intelligent pre-computation, enabling sub-second response times for large-scale queries. Kyligence, founded in 2016, aims to deliver an AI-assisted SQL/OLAP platform that enhances interactive analytics across various data environments.
Achieving Massive Concurrency & Sub-second Query Latency on Cloud Warehouses & Data Lakes with Kyligence Cloud
- 1. DATA ORCHESTRATION SUMMIT 2020 Achieving Massive Concurrency on Cloud Warehouses and Data Lakes with Kyligence Cloud George Demarest Head of Marketing Kyligence, US
- 2. DATA ORCHESTRATION SUMMIT Modern Data Analytics Stack Processing LayerData Storage Layer Presentation Layer Data Services BI Tools and Dashboards Natural Language ProcessingPredictive Analytics Machine Learning Data Science Data Engineering Data Modeling
- 3. DATA ORCHESTRATION SUMMIT Where We Spend Our Time on ML Data Engineering Machine Learning Operations 80% 5% 15%
- 4. DATA ORCHESTRATION SUMMIT Engineering the Data Platforms 80%
- 5. DATA ORCHESTRATION SUMMIT 2020 Who Owns Analytics Performance? Data Engineers? Developers? DataOps?
- 6. DATA ORCHESTRATION SUMMIT What if… …you could accurately predict and optimize response times for 95% of your queries
- 7. DATA ORCHESTRATION SUMMIT Distributed OLAP – Precomputation Data Volume Response Time Pre-Computation Run-Time Computation How many beers were sold in L.A. in June?
- 8. DATA ORCHESTRATION SUMMIT Apache Kylin Top Level Project The only open-source distributed OLAP platform AwardWinning InfoWorld’s Bossies 2015 & 2016 (Best of Open Source Software Awards) Sub-Second Interactive Query Large scale, high concurrency, multi-dimensional, sub-second query latency 1,000+ Organizations Adopted by thousands of organizations globally
- 9. DATA ORCHESTRATION SUMMIT Kyligence Cloud: Extreme Analytics Supports All Data Platforms • Data Warehouse • Data Lake • Streaming Data • Cloud Storage Intelligent Precomputation • Multi-dimensional cubes and table Indexes • AI-assisted query optimization, data modeling Supports All BI Tools • Excel • Power BI • Tableau • MicroStrategy Massive Concurrency • 1000s+ concurrent queries • 10s-100s analysts running multi-threaded programs Powered by Apache Kylin
- 10. DATA ORCHESTRATION SUMMIT Kyligence Analytical Cloud ANSI SQL MDX REST APIFinance Marketing Sales Precomputation Layer Distributed cubes & table indexes Modeling Layer AI-assisted design Unified security & business context Semantic Layer Data as a Service Machine Learning Data as a Service SaaS & Apps CRM HCM SCM Low Latency Queries Any BI Tool Any Cloud Any Data Platform Data Warehouse Streaming Data Data Lake AI Augmented Engine Intelligent Precomputation Platform to Accelerate Cloud Analytics
- 11. DATA ORCHESTRATION SUMMIT Kyligence Cloud ANSI SQL MDX REST APIFinance Marketing Sales Precomputation Layer Distributed cubes & table indexes Modeling Layer AI-assisted design Unified security & business context Semantic Layer Data as a Service Machine Learning Data as a Service SaaS & Apps CRM HCM SCM Low Latency Queries Any BI Tool Any Cloud Any Data Platform Data Warehouse Streaming Data Data Lake AI Augmented Engine
- 12. DATA ORCHESTRATION SUMMIT Intelligent Query Routing Route to Precomputation Layer or Pushdown to Native Engine
- 13. DATA ORCHESTRATION SUMMIT Kyligence AI Augmented Engine Model training Pre-aggregation Data processing Auto-Generate Cube & Index AI Augmented Engine Self training Group wisdom Queries across the org ingested as input Benefits - Continuous model improvement without human intervention - Greater user satisfaction Identify and manage most valuable data automatically
- 14. DATA ORCHESTRATION SUMMIT Kyligence = Kylin + Intelligence Mission: Develop an AI-Assisted SQL/OLAP Platform for Interactive Analytics at Unprecedented Scale • Founded in 2016 by the creators of Apache Kylin • CRN Top-10 big data startups in 2018 • Global Presence: San Jose, Shanghai, Beijing, Seattle, New York • VCs: Fidelity International, Shunwei Capital, Broadband Capital, Coatue, Redpoint, Cisco
- 15. DATA ORCHESTRATION SUMMIT Kyligence Cloud: Extreme Analytics Supports All Data Platforms • Data Warehouse • Data Lake • Streaming Data • Cloud Storage Intelligent Precomputation • Multi-dimensional cubes and table Indexes • AI-assisted query optimization, data modeling Supports All BI Tools • Excel • Power BI • Tableau • MicroStrategy Massive Concurrency • 1000s+ concurrent queries • 10s-100s analysts running multi-threaded programs Powered by Apache Kylin