Download as PDF, PPTX
Uploaded byStratebi
PCM18 (Big Data Analytics)
The document discusses the integration of Apache Kylin and Pentaho for big data OLAP solutions, highlighting Kylin's strengths in query latency and compatibility with various BI tools. It details the benchmark tests comparing Kylin with Vertica and PostgreSQL, showcasing Kylin's superior performance, especially for large datasets. Additionally, it outlines challenges encountered during integration and proposes solutions, emphasizing the necessity of open-source tools in modern data analysis environments.
More Related Content
Similar to PCM18 (Big Data Analytics)
![[DSC Europe 25] Jurij Kodre - What Can Healthcare Learn From Finance.pptx](https://cdn.slidesharecdn.com/ss_thumbnails/q87nzrwtd6ihsqqtrqdr-7-251126104228-00657fba-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Stefan_Milosevic - AI and digital twins in precision oncology...](https://cdn.slidesharecdn.com/ss_thumbnails/nnmuciuxr2ugh4d9pzkg-1-251126104228-148c7fe8-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Milan Misic - RAG, recommenders and face recognition applica...](https://cdn.slidesharecdn.com/ss_thumbnails/mxe0wzfeqkortbfecopo-8-251128093135-51a402bb-thumbnail.jpg?width=640&height=640&fit=bounds)
PCM18 (Big Data Analytics)
- 2. www.stratebi.com Emilio Arias Co-Founder atStrateBI. Follow us on Twittter @Stratebi @TodoBI_OS
- 3. www.stratebi.com Roberto Tardío Head ofBig Data at StrateBI. Follow me on Twittter @RoberTardio
- 4.
- 5.
- 6. www.stratebi.com • OLAP (On-LineAnalytical Processing) • Analytical systems that enable interactive queries. • Requires very low query latency: Milliseconds-Seconds. • Usually supports SQL and, sometimes, MDX query language. • Enables KPI’s data aggregation and filtering across hierarchical multidimensional structures (OLAP cubes). • Used as data source for diferents goals: • Detailed data analysis (OLAP views). • Dashboarding. • Reporting.
- 7. www.stratebi.com • Big DataOLAP • Big Data: Volume, Variety and Velocity. • OLAP applications over Big Data sets. • Main challenges: • Very low query latency over fact and dimension tables of billions to trillions of rows. • Support for ANSI SQL and BI Tools integration. • Real time data ingestion and processing.
- 8. www.stratebi.com • Current Approaches(some of them).
- 9. www.stratebi.com • Why ApacheKylin? • Sub-second queries over +12 billion rows fact tables. • Best query latency results (in our deployements and benchmarks) • ANSI SQL and BI Tools integration. • Integration with Pentaho possible through JDBC, Mondrian and PME • Also Superset, Tableau, Power BI, Zeppelin, Microstrategy… • Star and snowflake schemas full support • Not all tools support it (e.g. Druid) • Near Real time data ingestion (Kafka) and processing. • It is an Apache open-source project. • Currently in version 2.5
- 10. www.stratebi.com • Apache KylinArchitecture • M-OLAP approach: • Data pre-aggregation. • Enables only analytical queries. • Hadoop based tool • Full scalability • Hadoop nodes • Hbase and Kylin separated clusters (if needed)
- 11.
- 12. www.stratebi.com • A reallove story
- 13. www.stratebi.com • Why ApacheKylin and Pentaho BA Server? • It is becoming more and more necessary to provide dashboarding, reporting and OLAP viewing over Big Data scenarios. • Using our STTools Pentaho plugins: STPivot, STReport, STDashboard,… • Also Pentaho Reporting, Community Dashboard Editor, Saiku (plugin),… • Both Kylin and Pentaho are leading BI & Big Data open-source tools. • Pentaho enables integration with most-known Big Data tools: Hive, Impala, Spark SQL,… • Integration with Pentaho possible through JDBC, Mondrian and PME • Mondrian 4.X using existing Mondrian 4.4 (lagunitas) • Mondrian 3.X, with a great effort of our team. • Using Pentaho BA Sever 7.1
- 14. www.stratebi.com • Identified issuesand solutions: Kylin and Mondrian 3.X (3.14) • Issue 1: Kylin needs ANSI-92 inner joins but Mondrian 3.X generated old-style joins. • Solution: We defined a Mondrian dialect and we used this patch to implement allowsJoinOn() method. • Issue 2: Mondrian native cross join and nonempty properties caused invalid SQL code for Kylin. • Solution: We disabled these properties for Kylin dialect. • Issue 3: Kylin needs the fact table to be the first table in the from SQL clause. • Workaround: We modified Mondrian code to identify fact tables using a name prefix (F or FT) and thus place them first in the from clause.
- 15. www.stratebi.com • Identified issuesand solutions: Kylin and Mondrian 3.X • Some interesting used references: • How to implement Kylin dialect for Mondrian • https://web.archive.org/web/20171010103502/http://dekarlab.de/wp/?p=443 • Pentaho JIRA - MONDRIAN-955 • Mondrian should support the Dialect.allowsJoinOn() option • Patch • Pentaho JIRA - MONDRIAN-2364 • Add dialect for Apache Kylin
- 16. www.stratebi.com • Identified issuesand solutions: Kylin and Pentaho Metadata Editor • Issue 1: There is no dialect for Kylin in PME. • Solution: Definition of the Kylin dialect using the Hive 2 SQL dialect. • Works perfectly without changing anything. • JDBC connections between Pentaho BA Server and Kylin: • Initially we used the generic connection through a JDBC driver. • To simplify the connection, we defined the connection interface for Kylin in Pentaho BA Server. • We have used Pentaho BA Server 7.1 but a connection to Kylin has not yet been included in Pentaho 8.1.
- 17. www.stratebi.com • Enabling securityat schemas, concepts and data levels: • Mondrian 3.X • We could not use views to filter data (Kylin approach limitation) • Solution: We have used Mondrian Dynamic Schema Processor • We extended the typical Mondrian DSP class using a variable that replaces a piece of XML from the schema. • Pentaho Metadata Editor • PME requires roles and users tables be created in the same data source, but Kylin does not allow it (Kylin approach limitation). • Solution: We have created JDBCSecuritySqlGenerator • Extension of this PME existing security class. • The security is defined in a file we called securitySQLGenerator-properties.xml.
- 18. www.stratebi.com • What havewe obtained? • Dasboarding, reporting and OLAP viewing using our Pentaho STTools over cubes with more than a billion rows (1.000.000.000) • Enabling sub-second Roll-up, Drill-down, Slice and Dice and Pivot OLAP operations. • We have carried the first deployement of Kylin for a Spain based company. • Try our demo with Kylin, Pentaho and STPivot viewer (Marketplace available) • http://bigdata.stratebi.com/kylin-olap/index.htm
- 19.
- 20. www.stratebi.com • Kylin appliedto digital marketing scenario • Initial Scenario • OLAP system for data analysis using an in-house reporting tool. • Based on MySQL (80% queries) + Redshift (20% queries) • Several million rows per hour in some fact tables • Goals • Reduce query latency (some queries take >20s to run) • Reduce ETL processing time: "Data freshness". • Implementation of Open-Source BI tools (STTools) • Self-service OLAP, reporting and dashboarding
- 21. www.stratebi.com • Kylin appliedto digital marketing scenario • Architecture
- 22. www.stratebi.com • Kylin appliedto digital marketing scenario • Goals achieved • Reduced query latency: User queries were compared for the company's three most important reports. • Kylin query executions times are 4 times faster than Redshift. • Most Kylin queries have response times below 1 second. • Some very complex queries that in Redshift take about 30 seconds are executed in over 400 milliseconds using Kylin • Full integration with open source BI tools (STTools) • STPivot, STReport, STDashboard • Security implemented at schema and data levels (Mondrian and PME).
- 23. www.stratebi.com • Kylin appliedto digital marketing scenario • Kylin vs Redshift Redshift Kylin
- 24.
- 25. www.stratebi.com • Why Verticais an alternative to Kylin for Big Data OLAP? • Sub-second queries over billions of rows fact tables. • In our implementations and benchmark it achieves very good query latency results. • But it is not as fast as Kylin for extremely huge fact tables. • ANSI SQL and BI Tools integration. • Integration with Pentaho possible through JDBC, Mondrian and PME • Also Superset, Tableau, Power BI, Zeppelin, Microstrategy… • Star and snowflake schemas full support • Near Real time data ingestion and processing. • Microfocus Vertica is not an open-source project. • But there is a free community version, enough for much typical Big Data scenarios.
- 26. www.stratebi.com • Vertica Architecture •Distributed processing in cluster mode. • But it does not need a hadoop cluster to work. • Although it does support integration with Hadoop (e.g. Spark or Hive) • Columnar and distributed storage • Hybrid OLAP (tables, projections, flattened tables…)
- 27. www.stratebi.com • Integration withPentaho and STTools • Seamless integration with Pentaho PDI for data warehouse loading • Including bulk load steps • We have also integrated Vertica with Pentaho BA Server for several successfully use cases • Be careful defining the Mondrian OLAP scheme to achieve good performance. • In PME we have faced similar issues to Kylin (use of PostgreSQL dialect) • Retail Sector use case • + 3,000 points of sales = high concurrency • Volumetrics determined by sales line level detail • Need for highly customized graphics (we have implemented a lot of CDE dashboards)
- 28.
- 29. www.stratebi.com • Why aBig Data OLAP Benchmark? • To test the performance of the two most powerful Big Data OLAP tools • Kylin vs Vertica • Compare their performance against OLAP implementations in traditional databases • PostgreSQL: Open source relational database that has a good performance for OLAP systems.
- 30. www.stratebi.com • Benchmark implementation •We have used the SSB benchmark • A star scheme version of the best known TPC-H (industry-standard) • Kyligence team has an implementation of the SSB benchmark for Apache Kylin. • Including schemas and data generator. • We have adapted it to use with Vertica and PostgreSQL. • It provides a set of 13 analytical queries
- 31. www.stratebi.com • Test performed •Number of rows of facts and dimensions tables for each test performed. • Hardware used LINEORDER CUSTOMER PART SUPPLIER DATE Test – Role of table Fact (KPI) Dimension Dimension Dimension Dimension 100M 100.000.000 40.000 32.000 20.000 2.556 500M 500.000.000 200.000 48.000 100.000 2.556 1.000M 1.000.000.000 400.000 56.000 200.000 2.556 Tool Distributed Processing Kind of hardware Nº of hosts Processor Cores RAM Memory Kylin 2.4 Yes Dedicated Cloud 3 Intel(R) Atom(TM) CPU C2750 @ 2.40GHz 8 32 Gb Vertica 9.1 Yes Dedicated Cloud 3 Intel(R) Atom(TM) CPU C2750 @ 2.40GHz 8 32 Gb PostgreSQL 9.6 No Dedicated Cloud 1 Intel(R) Atom(TM) CPU C2750 @ 2.40GHz 8 32 Gb
- 32. www.stratebi.com • Benchmark Results TestP1 – 100M (seconds) P1 – 500M (seconds) P1 – 1.000M (seconds) Query Kylin Vertica Postgre Kylin Vertica Postgre Kylin Vertica Postgre Q1.1 0.2 0.2 22.4 0.3 0.3 +280 0.6 0.6 - Q1.2 0.2 0.4 18.7 0.3 0.2 +280 0.5 0.3 - Q1.3 0.2 0.4 18.5 0.3 0.3 +280 0.6 0.2 - Q2.1 0.3 1.1 18.1 0.4 2.7 +280 0.6 9.1 - Q2.2 0.3 0.8 16.3 0.4 2.7 +280 0.7 8.2 - Q2.3 0.3 0.8 15.2 0.4 2.2 +280 0.6 7.4 - Q3.1 0.3 1.4 23.9 0.4 3.7 +280 0.8 15.1 - Q3.2 0.6 0.7 18.5 0.8 0.7 +280 0.9 9.8 - Q3.3 0.3 0.9 15.8 0.3 0.6 +280 0.7 3.7 - Q3.4 0.2 0.6 15.9 0.2 0.2 +280 0.2 1.0 - Q4.1 0.3 1.4 23.7 0.4 7.3 +280 0.7 14.7 - Q4.2 0.3 1.0 23.3 0.4 2.0 +280 0.7 3.8 - Q4.3 2.5 0.8 17.1 2.4 1.3 +280 2.9 2.0 -
- 33. www.stratebi.com • Benchmark Results Relationshipbetween row size in the fact table and query latency between Kylin and Vertica
- 34. www.stratebi.com • Benchmark Results •Kylin and Vertica are both suitable for Big Data OLAP applications. • Apache Kylin has the best query performance. • But high hardware, software (Hadoop) and know-how requirements. • 100% open source version without limitations. • Vertica is the alternative to Kylin for less extreme Big Data scenarios. • Lower hardware, software and know-how requirements. • Free community version with some limitations. • PostgreSQL is not suitable for Big Data OLAP.
- 35.
- 36. www.stratebi.com • Pentaho alsointegrates with many other Big Data tools • Lince Big Data Stack • Our selection of Big Data tools based on experience and tests. • All of them allow the integration with Pentaho open source tools. • Lince BI tools (formerly STTools) are used to analyze the data from Big Data repositories. • STPivot: OLAP Viewer. • STReport: Ad-Hoc Reporting. • STDashboard: Fast Dashboards. • STCard: Balanced Scorecards.
- 37.
- 38. www.stratebi.com • Visit ourBig Data demos website • http://bigdata.stratebi.com/ • Pentaho • Kylin + STPivot (Mondrian 4.X) • Hadoop + PDI (HDFS, Hive, Oozie,…steps) • PDI + SparkMlib + Zeppelin • Other open source Big Data tools • Kafka + Spark Streaming • Kylin + Superset • Neo4J • …and much more.
- 39.
- 40. www.stratebi.com • Pentaho BAserver enables Big Data OLAP in combination with Kylin or Vertica. • Easy to integrate through JDBC connector with SQL based plugins (CDE dashboards) • We have worked hard to integrate these tools with Mondrian 3.X and PME 7.1. • Best performance results with the integration between Pentaho, Kylin and STTools • Sub-second Roll-up, Drill-down, Slice and Dice and Pivot OLAP operations. • Experienced performance with STTools is really good, but we have to extend our benchmark to test it (Kylin with Mondrian or PME) • Pentaho tools are useful for Big Data ETL and analysis • However, our experience tells us that many of the Pentaho Big Data connectors and features are very hard to configure. • We propose to include Kylin and Vertica dialects (Mondrian and PME) in future Pentaho versions.