Introduction of MariaDB AX / TX
2 likes•2,586 views
MariaDB TX 2.0 introduces enhancements for transactional and analytical workloads, including comprehensive SQL capabilities, increased performance, and disaster recovery features. It also includes a new storage engine, MyRocks, and enhanced database proxies for scalability and high performance. Additionally, MariaDB AX (ColumnStore) offers a columnar storage solution optimized for big data analytics, allowing for flexible deployment and high data compression.
Introduction of MariaDB AX / TX
- 1. The picture can't be displayed. Introduction of MariaDB AX / TX September 7, 2017 GOTO Satoru
- 2. MariaDB TX MariaDB AX for Transactional workloads ● MariaDB Server ● MariaDB MaxScale ● database connectors ● services ● support ● tools for Analytic workloads ● MariaDB Server ● MariaDB ColumnStore ● MariaDB MaxScale ● database connectors ● services ● support ● tools
- 3. The picture can't be displayed. MariaDB TX 2.0 Overview
- 4. MariaDB TX 2.0 Software licensing and support for databases, the database proxies and database connectors Services access to remote administration, enterprise architecture, migration planning services and more Tools tools for administration, monitoring, backup and replication management Transaction Platform 4
- 5. MariaDB TX 2.0 MariaDB Services Remote Administration Enterprise Architecture Migration Management Technical Support MariaDB Tools MariaDB Admin MariaDB Monitor MariaDB Backup MariaDB Notifications Pluggable Storage Replication MariaDB Server Pluggable Storage Replication MariaDB Server Pluggable Storage Replication MariaDB Server MariaDB Cluster MariaDB Connectors MariaDB MaxScale (Proxy) Failover ide Streaming Caching Security Routing Transaction Platform 5
- 6. Enterprise Reliability High Availability Disaster Recovery Scalability Performance Security Open Development Extensible Architecture Community Collaboration Flexible Data Model Streaming Integration Community Innovation
- 7. The picture can't be displayed. MariaDB TX 2.0 What’s new
- 8. MariaDB TX 2.0 Features Completeness Comprehensive SQL (CTEs and Window functions as well as JSON) that can handle multiple data models Compatibility Closing the gap with enterprise databases through schema compatibility Performance and Scalability New storage engine, MyRocks and enhanced database proxy for a high performance and scalable database Security and Disaster Recovery Tighter security with the most capable database firewall for data masking, DoS protection and fast disaster recovery 9
- 9. SQL Completeness Common Table Expressions Helps make complex queries easier to read and maintain by breaking them down into simpler building blocks Window Functions Helps eliminate expensive subqueries and self-joins, improving query performance, and helps make queries easier to read JSON functions Helps developers read and write native JSON documents using SQL functions Window function example SELECT depname, empno, salary, avg(salary) OVER (PARTITION BY depname) FROM empsalary; With CTEs and Window functions as well as JSON functions, MariaDB TX 2.0 delivers comprehensive SQL that can handle multiple data models depname empno salary avg Engineering 11 5200 4700 Engineering 8 4200 4700 HR 9 6000 5000 HR 5 4000 5000 Sales 2 7000 5500 Sales 4 4000 5500 10
- 10. Schema Compatibility Check constraints Helps a DBA to enforce data consistency on the database server level without the need to implement triggers Multiple triggers of the same type (BEFORE, AFTER) can be created per table Decimal point (Oracle compatibility) – Increased decimal point from 30 to 38 Check constraints example Schema compatibility features allow MariaDB TX 2.0 to be a practical alternative to legacy enterprise databases CREATE TABLE jsontable ( id INTEGER NOT NULL PRIMARY KEY AUTO_INCREMENT, jsonfield VARCHAR(1024), category VARCHAR(20) as (JSON_VALUE(jsonfield,'$.category')), KEY jsonkey (category), CHECK (JSON_VALID(jsonfield))); 11
- 11. Performance and Scalability MyRocks • Developed by Facebook as a more efficient alternative to InnoDB, based on RocksDB • 2x better compression, 10x less write amplification, faster replication, faster data loading • Supports broad adoption of solid state drives Query Cache Helps improve the performance of repeat queries by caching the results within the proxy, either in memory or on disk (2.8x performance improvement) New storage engine and enhanced database proxy for a high performance and scalable database Database Size Test Iteration (1 hour/iteration) Size(GB) InnoDB (uncompressed) InnoDB (compressed) MyRocks (compressed) 12
- 12. Security and Disaster Recovery Data Masking Helps meet HIPAA/PCI requirements by masking sensitive data on a per-column, per-user basis Binary Log based Rollback Helps DBAs recovery faster (minutes instead of hours) by rolling back recent transactions based on a point in time rather than restoring data from a backup, can apply to specific tables or the entire database Data masking example Tighter security with the most capable database firewall for data masking, DoS protection and fast disaster recovery SELECT name, ssn FROM tbl_customers WHERE id = 100; name ssn ------------------------------- Shane Johnson XXX-XX-XXXX 13
- 13. 10.1 (GA) Multi-master Replication (Galera) Data-at-Rest Encryption Password Validation Page Compression Facebook Defragmentation Spatial References (GIS) 10.2 (GA) 10.3 (DEV) MyRocks Flashback (Point in Time Rollback) Common Table Expressions Window Functions Check Constraints JSON & GeoJSON Functions Mult-Trigger Support Delayed Replication Binary Log Compression Per User Resource Limits Virtual Column Indexes Sequences & PL/SQL (for migration from Oracle ) System Versioned Tables As Of (Point in Time Querying) User-defined Aggregate Functions Intersect & Except Hidden Columns Evolution of MariaDB Server 14
- 14. MariaDB AX 1.0
- 15. What is MariaDB AX / ColumnStore? High performance columnar storage engine that supports a wide variety of analytical use cases in highly scalable distributed environments Parallel query processing for distributed environments Faster, More Efficient Queries Single Interface for OLTP and analytics Easy to Manage and Scale Easier Enterprise Analytics Power of SQL and Freedom of Open Source for Big Data Analytics Better Price Performance 16
- 16. MariaDB ColumnStore Architecture Columnar Distributed Data Storage Local Storage | SAN | EBS | GlusterFS | Ceph BI Tool SQL Client Custom Big Data App Application MariaDB SQL Front End Distributed Query Engine Data Storage User Module (UM) Performance Module (PM) 17
- 17. Row-oriented vs. Column-oriented format • Row oriented – Rows stored sequentially in a file – Scans through every record row by row • Column oriented: – Each column is stored in a separate file – Scans the only relevant column ID Fname Lname State Zip Phone Age Sex 1 Bugs Bunny NY 11217 (718) 938-3235 34 M 2 Yosemite Sam CA 95389 (209) 375-6572 52 M 3 Daffy Duck NY 10013 (212) 227-1810 35 M 4 Elmer Fudd ME 04578 (207) 882-7323 43 M 5 Witch Hazel MA 01970 (978) 744-0991 57 F ID 1 2 3 4 5 Fname Bugs Yosemite Daffy Elmer Witch Lname Bunny Sam Duck Fudd Hazel State NY CA NY ME MA Zip 11217 95389 10013 04578 01970 Phone (718) 938-3235 (209) 375-6572 (212) 227-1810 (207) 882-7323 (978) 744-0991 Age 34 52 35 43 57 Sex M M M M F SELECT Fname FROM Table 1 WHERE State = 'NY' 18
- 18. Scalability • Massively parallel architecture – Linear scalability as new nodes are added • Horizontal scaling – Add new data nodes as your data grows – Continue read queries when adding new nodes Shared-Nothing Distributed Data Storage Compressed by default User Module (UM) Performance Module (PM) Data Storage 19
- 19. Storage Architecture • Columnar storage – Each column stored as separate file – No index management for query performance tuning – Online Schema changes: Add new column without impacting running queries • Automatic horizontal partitioning – Logical partition every 8 Million rows – In memory metadata of partition min and max – No partition management for query performance tuning • Compression – Accelerate decompression rate – Reduce I/O for compressed blocks Column 1 Extent 1 (8 million rows, 8MB～64MB) Extent 2 (8 million rows) Extent M (8 million rows) Column 2 Column 3 ... Column N Data automatically arranged by • Column – Acts as Vertical Partitioning • Extents – Acts as horizontal partition Vertical Partition Horizontal Partition ... Vertical Partition Vertical Partition Vertical Partition Horizontal Partition Horizontal Partition 20
- 20. High Performance Data Ingestion • Fully parallel high speed data load – Parallel data loads on all PMs simultaneously – Multiple tables in can be loaded simultaneously – Read queries continue without being blocked • Micro-batch loading for real-time data flow Column 1 Extent 1 (8 million rows, 8MB～64MB) Extent 2 (8 million rows) Extent M (8 million rows) Column 2 ... Column N Horizontal Partition ... Horizontal Partition Horizontal Partition High Water Mark New Data being loaded Dataaccessedby runningqueries 21
- 21. Shared Nothing Distributed Data Storage SQL Column Primitives User Module Performance Module UM PM High Performance Query Processing • Query received and parsed by MariaDB Front End on UM • Storage Engine Plugin breaks down query in primitive operations and distributes across PM • Primitives processed on PM • One thread working on a range of rows • Execute column restrictions and projections • Execute group by/aggregation against local data • Each PM work on Primitives in parallel and fully distributed • Each primitive executes in a fraction of a second • Return intermediate results to UM Massively parallel, distributed query processing, Shared nothing architecture Primitives ↓↓↓↓ Intermediate ↑↑Results↑↑ 22
- 22. Horizontal Partition: 8 Million Rows Extent 2 Horizontal Partition: 8 Million Rows Extent 3 Horizontal Partition: 8 Million Rows Extent 1 Storage Architecture reduces I/O • Only touch column files that are in projection, filter and join conditions • Eliminate disk block touches to partitions outside filter and join conditions Extent 1: Min State: CA, Max State: NY Extent 2: Min State: OR, Max State: WY Extent 3: Min State: IA, Max State: TN SELECT Fname FROM Table 1 WHERE State = ‘NY’ High Performance Query Processing ID 1 2 3 4 ... 8M 8M+1 ... 16M 16M+1 ... 24M Fname Bugs Yosemite Daffy Hazel ... ... Jane ... Elmer Lname Bunny Sam Duck Fudd ... ... ... State NY CA NY ME ... MN WY TX OR ... VA TN IA NY ... PA Zip 11217 95389 10013 04578 ... ... ... Phone (718) 938-3235 (209) 375-6572 (212) 227-1810 (207) 882-7323 ... ... ... Age 34 52 35 43 ... ... ... Sex M M M F ... ... ... Vertical Partition Vertical Partition Vertical Partition Vertical Partition Vertical Partition … ELIMINATED PARTITION 23
- 23. Analytics • Analytics – In-database distributed analytics with complex join, aggregation, window functions – Extensible UDF for custom analytics – Cross Engine Join with other storage engines • BI connectivity – Out of box BI tool connectivity with ODBC/ JDBC or standard MariaDB Connectors Daily Running Average Revenue for each item SELECT item_id, server_date, daily_revenue, AVG(revenue) OVER (PARTITION BY item_id ORDER BY server_date RANGE INTERVAL '1' DAY PRECEDING ) running_avg FROM web_item_sales Item ID Server_date Revenue 1 02-01-2014 20,000.00 1 02-02-2014 5,001.00 2 02-01-2014 15,000.00 2 02-04-2014 34,029.00 2 02-05-2014 7,138.00 3 02-01-2014 17,250.00 3 02-03-2014 25,010.00 3 02-04-2014 21,034.00 3 02-05-2014 4,120.00 Running Average 20,000.00 12,500.50 15,000.00 34,209.00 20,583.50 17,250.00 250,100.00 12,577.00 20,583.50 24
- 24. Enterprise Grade • Enterprise grade security – SSL, role based access, audit • Flexibility of Platform – Run on on-premise commodity Linux servers – Run on AWS • High Availability – Automatic UM failover – Automatic PM failover with distributed data attachment across all PMs in SAN and EBS environment Shared-Nothing Distributed Data Storage Compressed by default User Module (UM) Performance Module (PM) Data Storage 25
- 25. MariaDB ColumnStore 1.0.11 (released 18th August 2017) Data Engine ● Columnar Engine based on MariaDB 10.1.26 Scale ● Columnar, Massively Parallel ● Linear scalability with automatic data partitioning ● Data compression designed to accelerate decompression rate, reducing disk I/O Performance ● High performance analytics ● Columnar optimized, massively parallel, distributed query processing on commodity servers Data Ingestion ● High speed parallel data load and extract without blocking reads Analytics ● In database analytics with complex joins, windowing functions ● Extensible User Defined Functions (UDF) for custom analytics ● Out of box BI Tools connectivity, Analytics integration with R Enterprise Grade ● Cross join tables between MariaDB and ColumnStore for full insight ● SSL support, Auditability, Role Based Access ● Built-in High availability for UM and PM Ease of Use ● Automatic horizontal partitioning ● No index, views or manual partition tuning needed ● Online schema changes while read queries continue ● Deploy anywhere on premise or AWS 26
- 26. ColumnStore 1.1 Theme: Streaming Analytics Enablement Data Engine ● Columnar Engine based on MariaDB 10.2 ● Re-implement Window functions based on server code Streaming ● Native API for ColumnStore files - BSL(Business Source License) ● Streaming Insert via MaxScale ● Streaming Change Data Capture(CDC) via MaxScale & Kafka Performance ● Small table / small result set performance improvement Data Ingestion ● High speed parallel data load and extract w/o blocking reads Analytics ● In database analytics with complex joins, windowing functions ● Extensible User Defined Functions (UDF) for custom analytics ● Out of box BI Tools connectivity, Analytics integration with R (www.r-project.org) High Availability ● Integrated GlusterFS Security ● Audit Plugin integration Ease of Use ● Non-root support Phase 1 ● Backup/Restore tool - BSL(Business Source License) 27
- 27. MariaDB AX / ColumnStore in Summary Flexible deployment: cloud or on-premise commodity server Open source big data Analytics High data compression In-database distributed analytics Cross-join with OLTP engines Enterprise grade security and high availability Easy to manage and scale Parallel, distributed query processing Columnar optimized High data compression Faster, More Efficient Queries Easier Enterprise Analytics Better Price Performance 28
- 28. Where to find MariaDB AX / ColumnStore? SOFTWARE DOWNLOAD https://mariadb.com/downloads/columnstore SOURCE https://github.com/mariadb-corporation/mariadb-columnstore-engine DOCUMENTATION https://mariadb.com/kb/en/mariadb/mariadb-columnstore/ BLOGS https://mariadb.com/blog-tags/columnstore 29
- 29. The picture can't be displayed. M|18 FEB 26–27, 2018 CONRAD NEW YORK m18.mariadb.com
- 30. The picture can't be displayed. Thank you