MariaDB CONNECT Storage Engine
1 like4,673 views
The document discusses the MariaDB CONNECT storage engine, which allows querying external file formats from within MariaDB. It was created by database expert Olivier Bertrand and brings business intelligence capabilities to MariaDB by enabling access to data sources like CSV, XML, Excel and other formats without needing ETL processes. The storage engine uses the MySQL plugin architecture and implements features like indexing, condition pushdown, and support for ODBC, MySQL tables, and various file types.
MariaDB CONNECT Storage Engine
- 1. SkySQL MariaDB CONNECT Storage Engine Serge Frezefond SkySQL Ab 2012 Confidential
- 2. SkySQL Ab 2012 Confidential Goal of the CONNECT Storage Engine : BI on various file formats Most of the data in companies is in various external datasources (many in non relational database format) : – Dbase, Firebird, SQlite – csv, – xml, – stored per column... – Microsoft Access & Excel – Distributed mysql servers Targeting BI data access on these formats. Not targeted for OLTP
- 3. SkySQL Ab 2012 Confidential Behind the scene Traditional BI Data is processed by an ETL – Change in the data model(denormalization...) Agregates are computed – Need to be defined and maintained Might need to move data out of RDBMS to other kind of datastore – OLAP, Collumn store, Hadoop/Hbase ... Specific tools are used to query the data IT is involved to maintain this machinery
- 4. SkySQL Ab 2012 Confidential MariaDB CONNECT Storage Engine : created by Olivier Bertrand IBM database researcher – Now retired, 50 years expertise programing Very experienced on databases – Worked on system-R, DB2, natural language query ... Discovered MySQL when looking for friendly place to test new concepts.(2004) – Decided to go open source – Started to appreciate the MariaDB openess and friendlyness
- 5. SkySQL Ab 2012 Confidential How did the CONNECT Storage Engine move to MariaDB? Olivier met Monty creator of MySQL and MariaDB a few years ago (2004 for other concepts) SkySQL met Olivier and we start working together : – First access to launchpad, go to linux, test, Olivier start working with MariaDB team : – Testing, bug fixes, security, test cases ... SkySQL / MariaDB and Olivier agreed that is was ready to be released and supported under GPL – MariaDB flexibility ease integration
- 6. SkySQL Ab 2012 Confidential The CONNECT Storage Engine Uses the MySQL Plugin Architecture A major differentiator of MySQL Allow very specific datastore to interact with the MySQL sql layer Allow advanced interaction – Specific Create Table parameters(MariaDB) – Condition push down Allow join with other storage engines – InnoDB / MyISAM tables
- 7. SkySQL Ab 2012 Confidential The CONNECT Storage Engine implements advanced features ● Support of external data sources : – Odbc, MySQL, WMI ... ● Support multi files tables ● Support Big File Table > 2G ● Support virtual tables (DIR) ● Add autocreate of tables : – The structure is discovered from the data source ● Use MariaDB create table new parameters capability (avoid comments polution) ● Support compressed tables
- 8. SkySQL Ab 2012 Confidential The CONNECT Storage Engine implements advanced features ● Add indexing to files – index optimized for read ● Condition Push down – Used with ODBC and MySQL to push condition to the target database. Big perf gain. ● Support MariaDB virtuals columns ● Support of special columns : – Rowid, fileid, tabid, servid ● Muti tables table (like merge) – Different structure, not myisam only, remotely distributed tables
- 9. SkySQL Ab 2012 Confidential CONNECT Storage Engine vs. Federated(X) storage Engine - support ODBC – With muti files ODBC - support access to MySQL table - support the limit clause
- 10. SkySQL Ab 2012 Confidential CONNECT Storage Engine ODBC component Allow to access to any datasource accessible through ODBC. – Excel – Access – Firebird – SQLite – SQL Server, Oracle, DB2 – ... Possibility to do multifiles ODBC – To query consolidated monthly excel datasheet
- 11. SkySQL Ab 2012 Confidential The CONNECT Storage Engine implements advanced features Catalog table : – For Example Describe for odbc table – No need to do create table – Access to data / column metadata Memory file maping – For file type table (not xml) Table format .ini
- 12. SkySQL Ab 2012 Confidential The CONNECT Storage Engine implements advanced features Multiple CONNECT tables can be created on the same underlying file – Indexes can be shared between tables
- 13. SkySQL Ab 2012 Confidential Connect Storage Engine VEC table / Column store col1col1 col2col1 col3col2col1 row3row2row1 col3 col1 freefree freefreefree col3 free -1 or per column file - Indexes work - Fixed size record
- 14. SkySQL Ab 2012 Confidential Connect Storage Engine Table List Table (// Merge) col1col1 col2 col1 col2 col3 col3col1 col2 col3col1 col2 ODBC table MySQL table col4 Muti tables table (like merge) – Different structure, not myisam only, remotely distributed tables
- 15. SkySQL Ab 2012 Confidential Connect Storage Engine vs. MySQL Merge tables Table list table : - support non MyISAM tables - no need to the exact same structure for table - underlying tables can be remote – Distributed architecture
- 16. SkySQL Ab 2012 Confidential Importing /exporting MySQL data in various formats Importing file data into MySQL tables – Here for example from an XML file : • create table biblio select * from xsampall2; Exporting data from MySQL – Here for example we export to XML format : create table handout engine=CONNECT table_type=XML file_name='handout.htm' header=yes option_list='name=TABLE,coltype=HTML,attribute=border=1;cellpadding=5' select plugin_name handler, plugin_version version, plugin_author author, plugin_description description, plugin_maturity maturity from information_schema.plugins where plugin_type = 'STORAGE ENGINE';
- 17. SkySQL Ab 2012 Confidential Where is the MariaDB Connect Storage Engine available ? ● It is 100 % open source ● It is available on MariaDB launchpad ● Open Bug database ● Public Roadmap ● Released test cases ● Improvement request / worklog
- 18. SkySQL Ab 2012 Confidential How you can help Adopt it / Test it. Bugs : report bugs / propose fixes Documentation : help improve it Sharing : test it, blog about it, – Share your experience about interesting usages.
- 19. SkySQL Ab 2012 Confidential Conclusion ● The MariaDB Connect Storage Engine : ● Open MariaDB to BI and data analysis ● Brings real value to MariaDB users ● Illustrates openess of MariaDB community ● Supported by SkySQL / MariaDB
- 20. SkySQL Ab 2012 Confidential 20 Thank You Serge Frezefond www.skysql.coml www.facebook.com/skysql www.linkedin.com/company/skysql
- 21. SkySQL Ab 2012 Confidential Enhancements considered for MariaDB connect storage engine - parallel access to multitables table -