Introducing new SQL syntax and improving performance with preparse Query Rewrite Plugins
3 likes1,085 views
This document discusses a new preparse query rewrite plugin for MySQL that allows adding new SQL syntax like the FILTER clause from SQL:2003. The plugin works by catching the query before parsing and rewriting parts using regular expressions. It also describes extending the plugin to support custom optimizer hints by modifying and restoring thread-specific variable values.
![Memory management for plugins
#include <my_thread.h> // my_thread_handle needed by mysql_memory.h
#include <mysql/psi/mysql_memory.h>
...
static PSI_memory_key key_memory_filter;
static PSI_memory_info all_rewrite_memory[]=
{
{ &key_memory_filter, "filter", 0 }
};
static int filter_plugin_init(MYSQL_PLUGIN plugin_ref)
{
plugin_info_ptr= plugin_ref;
const char* category= "sql";
int count;
count= array_elements(all_rewrite_memory);
mysql_memory_register(category, all_rewrite_memory, count);
return 0; /* success */
}
12 www.percona.com](https://image.slidesharecdn.com/queryrewriteplugins-160125125649/85/Introducing-new-SQL-syntax-and-improving-performance-with-preparse-Query-Rewrite-Plugins-16-320.jpg)
![Catching up the query
static int filter(MYSQL_THD thd, // MySQL Thread object
mysql_event_class_t event_class, // Class of the event
const void *event // Event itself
)
{
const struct mysql_event_parse *event_parse=
static_cast<const struct mysql_event_parse *>(event);
if (event_parse->event_subclass != MYSQL_AUDIT_PARSE_PREPARSE)
return 0;
string subject= event_parse->query.str; // Original query
string rewritten_query;
//requires std::regex and GCC 4.9+
regex filter_clause_star("(COUNT)((s**s*))s+"
+ "FILTERs*(s*WHEREs+([^)]+)s*)",
ECMAScript | icase);
rewritten_query= regex_replace(subject, filter_clause_star,
"$1(CASE WHEN $3 THEN 1 ELSE NULL END)");
...
15 www.percona.com](https://image.slidesharecdn.com/queryrewriteplugins-160125125649/85/Introducing-new-SQL-syntax-and-improving-performance-with-preparse-Query-Rewrite-Plugins-19-320.jpg)
![Rewritten query
void _rewrite_query(const void *event,
const struct mysql_event_parse *event_parse,
char const* new_query
)
{
char *rewritten_query= static_cast<char *>(my_malloc(
key_memory_filter, strlen(new_query) + 1, MYF(0)));
strncpy(rewritten_query, new_query, strlen(new_query));
rewritten_query[strlen(new_query)]= ’0’;
event_parse->rewritten_query->str= rewritten_query; // Rewritten query
event_parse->rewritten_query->length=strlen(new_query);
// You must set this flag to inform MySQL Server what query was rewritten
*((int *)event_parse->flags)|=
(int)MYSQL_AUDIT_PARSE_REWRITE_PLUGIN_QUERY_REWRITTEN;
}
16 www.percona.com](https://image.slidesharecdn.com/queryrewriteplugins-160125125649/85/Introducing-new-SQL-syntax-and-improving-performance-with-preparse-Query-Rewrite-Plugins-20-320.jpg)
![Store previous values
// map to store modified variables
static map <my_thread_id, map<supported_hints_t, ulonglong> > modified_variables;
...
/* The job */
static int custom_hint(MYSQL_THD thd, mysql_event_class_t event_class,
const void *event)
{
...
// If we have a match store create map of thread variables
std::map<supported_hints_t, ulonglong> current;
...
After processing variables store them in modified_variables map
modified_variables[thd->thread_id()]= current;
...
21 www.percona.com](https://image.slidesharecdn.com/queryrewriteplugins-160125125649/85/Introducing-new-SQL-syntax-and-improving-performance-with-preparse-Query-Rewrite-Plugins-27-320.jpg)
![Modify variables
• Since we have access to MYSQL THD
this is easy:
switch(get_hint_switch(ssm[1]))
{
case JOIN_BUFFER_SIZE:
current[JOIN_BUFFER_SIZE]= thd->variables.join_buff_size;
thd->variables.join_buff_size= stoull(ssm[2]);
break;
case TMP_TABLE_SIZE:
current[TMP_TABLE_SIZE]= thd->variables.tmp_table_size;
thd->variables.tmp_table_size= stoull(ssm[2]);
break;
...
22 www.percona.com](https://image.slidesharecdn.com/queryrewriteplugins-160125125649/85/Introducing-new-SQL-syntax-and-improving-performance-with-preparse-Query-Rewrite-Plugins-28-320.jpg)
![Customization: credentials
• We have access to
• MYSQL_THR->security_context
• thd->security_context()->user().str
• Password still has to be in the configuration file, under
[client]
or
[toolname]
section
• System variables
• Since we are interested in backing up local server we will use
mysqld_unix_port
28 www.percona.com](https://image.slidesharecdn.com/queryrewriteplugins-160125125649/85/Introducing-new-SQL-syntax-and-improving-performance-with-preparse-Query-Rewrite-Plugins-38-320.jpg)
![Customization: variables
• Global variables - Example only!
• Thread variables
• Add to plugin declaration
static struct st_mysql_sys_var *mysqlbackup_plugin_sys_vars[] = {
MYSQL_SYSVAR(backup_dir),
MYSQL_SYSVAR(backup_tool),
MYSQL_SYSVAR(backup_tool_basedir),
MYSQL_SYSVAR(backup_tool_options),
NULL
};
mysql_declare_plugin(mysqlbackup_plugin)
{
MYSQL_AUDIT_PLUGIN,
&mysqlbackup_plugin_descriptor,
"mysqlbackup_plugin",
...
NULL, /* status variables */
mysqlbackup_plugin_sys_vars, /* system variables */
...
29 www.percona.com](https://image.slidesharecdn.com/queryrewriteplugins-160125125649/85/Introducing-new-SQL-syntax-and-improving-performance-with-preparse-Query-Rewrite-Plugins-41-320.jpg)
Introducing new SQL syntax and improving performance with preparse Query Rewrite Plugins
- 1. Preparse Query Rewrite Plugins New SQL syntax for fun & performance Sveta Smirnova Principal Support Engineer January, 30, 2016
- 2. Table of Contents •Introducing new SQL syntax •Working with results •Variables •Summary 2 www.percona.com
- 3. Introducing new SQL syntax 3 www.percona.com
- 4. MySQL often receives blames • From mascots and from humans 4 www.percona.com
- 5. MySQL often receives blames • From mascots and from humans • It cannot make a toast 4 www.percona.com
- 6. MySQL often receives blames • From mascots and from humans • It cannot make a toast • It does not support some syntax 4 www.percona.com
- 7. Or does it? • FILTER clause in MySQL on my home machine 5 www.percona.com
- 8. Or does it? • FILTER clause in MySQL on my home machine 5 www.percona.com
- 9. Or does it? • FILTER clause in MySQL on my home machine • But not in the user manual 5 www.percona.com
- 10. How is it done? • With 181 lines of code • Including comments! • And new Query Rewrite Plugin interface 6 www.percona.com
- 11. A little bit of history • First introduced in version 5.7.5 • Was available at MySQL Labs • Two types of plugins • Pre-parse • Post-parse 7 www.percona.com
- 12. Today • Part of Audit plugin interface • Step in at • MYSQL AUDIT GENERAL ALL • MYSQL AUDIT CONNECTION ALL • MYSQL AUDIT PARSE ALL • MYSQL AUDIT PARSE PREPARSE • MYSQL AUDIT PARSE POSTPARSE • MYSQL AUDIT AUTHORIZATION ALL • ... 8 www.percona.com
- 13. Plugin skeleton #include <mysql/plugin.h> #include <mysql/plugin_audit.h> - Audit plugin declaration ... static MYSQL_PLUGIN plugin_info_ptr; - Pointer to the plugin ... static int filter_plugin_init(MYSQL_PLUGIN plugin_ref); - Plugin initialization ... static int filter(MYSQL_THD thd, mysql_event_class_t event_class, const void *event); - Entry point for MYSQL_AUDIT_PARSE_PREPARSE ... static st_mysql_audit filter_plugin_descriptor; ... mysql_declare_plugin(filter_plugin); 9 www.percona.com
- 14. Plugin descriptor static st_mysql_audit filter_plugin_descriptor= { MYSQL_AUDIT_INTERFACE_VERSION, /* interface version */ NULL, filter, /* implements FILTER */ // You can also use MYSQL_AUDIT_PARSE_ALL { 0, 0, (unsigned long) MYSQL_AUDIT_PARSE_PREPARSE,} }; 10 www.percona.com
- 15. Plugin declaration mysql_declare_plugin(filter_plugin) { MYSQL_AUDIT_PLUGIN, &filter_plugin_descriptor, "filter_plugin", "Sveta Smirnova", "FILTER SQL:2003 support for MySQL", PLUGIN_LICENSE_GPL, filter_plugin_init, NULL, /* filter_plugin_deinit - TODO */ 0x0001, /* version 0.0.1 */ NULL, /* status variables */ NULL, /* system variables */ NULL, /* config options */ 0, /* flags */ } mysql_declare_plugin_end; 11 www.percona.com
- 16. Memory management for plugins #include <my_thread.h> // my_thread_handle needed by mysql_memory.h #include <mysql/psi/mysql_memory.h> ... static PSI_memory_key key_memory_filter; static PSI_memory_info all_rewrite_memory[]= { { &key_memory_filter, "filter", 0 } }; static int filter_plugin_init(MYSQL_PLUGIN plugin_ref) { plugin_info_ptr= plugin_ref; const char* category= "sql"; int count; count= array_elements(all_rewrite_memory); mysql_memory_register(category, all_rewrite_memory, count); return 0; /* success */ } 12 www.percona.com
- 17. SQL:2003 <filter clause> ::= FILTER <left paren> WHERE <search condition> <right paren> (10.9 <aggregate function>, 5WD-02-Foundation-2003-09.pdf, p.505) Only for aggregate functions: <computational operation> ::= AVG | MAX | MIN | SUM | EVERY | ANY | SOME | COUNT | STDDEV_POP | STDDEV_SAMP | VAR_SAMP | VAR_POP | COLLECT | FUSION | INTERSECTION <set quantifier> ::= DISTINCT | ALL MySQL only supports COUNT | AVG | SUM | MAX | MIN | STDDEV_POP | STDDEV_SAMP | VAR_SAMP | VAR_POP 13 www.percona.com
- 18. Implementing FILTER clause • FILTER is practically CASE WHEN foo THEN bar ELSE NULL • So we only need to catch FUNCTION(var) FILTER(WHERE foo) • And replace it with CASE 14 www.percona.com
- 19. Catching up the query static int filter(MYSQL_THD thd, // MySQL Thread object mysql_event_class_t event_class, // Class of the event const void *event // Event itself ) { const struct mysql_event_parse *event_parse= static_cast<const struct mysql_event_parse *>(event); if (event_parse->event_subclass != MYSQL_AUDIT_PARSE_PREPARSE) return 0; string subject= event_parse->query.str; // Original query string rewritten_query; //requires std::regex and GCC 4.9+ regex filter_clause_star("(COUNT)((s**s*))s+" + "FILTERs*(s*WHEREs+([^)]+)s*)", ECMAScript | icase); rewritten_query= regex_replace(subject, filter_clause_star, "$1(CASE WHEN $3 THEN 1 ELSE NULL END)"); ... 15 www.percona.com
- 20. Rewritten query void _rewrite_query(const void *event, const struct mysql_event_parse *event_parse, char const* new_query ) { char *rewritten_query= static_cast<char *>(my_malloc( key_memory_filter, strlen(new_query) + 1, MYF(0))); strncpy(rewritten_query, new_query, strlen(new_query)); rewritten_query[strlen(new_query)]= ’0’; event_parse->rewritten_query->str= rewritten_query; // Rewritten query event_parse->rewritten_query->length=strlen(new_query); // You must set this flag to inform MySQL Server what query was rewritten *((int *)event_parse->flags)|= (int)MYSQL_AUDIT_PARSE_REWRITE_PLUGIN_QUERY_REWRITTEN; } 16 www.percona.com
- 22. Can we do better? • Playing with syntax is fun • But can we introduce something more MySQL-ish? 18 www.percona.com
- 23. Custom hint plugin • MySQL 5.7 has Optimizer Hints SELECT /*+ NO_RANGE_OPTIMIZATION(t3 PRIMARY, f2_idx) */ f1 FROM t3 WHERE f1 > 30 AND f1 < 33; SELECT /*+ BKA(t1) NO_BKA(t2) */ * FROM t1 INNER JOIN t2 WHERE ...; SELECT /*+ NO_ICP(t1, t2) */ * FROM t1 INNER JOIN t2 WHERE ...; SELECT /*+ SEMIJOIN(FIRSTMATCH, LOOSESCAN) */ * FROM t1 ...; EXPLAIN SELECT /*+ NO_ICP(t1) */ * FROM t1 WHERE ...; 19 www.percona.com
- 24. Custom hint plugin • MySQL 5.7 has Optimizer Hints • But sometimes thread-specific buffers affect query execution • Common workaround exists: SET tmp_table_size=1073741824; SELECT * FROM t1 INNER JOIN t2 WHERE ...; SET tmp_table_size=DEFAULT; • Workaround requires processing result set of each of these statements 19 www.percona.com
- 25. Custom hint plugin • MySQL 5.7 has Optimizer Hints • But sometimes thread-specific buffers affect query execution • Workaround requires processing result set of each of these statements • This is why I extended optimizer hint syntax SELECT /*+ join_buffer_size=16384 */ f1 FROM t3 WHERE f1 > 30 AND f1 < 33; SELECT /*+ tmp_table_size=1073741824 BKA(t1) NO_BKA(t2) */ * FROM t1 INNER JOIN t2 WHERE ...; 19 www.percona.com
- 26. New features • For Custom Hints we need to: • Store previous values of thread variables we are going to modify • Modify variables • Revert them back before sending result 20 www.percona.com
- 27. Store previous values // map to store modified variables static map <my_thread_id, map<supported_hints_t, ulonglong> > modified_variables; ... /* The job */ static int custom_hint(MYSQL_THD thd, mysql_event_class_t event_class, const void *event) { ... // If we have a match store create map of thread variables std::map<supported_hints_t, ulonglong> current; ... After processing variables store them in modified_variables map modified_variables[thd->thread_id()]= current; ... 21 www.percona.com
- 28. Modify variables • Since we have access to MYSQL THD this is easy: switch(get_hint_switch(ssm[1])) { case JOIN_BUFFER_SIZE: current[JOIN_BUFFER_SIZE]= thd->variables.join_buff_size; thd->variables.join_buff_size= stoull(ssm[2]); break; case TMP_TABLE_SIZE: current[TMP_TABLE_SIZE]= thd->variables.tmp_table_size; thd->variables.tmp_table_size= stoull(ssm[2]); break; ... 22 www.percona.com
- 29. Revert variables back • First we need to tell plugin descriptor what we need MYSQL AUDIT GENERAL RESULT static st_mysql_audit custom_hint_plugin_descriptor= { MYSQL_AUDIT_INTERFACE_VERSION, /* interface version */ NULL, custom_hint, /* implements custom hints */ { (unsigned long) MYSQL_AUDIT_GENERAL_RESULT, 0, (unsigned long) MYSQL_AUDIT_PARSE_PREPARSE, } }; 23 www.percona.com
- 30. Revert variables back • First we need to tell plugin descriptor what we need MYSQL AUDIT GENERAL RESULT • Then revert variables before sending result if (event_general->event_subclass == MYSQL_AUDIT_GENERAL_RESULT) { map<my_thread_id, map<supported_hints_t, ulonglong> >::iterator current= modified_variables.find(thd->thread_id()); if (current != modified_variables.end()) { for (map<supported_hints_t, ulonglong>::iterator it= current->second.begin(); it!= current->second.end(); ++it) { switch(it->first) { case JOIN_BUFFER_SIZE: thd->variables.join_buff_size= it->second; break; 23 www.percona.com
- 31. Revert variables back • First we need to tell plugin descriptor what we need MYSQL AUDIT GENERAL RESULT • Then revert variables before sending result • And, finally, erase stored values for current thread: modified_variables.erase(current); 23 www.percona.com
- 32. Before Custom Hint Plugin mysql> flush status; Query OK, 0 rows affected (0.00 sec) mysql> select count(*), sum(c) from -> (select s, count(s) c from joinit where i < 1000000 group by s) t; +----------+--------+ | count(*) | sum(c) | +----------+--------+ | 737882 | 737882 | +----------+--------+ 1 row in set (24.70 sec) mysql> show status like ’Created_tmp_disk_tables’; +-------------------------+-------+ | Variable_name | Value | +-------------------------+-------+ | Created_tmp_disk_tables | 2 | -- 2 temporary tables on disk +-------------------------+-------+ 1 row in set (0.00 sec) 24 www.percona.com
- 33. Custom Hint Plugin at work mysql> flush status; Query OK, 0 rows affected (0.00 sec) mysql> select /*+ tmp_table_size=134217728 max_heap_table_size=134217728 */ -> count(*), sum(c) from -> (select s, count(s) c from joinit where i < 1000000 group by s) t; +----------+--------+ | count(*) | sum(c) | +----------+--------+ | 737882 | 737882 | +----------+--------+ 1 row in set, 2 warnings (6.21 sec) -- 4 times speed gain! mysql> show status like ’Created_tmp_disk_tables’; +-------------------------+-------+ | Variable_name | Value | +-------------------------+-------+ | Created_tmp_disk_tables | 0 | -- No disk-based temporary table! +-------------------------+-------+ 1 row in set (0.00 sec) 25 www.percona.com
- 35. BACKUP DATABASE plugin • Very simple syntax • mysql> BACKUP SERVER; +---------------------------------+ | Backup finished with status OK! | +---------------------------------+ | Backup finished with status OK! | +---------------------------------+ 1 row in set, 1 warning (42.92 sec) 27 www.percona.com
- 36. BACKUP DATABASE plugin • Very simple syntax • Supports many tools • mysqldump • mysqlpump • mysqlbackup • XtraBackup - Planned! 27 www.percona.com
- 37. BACKUP DATABASE plugin • Very simple syntax • Supports many tools • Needs to pass options • Credentials • Backup directory • Custom 27 www.percona.com
- 38. Customization: credentials • We have access to • MYSQL_THR->security_context • thd->security_context()->user().str • Password still has to be in the configuration file, under [client] or [toolname] section • System variables • Since we are interested in backing up local server we will use mysqld_unix_port 28 www.percona.com
- 39. Customization: variables • Global variables - Example only! static MYSQL_SYSVAR_STR(backup_dir, backup_dir_value, PLUGIN_VAR_MEMALLOC, "Default directory...", NULL, NULL, NULL); static MYSQL_SYSVAR_ENUM(backup_tool, backup_tool_name, PLUGIN_VAR_RQCMDARG, "Backup tool. Possible values: mysqldump|mysqlbackup", NULL, NULL, MYSQLDUMP, &supported_tools_typelib); • Thread variables • Add to plugin declaration 29 www.percona.com
- 40. Customization: variables • Global variables - Example only! • Thread variables static MYSQL_THDVAR_STR(backup_dir, PLUGIN_VAR_MEMALLOC, "Default directory...", NULL, NULL, NULL); static MYSQL_THDVAR_ENUM(backup_tool, PLUGIN_VAR_RQCMDARG, "Backup tool. Possible values: mysqldump|mysqlbackup|mysqlpump", NULL, NULL, MYSQLDUMP, &supported_tools_typelib); ... • Add to plugin declaration 29 www.percona.com
- 41. Customization: variables • Global variables - Example only! • Thread variables • Add to plugin declaration static struct st_mysql_sys_var *mysqlbackup_plugin_sys_vars[] = { MYSQL_SYSVAR(backup_dir), MYSQL_SYSVAR(backup_tool), MYSQL_SYSVAR(backup_tool_basedir), MYSQL_SYSVAR(backup_tool_options), NULL }; mysql_declare_plugin(mysqlbackup_plugin) { MYSQL_AUDIT_PLUGIN, &mysqlbackup_plugin_descriptor, "mysqlbackup_plugin", ... NULL, /* status variables */ mysqlbackup_plugin_sys_vars, /* system variables */ ... 29 www.percona.com
- 43. More possibilities • Custom locks • Access to thread- and server-specific variables • Fine control at multiple steps of query execution • More 31 www.percona.com
- 44. Code • https://github.com/svetasmirnova/ • filter plugin • custom hint plugin • mysqlbackup plugin 32 www.percona.com
- 45. More information • MySQL source dir/plugin • rewriter • rewrite example • Writing Audit Plugins manual • MySQL Services for Plugins manual 33 www.percona.com
- 46. Place for your questions ??? 34 www.percona.com