Preparse Query Rewrite Plugins
1 like657 views
This document discusses new SQL syntax and query rewrite plugins in MySQL. It introduces FILTER clauses and custom optimizer hints and describes how to implement them using query rewrite plugins. Key points covered include the plugin interface, parsing and rewriting queries, managing memory, and customizing variables. The goal is to add new SQL features and control query execution through simple syntax extensions. Examples of implementing FILTER clauses and custom optimizer hints are provided to demonstrate how rewrite plugins work.
![#include <my_thread.h> // my_thread_handle needed by mysql_memory.h
#include <mysql/psi/mysql_memory.h>
...
static PSI_memory_key key_memory_filter;
static PSI_memory_info all_rewrite_memory[]=
{
{ &key_memory_filter, "filter", 0 }
};
static int filter_plugin_init(MYSQL_PLUGIN plugin_ref)
{
plugin_info_ptr= plugin_ref;
const char* category= "sql";
int count;
count= array_elements(all_rewrite_memory);
mysql_memory_register(category, all_rewrite_memory, count);
return 0; /* success */
}
Memory management for plugins
12](https://image.slidesharecdn.com/queryrewriteplugins-160927124521/85/Preparse-Query-Rewrite-Plugins-16-320.jpg)
![static int filter(MYSQL_THD thd, // MySQL Thread object
mysql_event_class_t event_class, // Class of the event
const void *event // Event itself
)
{
const struct mysql_event_parse *event_parse=
static_cast<const struct mysql_event_parse *>(event);
if (event_parse->event_subclass != MYSQL_AUDIT_PARSE_PREPARSE)
return 0;
string subject= event_parse->query.str; // Original query
string rewritten_query;
//requires std::regex and GCC 4.9+
regex filter_clause_star("(COUNT)((s**s*))s+"
+ "FILTERs*(s*WHEREs+([^)]+)s*)",
ECMAScript | icase);
rewritten_query= regex_replace(subject, filter_clause_star,
"$1(CASE WHEN $3 THEN 1 ELSE NULL END)");
Catching up the query
15](https://image.slidesharecdn.com/queryrewriteplugins-160927124521/85/Preparse-Query-Rewrite-Plugins-19-320.jpg)
![void _rewrite_query(const void *event,
const struct mysql_event_parse *event_parse,
char const* new_query
)
{
char *rewritten_query= static_cast<char *>(my_malloc(
key_memory_filter, strlen(new_query) + 1, MYF(0)));
strncpy(rewritten_query, new_query, strlen(new_query));
rewritten_query[strlen(new_query)]= ’0’;
event_parse->rewritten_query->str= rewritten_query; // Rewritten query
event_parse->rewritten_query->length=strlen(new_query);
// You must set this flag to inform MySQL Server what query was rewritten
*((int *)event_parse->flags)|=
(int)MYSQL_AUDIT_PARSE_REWRITE_PLUGIN_QUERY_REWRITTEN;
}
Rewritten query
16](https://image.slidesharecdn.com/queryrewriteplugins-160927124521/85/Preparse-Query-Rewrite-Plugins-20-320.jpg)
![// map to store modified variables
static map <my_thread_id, map<supported_hints_t, ulonglong> > modified_variables;
...
/* The job */
static int custom_hint(MYSQL_THD thd, mysql_event_class_t event_class,
const void *event)
{
...
// If we have a match store create map of thread variables
std::map<supported_hints_t, ulonglong> current;
...
After processing variables store them in modified_variables map
modified_variables[thd->thread_id()]= current;
...
Store previous values
21](https://image.slidesharecdn.com/queryrewriteplugins-160927124521/85/Preparse-Query-Rewrite-Plugins-29-320.jpg)
![• Since we have access to MYSQL THD this
is easy:
switch(get_hint_switch(ssm[1]))
{
case JOIN_BUFFER_SIZE:
current[JOIN_BUFFER_SIZE]= thd->variables.join_buff_size;
thd->variables.join_buff_size= stoull(ssm[2]);
break;
case TMP_TABLE_SIZE:
current[TMP_TABLE_SIZE]= thd->variables.tmp_table_size;
thd->variables.tmp_table_size= stoull(ssm[2]);
break;
...
Modify variables
22](https://image.slidesharecdn.com/queryrewriteplugins-160927124521/85/Preparse-Query-Rewrite-Plugins-30-320.jpg)
![•
We have access to
• MYSQL_THR->security_context
thd->security_context()->user().str
Password still has to be in the configuration file, under
[client]
or
[toolname]
section
• System variables
Since we are interested in backing up local server we will use
mysqld_unix_port
Customization: credentials
28](https://image.slidesharecdn.com/queryrewriteplugins-160927124521/85/Preparse-Query-Rewrite-Plugins-40-320.jpg)
![• Global variables - Example only!
• Thread variables
•
Add to plugin declaration
static struct st_mysql_sys_var *mysqlbackup_plugin_sys_vars[] = {
MYSQL_SYSVAR(backup_dir),
...
MYSQL_SYSVAR(backup_tool_options),
NULL
};
mysql_declare_plugin(mysqlbackup_plugin) {
MYSQL_AUDIT_PLUGIN,
&mysqlbackup_plugin_descriptor,
"mysqlbackup_plugin",
...
NULL, /* status variables */
mysqlbackup_plugin_sys_vars, /* system variables */
...
Customization: variables
29](https://image.slidesharecdn.com/queryrewriteplugins-160927124521/85/Preparse-Query-Rewrite-Plugins-43-320.jpg)
Preparse Query Rewrite Plugins
- 1. Preparse Query Rewrite Plugins New SQL syntax for fun & performance September, 13, 2016 Sveta Smirnova
- 2. •Introducing new SQL syntax •Working with results •Variables •Summary Table of Contents 2
- 3. Introducing new SQL syntax
- 4. • From mascots and from humans MySQL often receives blames 4
- 5. • From mascots and from humans • It cannot make a toast MySQL often receives blames 4
- 6. • From mascots and from humans • It cannot make a toast • It does not support some syntax MySQL often receives blames 4
- 7. • FILTER clause in MySQL on my home machine Or does it? 5
- 8. • FILTER clause in MySQL on my home machine Or does it? 5
- 9. • FILTER clause in MySQL on my home machine • But not in the user manual Or does it? 5
- 10. • With 181 lines of code • Including comments! • And new Query Rewrite Plugin interface How is it done? 6
- 11. • First introduced in version 5.7.5 • Was available at MySQL Labs • Two types of plugins • Pre-parse • Post-parse A little bit of history 7
- 12. • Part of Audit plugin interface • Step in at • MYSQL AUDIT GENERAL ALL • MYSQL AUDIT CONNECTION ALL • MYSQL AUDIT PARSE ALL MYSQL AUDIT PARSE PREPARSE MYSQL AUDIT PARSE POSTPARSE • MYSQL AUDIT AUTHORIZATION ALL • ... Today 8
- 13. #include <mysql/plugin.h> #include <mysql/plugin_audit.h> - Audit plugin declaration ... static MYSQL_PLUGIN plugin_info_ptr; - Pointer to the plugin ... static int filter_plugin_init(MYSQL_PLUGIN plugin_ref); - Plugin initialization ... static int filter(MYSQL_THD thd, mysql_event_class_t event_class, const void *event); - Entry point for MYSQL_AUDIT_PARSE_PREPARSE ... static st_mysql_audit filter_plugin_descriptor; ... mysql_declare_plugin(filter_plugin); Plugin skeleton 9
- 14. static st_mysql_audit filter_plugin_descriptor= { MYSQL_AUDIT_INTERFACE_VERSION, /* interface version */ NULL, filter, /* implements FILTER */ // You can also use MYSQL_AUDIT_PARSE_ALL { 0, 0, (unsigned long) MYSQL_AUDIT_PARSE_PREPARSE,} }; Plugin descriptor 10
- 15. mysql_declare_plugin(filter_plugin) { MYSQL_AUDIT_PLUGIN, &filter_plugin_descriptor, "filter_plugin", "Sveta Smirnova", "FILTER SQL:2003 support for MySQL", PLUGIN_LICENSE_GPL, filter_plugin_init, NULL, /* filter_plugin_deinit - TODO */ 0x0001, /* version 0.0.1 */ NULL, /* status variables */ NULL, /* system variables */ NULL, /* config options */ 0, /* flags */ } mysql_declare_plugin_end; Plugin declaration 11
- 16. #include <my_thread.h> // my_thread_handle needed by mysql_memory.h #include <mysql/psi/mysql_memory.h> ... static PSI_memory_key key_memory_filter; static PSI_memory_info all_rewrite_memory[]= { { &key_memory_filter, "filter", 0 } }; static int filter_plugin_init(MYSQL_PLUGIN plugin_ref) { plugin_info_ptr= plugin_ref; const char* category= "sql"; int count; count= array_elements(all_rewrite_memory); mysql_memory_register(category, all_rewrite_memory, count); return 0; /* success */ } Memory management for plugins 12
- 17. <filter clause> ::= FILTER <left paren> WHERE <search condition> <right paren> (10.9 <aggregate function>, 5WD-02-Foundation-2003-09.pdf, p.505) Only for aggregate functions: <computational operation> ::= AVG | MAX | MIN | SUM | EVERY | ANY | SOME | COUNT | STDDEV_POP | STDDEV_SAMP | VAR_SAMP | VAR_POP | COLLECT | FUSION | INTERSECTION <set quantifier> ::= DISTINCT | ALL MySQL only supports COUNT | AVG | SUM | MAX | MIN | STDDEV_POP | STDDEV_SAMP | VAR_SAMP | VAR_POP SQL:2003 13
- 18. • FILTER is practically CASE WHEN foo THEN bar ELSE NULL • So we only need to catch FUNCTION(var) FILTER(WHERE foo) • And replace it with CASE Implementing FILTER clause 14
- 19. static int filter(MYSQL_THD thd, // MySQL Thread object mysql_event_class_t event_class, // Class of the event const void *event // Event itself ) { const struct mysql_event_parse *event_parse= static_cast<const struct mysql_event_parse *>(event); if (event_parse->event_subclass != MYSQL_AUDIT_PARSE_PREPARSE) return 0; string subject= event_parse->query.str; // Original query string rewritten_query; //requires std::regex and GCC 4.9+ regex filter_clause_star("(COUNT)((s**s*))s+" + "FILTERs*(s*WHEREs+([^)]+)s*)", ECMAScript | icase); rewritten_query= regex_replace(subject, filter_clause_star, "$1(CASE WHEN $3 THEN 1 ELSE NULL END)"); Catching up the query 15
- 20. void _rewrite_query(const void *event, const struct mysql_event_parse *event_parse, char const* new_query ) { char *rewritten_query= static_cast<char *>(my_malloc( key_memory_filter, strlen(new_query) + 1, MYF(0))); strncpy(rewritten_query, new_query, strlen(new_query)); rewritten_query[strlen(new_query)]= ’0’; event_parse->rewritten_query->str= rewritten_query; // Rewritten query event_parse->rewritten_query->length=strlen(new_query); // You must set this flag to inform MySQL Server what query was rewritten *((int *)event_parse->flags)|= (int)MYSQL_AUDIT_PARSE_REWRITE_PLUGIN_QUERY_REWRITTEN; } Rewritten query 16
- 22. • Playing with syntax is fun • But can we introduce something more MySQL-ish? Can we do better? 18
- 23. • MySQL 5.7 has Optimizer Hints SELECT /*+ NO_RANGE_OPTIMIZATION(t3 PRIMARY, f2_idx) */ f1 FROM t3 WHERE f1 > 30 AND f1 < 33; SELECT /*+ BKA(t1) NO_BKA(t2) */ * FROM t1 INNER JOIN t2 WHERE ...; SELECT /*+ NO_ICP(t1, t2) */ * FROM t1 INNER JOIN t2 WHERE ...; SELECT /*+ SEMIJOIN(FIRSTMATCH, LOOSESCAN) */ * FROM t1 ...; EXPLAIN SELECT /*+ NO_ICP(t1) */ * FROM t1 WHERE ...; Custom hint plugin 19
- 24. • MySQL 5.7 has Optimizer Hints • But sometimes thread-specific buffers affect query execution • Common workaround exists:SET tmp_table_size=1073741824; SELECT * FROM t1 INNER JOIN t2 WHERE ...; SET tmp_table_size=DEFAULT; • Workaround requires processing result set of each of these statements Custom hint plugin 19
- 25. • MySQL 5.7 has Optimizer Hints • But sometimes thread-specific buffers affect query execution • Workaround requires processing result set of each of these statements • Percona Server has SET STATEMENT mysql> SET STATEMENT max_statement_time=1000 FOR SELECT user FROM user; +------------------+ | user | +------------------+ | foo | | root | ... Custom hint plugin 19
- 26. • MySQL 5.7 has Optimizer Hints • But sometimes thread-specific buffers affect query execution • Workaround requires processing result set of each of these statements • SET STATEMENT not supported by forksmysql> SET STATEMENT max_statement_time=1000 FOR SELECT user FROM user; ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ’max_statement_time=1000 FOR SELECT user FROM user’ at line 1 Custom hint plugin 19
- 27. • MySQL 5.7 has Optimizer Hints • But sometimes thread-specific buffers affect query execution • Workaround requires processing result set of each of these statements • SET STATEMENT not supported by forks • This is why I extended optimizer hint syntax SELECT /*+ join_buffer_size=16384 */ f1 FROM t3 WHERE f1 > 30 AND f1 < 33; SELECT /*+ tmp_table_size=1073741824 BKA(t1) NO_BKA(t2) */ * FROM t1 INNER JOIN t2 WHERE ...; Custom hint plugin 19
- 28. • For Custom Hints we need to: • Store previous values of thread variables we are going to modify • Modify variables • Revert them back before sending result New features 20
- 29. // map to store modified variables static map <my_thread_id, map<supported_hints_t, ulonglong> > modified_variables; ... /* The job */ static int custom_hint(MYSQL_THD thd, mysql_event_class_t event_class, const void *event) { ... // If we have a match store create map of thread variables std::map<supported_hints_t, ulonglong> current; ... After processing variables store them in modified_variables map modified_variables[thd->thread_id()]= current; ... Store previous values 21
- 30. • Since we have access to MYSQL THD this is easy: switch(get_hint_switch(ssm[1])) { case JOIN_BUFFER_SIZE: current[JOIN_BUFFER_SIZE]= thd->variables.join_buff_size; thd->variables.join_buff_size= stoull(ssm[2]); break; case TMP_TABLE_SIZE: current[TMP_TABLE_SIZE]= thd->variables.tmp_table_size; thd->variables.tmp_table_size= stoull(ssm[2]); break; ... Modify variables 22
- 31. • First we need to specify what we need MYSQL AUDIT GENERAL RESULTstatic st_mysql_audit custom_hint_plugin_descriptor= { MYSQL_AUDIT_INTERFACE_VERSION, /* interface version */ NULL, custom_hint, /* implements custom hints */ { (unsigned long) MYSQL_AUDIT_GENERAL_RESULT, 0, (unsigned long) MYSQL_AUDIT_PARSE_PREPARSE, } }; Revert variables back 23
- 32. • First we need to specify what we need MYSQL AUDIT GENERAL RESULT • Then revert variables before sending result if (event_general->event_subclass == MYSQL_AUDIT_GENERAL_RESULT) { map<my_thread_id, map<supported_hints_t, ulonglong> >::iterator current= modified_variables.find(thd->thread_id()); if (current != modified_variables.end()) { for (map<supported_hints_t, ulonglong>::iterator it= current->second.begin(); it!= current->second.end(); ++it) { switch(it->first) { case JOIN_BUFFER_SIZE: thd->variables.join_buff_size= it->second; break; Revert variables back 23
- 33. • First we need to specify what we need MYSQL AUDIT GENERAL RESULT • Then revert variables before sending result • And, finally, erase stored values for current thread: modified_variables.erase(current); Revert variables back 23
- 34. mysql> flush status; Query OK, 0 rows affected (0.00 sec) mysql> select count(*), sum(c) from -> (select s, count(s) c from joinit where i < 1000000 group by s) t; +----------+--------+ | count(*) | sum(c) | +----------+--------+ | 737882 | 737882 | +----------+--------+ 1 row in set (24.70 sec) mysql> show status like ’Created_tmp_disk_tables’; +-------------------------+-------+ | Variable_name | Value | +-------------------------+-------+ | Created_tmp_disk_tables | 2 | -- 2 temporary tables on disk +-------------------------+-------+ 1 row in set (0.00 sec) Before Custom Hint Plugin 24
- 35. mysql> flush status; Query OK, 0 rows affected (0.00 sec) mysql> select /*+ tmp_table_size=134217728 max_heap_table_size=134217728 */ -> count(*), sum(c) from -> (select s, count(s) c from joinit where i < 1000000 group by s) t; +----------+--------+ | count(*) | sum(c) | +----------+--------+ | 737882 | 737882 | +----------+--------+ 1 row in set, 2 warnings (6.21 sec) -- 4 times speed gain! mysql> show status like ’Created_tmp_disk_tables’; +-------------------------+-------+ | Variable_name | Value | +-------------------------+-------+ | Created_tmp_disk_tables | 0 | -- No disk-based temporary table! +-------------------------+-------+ 1 row in set (0.00 sec) Custom Hint Plugin at work 25
- 36. Variables
- 37. • Very simple syntax • mysql> BACKUP SERVER; +---------------------------------+ | Backup finished with status OK! | +---------------------------------+ | Backup finished with status OK! | +---------------------------------+ 1 row in set, 1 warning (42.92 sec) BACKUP DATABASE plugin 27
- 38. • Very simple syntax • Supports many tools • mysqldump • mysqlpump • mysqlbackup • xtrabackup BACKUP DATABASE plugin 27
- 39. • Very simple syntax • Supports many tools • Needs to pass options • Credentials • Backup directory • Custom BACKUP DATABASE plugin 27
- 40. • We have access to • MYSQL_THR->security_context thd->security_context()->user().str Password still has to be in the configuration file, under [client] or [toolname] section • System variables Since we are interested in backing up local server we will use mysqld_unix_port Customization: credentials 28
- 41. • Global variables - Example only! static MYSQL_SYSVAR_STR(backup_dir, backup_dir_value, PLUGIN_VAR_MEMALLOC, "Default directory...", NULL, NULL, NULL); static MYSQL_SYSVAR_ENUM(backup_tool, backup_tool_name, PLUGIN_VAR_RQCMDARG, "Backup tool. Possible values: mysqldump|mysqlbackup", NULL, NULL, MYSQLDUMP, &supported_tools_typelib); • Thread variables • Add to plugin declaration Customization: variables 29
- 42. • Global variables - Example only! • Thread variables static MYSQL_THDVAR_STR(backup_dir, PLUGIN_VAR_MEMALLOC, "Default directory...", NULL, NULL, NULL); static MYSQL_THDVAR_ENUM(backup_tool, PLUGIN_VAR_RQCMDARG, "Backup tool. Possible values: mysqldump|mysqlbackup|mysqlpump", NULL, NULL, MYSQLDUMP, &supported_tools_typelib); ... • Add to plugin declaration Customization: variables 29
- 43. • Global variables - Example only! • Thread variables • Add to plugin declaration static struct st_mysql_sys_var *mysqlbackup_plugin_sys_vars[] = { MYSQL_SYSVAR(backup_dir), ... MYSQL_SYSVAR(backup_tool_options), NULL }; mysql_declare_plugin(mysqlbackup_plugin) { MYSQL_AUDIT_PLUGIN, &mysqlbackup_plugin_descriptor, "mysqlbackup_plugin", ... NULL, /* status variables */ mysqlbackup_plugin_sys_vars, /* system variables */ ... Customization: variables 29
- 44. Summary
- 45. • Custom locks • Access to thread- and server-specific variables • Fine control at multiple steps of query execution • More More possibilities 31
- 46. • https://github.com/svetasmirnova/ • filter plugin • custom hint plugin • mysqlbackup plugin Code 32
- 47. • MySQL source dir/plugin • rewriter • rewrite example • Writing Audit Plugins manual • MySQL Services for Plugins manual More information 33
- 48. When: October 3-5, 2016 Where: Amsterdam, Netherlands Percona Live Europe Open Source Database Conference is the premier event for the diverse and active open source community, as well as businesses that develop and use open source software. Use promo code ParisMeetup to get 25 euros off. Register now Sponsorship opportunities available as well here. Join us at Percona Live Europe 34
- 49. ??? Place for your questions 35