FOSDEM19 MySQL Component Infrastructure

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
MySQL 8.0
Component Infrastructure
Georgi “Joro” Kodinov
Team Lead, MySQL SrvGen Team
Why ? What's in it ? What's next ? How to use it ?

Georgi “Joro” Kodinov, MySQL @ Oracle
 Server General Team Lead
 Works on MySQL since 2006
 Specializes in:
 Security
 Client/server protocol
 Performance monitoring
 Component infrastructure
 Loves history, diverse world cultures, gardening
 A devoted Formula 1 fan (Go, Vettel !)

Agenda
3
 Why ?
 Architecture
 The inventory
 How to write my own …
 Tips & tricks
 What’s next ?
Homework !

Components: Why ?
• Too much technical debt accumulated in plugins
• Simpler and more extensible infrastructure
• Better code isolation and encapsulation
• Explicit dependencies
• All components are equal: can call and can be called
4

Component Infrastructure Architecture
5

Server Component
Server Binary
Server Functionality
Persisted Loader
File Scheme
Implementation
MySQL Server Modularization: The Big Picture
The Minimal Chassis
Implementation 1
Implementation 2
…
Registry
Component 1
Component 2
…
Dynamic Loader
Component 1
Component 2
Component 3
Implementation 1
Implementation 2
Plugins
Plugin
Services
Plugin
APIs

Components: Terminology
7
Component
Code
Service
Implementations
Contains
Services
Implement
Registry
Register
Service
References
Dynamic
Loader
Load
Unload
Persisted Dynamic
Loader
Encapsulate
INSTALL COMPONENT
UNINSTALL COMPONENT
UDFs
Performance Schema
System Variables, etc.

Basic Concepts: A Component
• A container for code.
– Can be an executable or a shared library
– As a compromise it can be a static library too, but that’s making the lines fuzzy, so NO
• Has init() and deinit() methods (called by the dyloader)
– Macros always add a registry reference
• Can consume services from the registry
– Should not access code in other components in any other way (linker) !
– Can have the dyloader fill these references in at load time
• Can provide service implementations to the registry
– Can have the dyloader register these at load time
8

Basic Concepts: A Service
• An ABSTRACT NAMED interface definition !
– Although it can’t be in the registry without at least one implementation
• Can have multiple implementations:
– One implementation is default at any time
• It’s STATELESS !
– If you want state (a.k.a. data instances) you need to provide factories for these and
use handles
• Is an IMMUTABLE definition
– I.e. if you need to add a method or change any definition of any parameter you need
a new name for the service.
9

Basic Concepts: A Service Contd.
• Has a public header file to describe the pointer returned
– The header must be self-sufficient (include all of the headers it depends on)
• It better (controversial, but explained later):
– use “simple” data types. A pointer to a 200+ member structure is a RED flag !
– use inout parameters and reserve return value for state.
10

Basic Concepts: A Service Implementation
• A NAMED implementation of an abstract service interface registered in the
registry
– The name is prefixed with the service name and delimited with a dot. E.g. foo.bar is
bar’s implementation of the foo service.
– The implementation name is usually the component name !
• Resides in a component
• Pointer to it is stored in and returned by the registry
– Reference counted !
• Usually a C structure with a bunch of function pointers
– Defined in the service header, registry is agnostic to that. Macros provided to do it
this way
11

Basic Concepts: A Related Service Implementation
• A normal service implementation
• Compatible with another service implementation
– Can operate on the same objects as the other service implementation
• Allows implementing sets of services
– And breaking large APIs in small logically independent sub-parts
• Usually residing in the same component as the other service
implementation.
• Example: registry query and registry registration services
12

Basic concepts: The Registry
• A set of services to manipulate a global map of services.
• Does reference counting on service implementations
• Allows one to:
– Register service implementations
– Query for implementations by name and get a counted reference
– Release a counted reference to a service implementation
– Enumerate the registered service implementations and their metadata (name/value)
– Search for related implementations to a service already acquired
• Usually the part that’s bootstrapped first as it depends on no services
13

Basic Concepts: The Dynamic Loader
• Requires the registry
• A single global ordered list of component generations.
• Loads and unloads component generations
– Handles registration/unregistration for the service implementations provided by
component(s)
• Can load multiple components in a single “generation” to handle circular
dependencies
• No persistent state
• Loading is done via calling “scheme” service implementations
– Currently only “file://”
14

Basic Concepts: The Persisted Dynamic Loader
• A separate implementation (usually in a different component)
• Requires the Dynamic loader
• Ensures persistency for the Dynamic loader:
– remembers the sequence of component generations loaded on a persistent storage
– re-loads the stored sequence on startup into the dynamic loader
• The current implementation inside the server component uses a system
table and provides SQL commands.
15

Concepts: The Minimal Chassis
• A static library
• The basic component infrastructure
– The registry
– The dynamic loader
• Enough to bootstrap a functional registry and a dynamic loader
• Can load any compliant component, given its service dependencies are met
by service implementations registered in the registry
• Great for (unit) testing !
• Allows using the component infrastructure outside of the server binary
16

Component Services Inventory
17

Component Services: The Inventory
• Service Registry
• Dynamic Loader
• Error logging
• System variables
• Status variables
• User defined functions
• Performance Schema
– New tables and Instrumentation
• Security Context
• Password validation
• Runtime errors
• Collated Strings
• Etc.
– 92 service related headers !
18

How To Write Your Own Components And
Services
19

How To Write Your Own Component/Service
• Use the utility macros
• Copy some of the existing components
– plenty of examples, specially in the tests
• Copy some of the existing services
– The services are done a bit differently in the server component
• Compile using a server source distribution
• Be careful !
– No crash protection, No security checks !
20
The Executive Summary

Relevant Source Tree Directories
21
Root
Components
Library_mysys
Mysql_server
C1
C2
Include MySQL Components
Services
Library_mysys
sql Server_component
Agenda:
 3d party Component Code
 Minimal Chassis
 Server Component Specific
 Implementation macros

Tutorial: How To Write A Component
#include <mysql/components/component_implementation.h>
#include <mysql/components/service_implementation.h>
#include <mysql/components/services/udf_registration.h>
REQUIRES_SERVICE_PLACEHOLDER(udf_registration);
// my code
static mysql_service_status_t deinit() { return false; }
static mysql_service_status_t init() { return false; }
22
Step 1: Add your component code: components/comp1/comp1.cc

BEGIN_COMPONENT_PROVIDES(comp1) END_COMPONENT_PROVIDES();
BEGIN_COMPONENT_REQUIRES(comp1)
REQUIRES_SERVICE(udf_registration),
END_COMPONENT_REQUIRES();
BEGIN_COMPONENT_METADATA(comp1)
METADATA("mysql.author", "Oracle Corporation"),
METADATA("mysql.license", "GPL"), END_COMPONENT_METADATA();
DECLARE_COMPONENT(comp1, “comp1")
init, deinit END_DECLARE_COMPONENT();
DECLARE_LIBRARY_COMPONENTS &COMPONENT_REF(comp1)
END_DECLARE_LIBRARY_COMPONENTS
23
Step 2: Add the component boilerplate: components/comp1/comp1.cc

MYSQL_ADD_COMPONENT(comp1
comp1.cc
TEST MODULE)
24
Step 3: Add a make file: components/comp1/CMakeFile.txt

Tutorial: How To Create a Service
#include <mysql/components/service.h>
/**
@ingroup group_components_services_inventory
*/
BEGIN_SERVICE_DEFINITION(host_application_signal)
/**
Send a signal.
*/
DECLARE_BOOL_METHOD(signal, (int signal_no, void *arg));
END_SERVICE_DEFINITION(host_application_signal)
25
Step 1: Add a Service definition in include/mysql/components/services/

#include <mysql/components/services/host_application_signal.h>
/**
An implementation of host application signal service for the mysql server as a host application.
*/
class mysql_component_host_application_signal_imp {
public:
static DEFINE_BOOL_METHOD(signal, (int signal_no, void *arg));
};
26
Step 2: OPTIONAL: Add an implementation header (server: sql/server_component/)

#include <sql/server_component/host_application_signal_imp.h>
#include <components/mysql_server/server_component.h>
/**
An implementation of host application signal service for the mysql server as a host application.
*/
class mysql_component_host_application_signal_imp {
public:
static DEFINE_BOOL_METHOD(signal, (int signal_no, void *arg));
};
27
Step 3: Add an implementation file (server: sql/server_component/, your component
dir)

#include <sql/server_component/host_application_signal_imp.h>
…
BEGIN_SERVICE_IMPLEMENTATION(comp1, host_application_signal)
mysql_component_host_application_signal_imp::signal
END_SERVICE_IMPLEMENTATION();
…
BEGIN_COMPONENT_PROVIDES(comp1)
PROVIDES_SERVICE(comp1, host_application_signal)
END_COMPONENT_PROVIDES();
28
Step 4: Add it to a component. (components/mysql_server/server_component.cc if
server)

Component Services Tips And Tricks
29

C++ Way
• Obj *inst;
• inst = new Obj(12);
• int ret;
• ret = Inst->m1(1);
• delete inst;
The Handle Way
• HOBJ hobj;
• obj_svc->create(&hobj);
– obj_svc->init(hobj, 12);
• int ret;
• obj_svc->m1(hobj, 1, &ret);
• obj_svc->dispose(hobj);
30
Working with State: Handles

Single Service
• HOBJ hobj;
• obj_svc->create(&hobj);
– obj_svc->init(hobj, 12);
• int ret;
• obj_svc->dispose(hobj);
Related Services
• HOBJ hobj;
• obj_factory_svc->create(&hobj);
– obj_integer_init_svc->init(hobj, 12);
• int ret;
• obj_factory_svc->dispose(hobj);
31
Working with State: Related Services

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 32
Related Service Sets: Why ?
C1
obj_svc.C1 obj_factory_svc.C1
obj_svc.C1 obj_svc.C1
obj_svc obj_svc.C1
obj_factory_svc.C1 obj_factory_svc.C1
obj_factory_svc obj_factory_svc.C2
obj_svc.C2 obj_svc.C2
obj_factory_svc.C2 obj_factory_svc.C2
C2

Random Guidelines
• Try to use the component infrastructure macros
• Use basic parameter types. Anything more complex goes into an
ANONYMOUS(!) handle and has service(s) to drive it
• Taking references is “expensive” (global structure). Reuse references.
• Do not make high volume calls services. Price to call a function pointer may
be too much !
• Make sure to avoid reference or handle leaks !
• Always use the designated destructor for handles !
• Keep the service reference with the consumer.
34

DEFINE_SERVICE_HANDLE(HOBJ);
BEGIN_SERVICE_DEFINITION(obj_svc)
DECLARE_BOOL_METHOD(create, (HOBJ *outHandle));
DECLARE_BOOL_METHOD(dispose, (HOBJ outHandle));
DECLARE_BOOL_METHOD(init, (HOBJ handle, int arg));
DECLARE_BOOL_METHOD(m1, (HOBJ handle, int arg, int
*outRet));
END_SERVICE_DEFINITION(obj_svc)
DEFINE_SERVICE_HANDLE(HOBJ);
BEGIN_SERVICE_DEFINITION(obj_factory_svc)
DECLARE_BOOL_METHOD(create, (HOBJ *outHandle));
DECLARE_BOOL_METHOD(dispose, (HOBJ outHandle));
END_SERVICE_DEFINITION(obj_factory_svc)
BEGIN_SERVICE_DEFINITION(obj_integer_init_svc)
DECLARE_BOOL_METHOD(init, (HOBJ handle, int arg));
END_SERVICE_DEFINITION(obj_integer_init_svc)
BEGIN_SERVICE_DEFINITION(obj_svc)
DECLARE_BOOL_METHOD(m1, (HOBJ handle, int arg, int
*outRet));
END_SERVICE_DEFINITION(obj_svc)
35
Working with State: Related Services Definitions

my_service<SERVICE_TYPE(obj_factory_svc)>
obj_factory_svc(“obj_factory_svc”, registry);
my_service<SERVICE_TYPE(obj_integer_init_svc)>
obj_int_init_svc(“obj_integer_init_svc”, registry);
36
Working with State: Related Services Use

obj_svc(“obj_svc.c1”, registry);
obj_svc(“obj_svc.c1”, registry);
obj_factory_svc(“obj_factory_svc.c1”, registry);
obj_int_init_svc(“obj_integer_init_svc.c1”, registry);
37
Working with State: 2nd Try

obj_factory_svc(“obj_factory_svc”, obj_svc, registry);
obj_int_init_svc(“obj_integer_init_svc”, obj_svc, registry);
38
Working with State: Final !

What’s Next ?
39

What’s Ahead ?
• Migration of major plugin APIs and plugin services into components
– Current State
• No new plugin APIs
• No new plugin service APIs
– Goal
• Deprecate the plugins !
• Further sub-division of the (now extremely chunky) server component
40

Safe Harbor Statement
The preceding is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.
41

FOSDEM19 MySQL Component Infrastructure

FOSDEM19 MySQL Component Infrastructure

More Related Content

What's hot (20)

Similar to FOSDEM19 MySQL Component Infrastructure (20)

More from Georgi Kodinov (20)

Recently uploaded (20)

FOSDEM19 MySQL Component Infrastructure