MySQL document_store
4 likes1,334 views
This document discusses using MySQL as a document store by introducing the MySQL X-Protocol and associated tools. Key points include: - The MySQL X-Plugin and MySQL Shell allow MySQL to function as a document store, uniting SQL and NoSQL approaches. - The X-Protocol extends the traditional client/server protocol to enable asynchronous communication and a universal API for different languages. - While the document store feature is available in MySQL 5.7.12+, the associated tools are still in early development stages and not production ready. - Examples demonstrate using the MySQL Shell to work with document collections, perform CRUD operations, and convert data between MySQL, MongoDB, and JSON formats in a schema-less manner
![It should be easy to switch
Universal API
MySQL Shell JavaScript Code
// Create a new collection
var myColl = db.createCollection('my_collection');
// Insert a document
myColl.add( { name: 'Sakila', age: 15 } ).execute();
// Insert several documents at once
myColl.add( [
{ name: 'Susanne', age: 24 },
{ name: 'Mike', age: 39 } ] ).execute();
10](https://image.slidesharecdn.com/mysqldocumentstore-170426214752/85/MySQL-document_store-10-320.jpg)
![It looks really easy to switch!
Universal API
MySQL Shell Python Code
# Create a new collection
myColl = db.createCollection('my_collection')
# Insert a document
myColl.add( { 'name': 'Sakila', 'age':15 } ).execute()
# Insert several documents at once
myColl.add( [
{ 'name': 'Susanne', 'age': 24 },
{ 'name': 'Mike', 'age' : 39 } ] ).execute()
11](https://image.slidesharecdn.com/mysqldocumentstore-170426214752/85/MySQL-document_store-11-320.jpg)
![When we start the server
Method 2 : at startup
mysqld [...] --plugin-load=mysqlx=mysqlx.so
--mysqlx-port=15000
# or in the configuration file
[mysqld]
# ...
plugin-load=mysqlx=mysqlx.so
mysqlx-port=15000
18](https://image.slidesharecdn.com/mysqldocumentstore-170426214752/85/MySQL-document_store-18-320.jpg)
![Some things have two faces
As seen from the old client
~/sandboxes/msb_5_7_17/use world_x
mysql [localhost] {msandbox} (world_x) > show tables;
+-------------------+
| Tables_in_world_x |
+-------------------+
| City |
| Country |
| CountryInfo |
| CountryLanguage |
+-------------------+
4 rows in set (0.00 sec)
27](https://image.slidesharecdn.com/mysqldocumentstore-170426214752/85/MySQL-document_store-27-320.jpg)
![Welcome to the machine!
And from the new client (1)
$ mysqlsh
--uri msandbox:msandbox@127.0.0.1:33060/world_x
Creating a Session to 'msandbox@127.0.0.1:33060/
world_x'
Node Session successfully established. Default schema
`mysql_x` accessible through db.
Welcome to MySQL Shell 1.0.9
[...]
Type 'help', 'h' or '?' for help, type 'quit' or
'q' to exit.
Currently in JavaScript mode. Use sql to switch to
SQL mode and execute queries.
mysql-js>
28](https://image.slidesharecdn.com/mysqldocumentstore-170426214752/85/MySQL-document_store-28-320.jpg)
![This reminds me of something ...
Retrieving data
mysql-js> nc.find()
[
{
"_id": "6eee6f07ab66e611564dfeeead98f1ef",
"name": "Frank",
"where_are_you_from": "London"
},
{
"_id": "94b470f7aa66e611564dfeeead98f1ef",
"city": "Paris",
"name": "Joe"
}
]
2 documents in set (0.00 sec)
32](https://image.slidesharecdn.com/mysqldocumentstore-170426214752/85/MySQL-document_store-32-320.jpg)
![The data is in layers
Sample data from world_x
mysql-js> ci.find().limit(1)
[
{
"GNP": 828,
"IndepYear": null,
"Name": "Aruba",
"_id": "ABW",
"demographics": {
"LifeExpectancy": 78.4000015258789,
"Population": 103000
},
"geography": {
"Continent": "North America",
"Region": "Caribbean",
"SurfaceArea": 193
},
"government": {
"GovernmentForm": "Nonmetropolitan Territory of The Netherlands",
"HeadOfState": "Beatrix"
}
}
]
38](https://image.slidesharecdn.com/mysqldocumentstore-170426214752/85/MySQL-document_store-38-320.jpg)
![Export the data from MongoDB
From MongoDB to MySQL (2)
docker exec -ti mongo mongo --quiet
--eval 'DBQuery.shellBatchSize=300; var
all=db.restaurants.find() ; all'
| perl -pe 's/(?:ObjectId|ISODate)(("[^"]+"))/
$1/g'
> all_recs.json
44](https://image.slidesharecdn.com/mysqldocumentstore-170426214752/85/MySQL-document_store-44-320.jpg)
![There is data like this:
Why do we need to filter
{
"_id" : ObjectId("57b81d385957bb0d60511ce5"),
"borough" : "Bronx",
"cuisine" : "Bakery",
"grades" : [
{
"date" : ISODate("2014-03-03T00:00:00Z"),
"grade" : "A",
"score" : 2
},
],
"name" : "Morris Park Bake Shop",
"restaurant_id" : "30075445"
}
45](https://image.slidesharecdn.com/mysqldocumentstore-170426214752/85/MySQL-document_store-45-320.jpg)
![The "truth"
general log
SELECT C.table_name AS name,
IF(ANY_VALUE(T.table_type)='VIEW', 'VIEW', IF(COUNT(*)
= COUNT(CASE WHEN (column_name = 'doc' AND data_type =
'json') THEN 1 ELSE NULL END) + COUNT(CASE WHEN
(column_name = '_id' AND generation_expression =
'json_unquote(json_extract(`doc`,''$._id''))') THEN 1
ELSE NULL END) + COUNT(CASE WHEN (column_name != '_id'
AND generation_expression RLIKE '^(json_unquote[[.
(.]])?json_extract[[.(.]]`doc`,''[[.$.]]([[...]]
[^[:space:][...]]+)+''[[.).]]{1,2}$') THEN 1 ELSE NULL
END), 'COLLECTION', 'TABLE')) AS type FROM
information_schema.columns AS C LEFT JOIN
information_schema.tables AS T USING (table_name)WHERE
C.table_schema = 'world_x' GROUP BY C.table_name ORDER
BY C.table_name
51](https://image.slidesharecdn.com/mysqldocumentstore-170426214752/85/MySQL-document_store-51-320.jpg)
![I'd say it needs more integration
how x-plugin finds "collections"
SELECT C.table_name AS name, IF(ANY_VALUE(T.table_type)='VIEW', 'VIEW',
IF(COUNT(*) = COUNT(CASE WHEN (column_name = 'doc' AND data_type = 'json') THEN 1
ELSE NULL END) + COUNT(CASE WHEN (column_name = '_id' AND generation_expression =
'json_unquote(json_extract(`doc`,''$._id''))') THEN 1 ELSE NULL END) + COUNT(CASE
WHEN (column_name != '_id' AND generation_expression RLIKE '^(json_unquote[[.
(.]])?json_extract[[.(.]]`doc`,''[[.$.]]([[...]][^[:space:][...]]+)+''[[.).]]
{1,2}$') THEN 1 ELSE NULL END), 'COLLECTION', 'TABLE')) AS type FROM
information_schema.columns AS C LEFT JOIN information_schema.tables AS T USING
(table_name)WHERE C.table_schema = 'world_x' GROUP BY C.table_name ORDER BY
C.table_name
+-----------------+------------+
| name | type |
+-----------------+------------+
| City | TABLE |
| Country | TABLE |
| CountryInfo | COLLECTION |
| CountryLanguage | TABLE |
+-----------------+------------+
52](https://image.slidesharecdn.com/mysqldocumentstore-170426214752/85/MySQL-document_store-52-320.jpg)
MySQL document_store
- 1. MySQL Document store: SQL and NoSQL united Giuseppe Maxia Vmware, Inc 1 The battle is over. SQL and noSQL can live together.
- 2. Who's this guy? About me ‣ Giuseppe Maxia, a.k.a. "The Data Charmer" ‣ QA Architect at VMware ‣ 25+ years development and DB experience ‣ Long timer MySQL community member. ‣ Oracle ACE Director ‣ Blog: http://datacharmer.blogspot.com ‣ Twitter: @datacharmer 2 A
- 3. ‣ Document store in a nutshell ‣ X-Protocol overview ‣ X-Plugin installation ‣ MySQL shell installation ‣ Getting started ‣ Example: with the shell ‣ Example: data to and from MongoDB ‣ A look inside Agenda 3
- 4. Better be clear about this Disclaimer ‣ This is community work. ‣ Non affiliation: • I don't work for Oracle. All I say here, good or bad, is my opinion. ‣ Not talking for my company: • All I say is my own stuff. My company does not influence or censor what I say here. 4
- 5. This technology does not work with every version Requirements ‣ MySQL 5.7.12 or later (contains the X-Plugin) ‣ MySQL shell (separate product) 5
- 6. This is not intuitive How DBAs see data 6 person_id name dept_id Person dept_id name Dept empl_ID name Title person_id dept_id person_dept person_id title_id start_date end_date person_title person_id salary start_date end_date salary
- 7. This is driven by most programming languages How developers see data 7
- 8. it's a clash of data structures DBAs vs. developers 8 xxxx: { xxxxxxxx aaaaaa, yyyyy: { yyyyyyyyy, yyyyyyyyy, yyyyyyyyy }, z: [ { zzzzz, zzzzz, zzzz}, { zzzzz, zzzzz, zzzz}, { zzzzz, zzzzz, zzzz} [ } xxxx xxxx xxxx xxxx aaaa aaaa aaaa yyyy yyyy yyyy aaaa xxxx xxxx_aaaa xxxx yyyy yyyy xxxx_yyyy xxx zzz zzz zzzz
- 9. A new protocol to talk to MySQL ‣ extends and replaces the traditional client/ server protocol ‣ allows asynchronous communication to the server ‣ uses different API calls • Javascript • Python • C# • Java X- Protocol overview 9
- 10. It should be easy to switch Universal API MySQL Shell JavaScript Code // Create a new collection var myColl = db.createCollection('my_collection'); // Insert a document myColl.add( { name: 'Sakila', age: 15 } ).execute(); // Insert several documents at once myColl.add( [ { name: 'Susanne', age: 24 }, { name: 'Mike', age: 39 } ] ).execute(); 10
- 11. It looks really easy to switch! Universal API MySQL Shell Python Code # Create a new collection myColl = db.createCollection('my_collection') # Insert a document myColl.add( { 'name': 'Sakila', 'age':15 } ).execute() # Insert several documents at once myColl.add( [ { 'name': 'Susanne', 'age': 24 }, { 'name': 'Mike', 'age' : 39 } ] ).execute() 11
- 12. The document store is not in the server by default ‣ MySQL server does not include the X-protocol ‣ You need to install a plugin for this ‣ and you need the MySQL shell (separate product) to use it 12
- 13. The server is GA , but ... !! WARNING !! ‣ The document store comes with MySQL 5.7.12+ ‣ HOWEVER • The tools ARE NOT GA quality • They are, actually, pretty much alpha software ‣ Be careful when using it in production 13
- 14. From the manual "Using MySQL as a document store is currently a preproduction feature" https://dev.mysql.com/doc/refman/5.7/en/ document-store.html 14
- 15. There are several components to the document store Readiness (as of June 2017) 15 MySQL SERVER 5.7.12+ PLUGIN SHELL 1.0.9 CONNECTORS GA GA GA Alpha node.js java javascript python .net c++
- 16. The plugin comes with the server, but you need to enable it X-Plugin installation ‣ Three methods: • with mysqlsh • at startup, using --plugin-load=mysqlx=mysqlx.so • in SQL, using INSTALL PLUGIN 16
- 17. Using the mysql shell itself Method 1 : with mysqlsh mysqlsh --classic --user=msandbox --password=msandbox --port=3306 --host=127.0.0.1 --dba enableXProtocol 17
- 18. When we start the server Method 2 : at startup mysqld [...] --plugin-load=mysqlx=mysqlx.so --mysqlx-port=15000 # or in the configuration file [mysqld] # ... plugin-load=mysqlx=mysqlx.so mysqlx-port=15000 18
- 19. At any moment Method 3 : in SQL install plugin mysqlx soname 'mysqlx.so'; 19
- 20. Gotchas ‣ X-Plugin listens to port 33060 ‣ When you install with method 1, you use port 3306 ‣ Afterwards, you use port 33060 20
- 21. You need the new client to use the new features MySQL Shell installation 21
- 22. There are packages for most operating systems MySQL Shell installation 22
- 23. Instead of installing ... Shell with Docker ‣ Using a Docker image ‣ Shell ready to use ‣ No side effects 23 MySQL shell image not updated since ~1 year ago
- 24. Using the latest mysqlsh release, there is a workaround However ... docker run -d -v $PWD/mysql-shell/bin/mysqlsh:/usr/bin/mysqlsh -v ~/data:/data --name mybox -e MYSQL_ROOT_PASSWORD=secret mysql/mysql-server docker run -ti mybox bash 24
- 25. Let's practice with real data ‣ Install MySQL 5.7.18 ‣ load plugin ‣ Download the world_x database • https://dev.mysql.com/doc/index-other.html ‣ load the database ‣ connect using mysql shell Getting started 25
- 26. Getting ready Examples with the shell 26
- 27. Some things have two faces As seen from the old client ~/sandboxes/msb_5_7_17/use world_x mysql [localhost] {msandbox} (world_x) > show tables; +-------------------+ | Tables_in_world_x | +-------------------+ | City | | Country | | CountryInfo | | CountryLanguage | +-------------------+ 4 rows in set (0.00 sec) 27
- 28. Welcome to the machine! And from the new client (1) $ mysqlsh --uri msandbox:[email protected]:33060/world_x Creating a Session to '[email protected]:33060/ world_x' Node Session successfully established. Default schema `mysql_x` accessible through db. Welcome to MySQL Shell 1.0.9 [...] Type 'help', 'h' or '?' for help, type 'quit' or 'q' to exit. Currently in JavaScript mode. Use sql to switch to SQL mode and execute queries. mysql-js> 28
- 29. Welcome to the machine! And from the new client (2) mysql-js> db.getTables() { "City": <Table:City>, "Country": <Table:Country>, "CountryLanguage": <Table:CountryLanguage> } mysql-js> db.getCollections() { "CountryInfo": <Collection:CountryInfo> } mysql-js> 29
- 30. Schema-less! Starting something new mysql-js> nc=db.createCollection('person') <Collection:person> mysql-js> mysql-js> db.getCollections() { "CountryInfo": <Collection:CountryInfo>, "person": <Collection:person> } mysql-js> 30
- 31. REALLY schema-less! Inserting data mysql-js> nc.add({ name: "Joe", city: "Paris"}) Query OK, 1 item affected (0.00 sec) mysql-js> nc.add({ name: "Frank", where_are_you_from: "London"}) Query OK, 1 item affected (0.01 sec) 31
- 32. This reminds me of something ... Retrieving data mysql-js> nc.find() [ { "_id": "6eee6f07ab66e611564dfeeead98f1ef", "name": "Frank", "where_are_you_from": "London" }, { "_id": "94b470f7aa66e611564dfeeead98f1ef", "city": "Paris", "name": "Joe" } ] 2 documents in set (0.00 sec) 32
- 33. The general log shows what we were doing Back to the old side 33 CREATE TABLE `world_x`.`person` (doc JSON,_id VARCHAR(32) GENERATED ALWAYS AS (JSON_UNQUOTE(JSON_EXTRACT(doc, '$._id'))) STORED PRIMARY KEY) CHARSET utf8mb4 ENGINE=InnoDB Query INSERT INTO `world_x`.`person` (doc) VALUES ('{"_id":"94b470f7aa66e611564dfeeead98f1ef","city ":"Paris","name":"Joe"}') Query INSERT INTO `world_x`.`person` (doc) VALUES ('{"_id":"6eee6f07ab66e611564dfeeead98f1ef","name ":"Frank","where_are_you_from":"London"}')
- 34. A classical organization with sales A complex example 34 person sales categories departments locations countries customersclass
- 35. The classical organization as a SQL result SQL to JSON (1) select name, sale_date, sale_amount as '$$$', customer, class_name from person inner join sales using (person_id) inner join customers using (customer_id) inner join class c on (c.class_id = sales.class_id); +--------+------------+-------+-------------+------------+ | name | sale_date | $$$ | customer | class_name | +--------+------------+-------+-------------+------------+ | Frank | 2003-10-01 | 23000 | DataSmart | software | | Frank | 2003-10-12 | 45000 | NewHardware | hardware | | Frank | 2003-11-04 | 60000 | DataSmart | services | | Susan | 2003-11-02 | 25000 | ViewData | software | | Susan | 2003-10-13 | 18000 | SmartEdu | services | | Martin | 2003-10-12 | 50000 | SmartEdu | hardware | +--------+------------+-------+-------------+------------+ 35
- 36. The classical organization as a collection of documents SQL to JSON (2) 36
- 37. The world_x database comes with some beefy data A bigger collection mysql-js> db.getCollections() { "CountryInfo": <Collection:CountryInfo>, "person": <Collection:person> } mysql-js> ci=db.getCollection('CountryInfo') <Collection:CountryInfo> 37
- 38. The data is in layers Sample data from world_x mysql-js> ci.find().limit(1) [ { "GNP": 828, "IndepYear": null, "Name": "Aruba", "_id": "ABW", "demographics": { "LifeExpectancy": 78.4000015258789, "Population": 103000 }, "geography": { "Continent": "North America", "Region": "Caribbean", "SurfaceArea": 193 }, "government": { "GovernmentForm": "Nonmetropolitan Territory of The Netherlands", "HeadOfState": "Beatrix" } } ] 38
- 39. Not always easy to get Complex queries are possible 39
- 40. Since they are both schema-less ... ‣ From MySQL to MongoDB • extract data from a document store • feed it to MongoDB ‣ From MongoDB to MySQL • create collection • extract data • filter off the oddities • feed it to MySQL shell Examples: to and from MongoDB 40
- 41. Extracting data From MySQL to Mongoldb (1) // extract.py import mysqlx import json session = mysqlx.get_session( ... ) schema = session.get_schema('world_x') collection = schema.get_collection('CountryInfo') result = collection.find().execute() docs = result.fetch_all() for doc in docs: doc = dict(doc) print(json.dumps(doc, indent=4)) session.close() # python extract.py > /data/country_info.json 41
- 42. import data to mongoldb From MySQL to Mongoldb (2) mongoimport --db test --collection countries --drop --file /data/country_info.json 42
- 43. First create the collection From MongoDB to MySQL (1) mysql-js> db.createCollection('restaurants') 43
- 44. Export the data from MongoDB From MongoDB to MySQL (2) docker exec -ti mongo mongo --quiet --eval 'DBQuery.shellBatchSize=300; var all=db.restaurants.find() ; all' | perl -pe 's/(?:ObjectId|ISODate)(("[^"]+"))/ $1/g' > all_recs.json 44
- 45. There is data like this: Why do we need to filter { "_id" : ObjectId("57b81d385957bb0d60511ce5"), "borough" : "Bronx", "cuisine" : "Bakery", "grades" : [ { "date" : ISODate("2014-03-03T00:00:00Z"), "grade" : "A", "score" : 2 }, ], "name" : "Morris Park Bake Shop", "restaurant_id" : "30075445" } 45
- 46. Importing into MySQL schema = session.get_schema('test') collection = schema.create_collection('restaurants') with open('all_recs.json', 'r') as json_data: for line in json_data: skip=re.match('Type', line) if not skip: rec = json.loads(line) collection.add(rec).execute() 46
- 47. What's a MySQL Document? A look inside ‣ mysqlsh calls it a "collection" ‣ mysql calls it a table • with a GENERATED _id field • with a json field 47
- 48. The old client view mysql show tables; +-------------------+ | Tables_in_world_x | +-------------------+ | City | | Country | | CountryInfo | | CountryLanguage | +-------------------+ 4 rows in set (0.00 sec) 48
- 49. The document store view mysqlsh mysql-js> db.getCollections() { "CountryInfo": <Collection:CountryInfo> } mysql-js> db.getTables() { "City": <Table:City>, "Country": <Table:Country>, "CountryLanguage": <Table:CountryLanguage> } 49
- 50. The old client view mysql show create table CountryInfoG ******* 1. row *************************** Table: CountryInfo Create Table: CREATE TABLE `CountryInfo` ( `doc` json DEFAULT NULL, `_id` varchar(32) GENERATED ALWAYS AS (json_unquote(json_extract(`doc`,'$._id'))) STORED ) ENGINE=InnoDB DEFAULT CHARSET=utf8 1 row in set (0.00 sec) 50
- 51. The "truth" general log SELECT C.table_name AS name, IF(ANY_VALUE(T.table_type)='VIEW', 'VIEW', IF(COUNT(*) = COUNT(CASE WHEN (column_name = 'doc' AND data_type = 'json') THEN 1 ELSE NULL END) + COUNT(CASE WHEN (column_name = '_id' AND generation_expression = 'json_unquote(json_extract(`doc`,''$._id''))') THEN 1 ELSE NULL END) + COUNT(CASE WHEN (column_name != '_id' AND generation_expression RLIKE '^(json_unquote[[. (.]])?json_extract[[.(.]]`doc`,''[[.$.]]([[...]] [^[:space:][...]]+)+''[[.).]]{1,2}$') THEN 1 ELSE NULL END), 'COLLECTION', 'TABLE')) AS type FROM information_schema.columns AS C LEFT JOIN information_schema.tables AS T USING (table_name)WHERE C.table_schema = 'world_x' GROUP BY C.table_name ORDER BY C.table_name 51
- 52. I'd say it needs more integration how x-plugin finds "collections" SELECT C.table_name AS name, IF(ANY_VALUE(T.table_type)='VIEW', 'VIEW', IF(COUNT(*) = COUNT(CASE WHEN (column_name = 'doc' AND data_type = 'json') THEN 1 ELSE NULL END) + COUNT(CASE WHEN (column_name = '_id' AND generation_expression = 'json_unquote(json_extract(`doc`,''$._id''))') THEN 1 ELSE NULL END) + COUNT(CASE WHEN (column_name != '_id' AND generation_expression RLIKE '^(json_unquote[[. (.]])?json_extract[[.(.]]`doc`,''[[.$.]]([[...]][^[:space:][...]]+)+''[[.).]] {1,2}$') THEN 1 ELSE NULL END), 'COLLECTION', 'TABLE')) AS type FROM information_schema.columns AS C LEFT JOIN information_schema.tables AS T USING (table_name)WHERE C.table_schema = 'world_x' GROUP BY C.table_name ORDER BY C.table_name +-----------------+------------+ | name | type | +-----------------+------------+ | City | TABLE | | Country | TABLE | | CountryInfo | COLLECTION | | CountryLanguage | TABLE | +-----------------+------------+ 52
- 53. A few tricks that could be useful More with mysql shell ‣ Using mysqlsh to export in JSON format from regular MySQL tables ‣ Running mysqlsh in Docker without a dedicated container. 53
- 54. where to find the examples used in this presentation Examples https://github.com/datacharmer/mysql-document-store 54
- 55. Q & A 55