MySQL to Neo4j: A DBA Perspective - David Stern @ GraphConnect NY 2013
9 likes•7,421 views
This document discusses the perspective of a DBA on using MySQL and Neo4j, comparing their configurations, partitioning methods, and performance optimization techniques. It highlights differences in schema enforcement, replication strategies, and authentication mechanisms between the two databases. Additionally, it emphasizes the tuning and configuration aspects essential for efficient use of MySQL and Neo4j.
![(MySQL)-[:to]->(neo4j)
A DBA Perspective
Dave Stern
@davestern1](https://image.slidesharecdn.com/mysql-to-neo4jadbaperspective-131108194500-phpapp01/85/MySQL-to-Neo4j-A-DBA-Perspective-David-Stern-GraphConnect-NY-2013-1-320.jpg)
![Built-in Tools
Data Browser
Backup Script
neo4j
$/p/e4/i/e4-akp-rmsnl:/06.8.7:32
otnojbnnojbcu fo ige/1.6121766
>-o/ei/e4-akppouto/031-20:01Z
t mdanojbcu/rdcin21-10T54:0
Promn fl bcu fo 'ige/1.6121766'
efrig ul akp rm snl:/06.8.7:32
...............
...............
.......
.......
[4Flscpe]
4 ie oid
Fl cnitnycek
ul ossec hc
.......... 1%
.......... 0
.......... 2%
.......... 0
.......... 3%
.......... 0
.......... 4%
.......... 0
.......... 5%
.......... 0
.......... 6%
.......... 0
.......... 7%
.......... 0
.......... 8%
.......... 0
.......... 9%
.......... 0
.......... 10
.......... 0%
Dn
oe](https://image.slidesharecdn.com/mysql-to-neo4jadbaperspective-131108194500-phpapp01/85/MySQL-to-Neo4j-A-DBA-Perspective-David-Stern-GraphConnect-NY-2013-24-320.jpg)
![Use
Querying via REST
PS ht:/oahs:44d/aacpe
OT tp/lclot77/bdt/yhr
Acp:apiainjo;castUF8
cet plcto/sn hre=TCnetTp:apiainjo
otn-ye plcto/sn
{
}
"ur":"tr x =nd:oeat_ne(ae{trNm}
qey
sat
oend_uoidxnm=satae)
mthpt =(-r-red
ac ah
x[]fin)
weefin.ae={ae rtr TP(),
hr rednm
nm} eun YEr"
"aas :{
prm"
"trNm":""
satae
I,
"ae :"o"
nm"
yu
}
Example response:
20 O
0: K
CnetTp:apiainjo;castUF8
otn-ye plcto/sn hre=T{
}
"oun":["YEr"]
clms
TP() ,
"aa :[["nw ]]
dt"
ko"](https://image.slidesharecdn.com/mysql-to-neo4jadbaperspective-131108194500-phpapp01/85/MySQL-to-Neo4j-A-DBA-Perspective-David-Stern-GraphConnect-NY-2013-53-320.jpg)
MySQL to Neo4j: A DBA Perspective - David Stern @ GraphConnect NY 2013
- 3. Dev Ops @ FiftyThree MySQL user & admin since 1998 Multiple tiers of masters & slaves Bare metal & AWS - EC2/RDS MySQL & Percona neo4j user & admin since 2012 neo4j 1.8, 1.9 AWS: Multiple 3-instance enterprise clusters
- 10. Comparison
- 11. Logical Partitioning MySQL Strictly enforced schema neo4j No logical databases No tables ...no schema ...no joins 2.0: schema-optional http://www.mysql.com/products/workbench/
- 12. Physical Partitioning & Sharding Improves write performance, usually disk I/O MySQL neo4j ind_ieprtbe nobfl_e_al No logical partitioning by DB or table Databases on separate partitions or devices Highly connected data: no clear separation Shard horizontally (e.g. by time range) Logs can be on separate partitions for I/O Shard vertically (e.g. by table or function) Logs can be on separate partitions for I/O gain gain
- 13. SCALE UP!
- 14. Authentication & Authorization MySQL msl slc Hs,d,ue,slc_rv isr_rv udt_rv dlt_rvfo d; yq> eet ot b sr eetpi, netpi, paepi, eeepi rm b +-----+----+-----+------+------+------+------+ --------------------------------------------|Hs ot |d b |ue sr |slc_rv|isr_rv|udt_rv|dlt_rv| eetpi netpi paepi eeepi +-----+----+-----+------+------+------+------+ --------------------------------------------|% |ts et | |Y |Y |Y |Y | |% |ts%| et_ |Y |Y |Y |Y | |lclot|Odr |amn oahs res di |Y |Y |Y |Y | |lclot|Eet |amn oahs vns di |Y |Y |Y |Y | |lclot|Eet |eet oahs vns vns |Y |Y |Y |N | |1. 0% |Eet |eet vns vns |Y |N |N |N | +-----+----+-----+------+------+------+------+ ---------------------------------------------
- 15. Authentication & Authorization neo4j No permissions No users How do you secure the DB? 1. Protect the database in a Private Network or VPC 2. Firewall: router, AWS Security Groups, iptables 3. Proxy requests via web server or Load Balancer If you must allow access, use HTTPS & authenticate at the proxy.
- 19. Replication SO SAE TP LV; STGOA slsaesi_one =1 E LBL q_lv_kpcutr ; SATSAE TR LV;
- 20. Replication vs. HA MySQL Free Slaves pull updates Eventual consistency One-way, asynchronous neo4j Enterprise edition: can cost $ depending on use Slaves can pull asynchronous updates Eventual consistency, optimistic pushes to slaves are the default Writes to any cluster member
- 21. JVM Buffers & Memory management =~ JVM settings The database itself is extendable via Java ... if you're into that sort of thing
- 24. Built-in Tools Data Browser Backup Script neo4j $/p/e4/i/e4-akp-rmsnl:/06.8.7:32 otnojbnnojbcu fo ige/1.6121766 >-o/ei/e4-akppouto/031-20:01Z t mdanojbcu/rdcin21-10T54:0 Promn fl bcu fo 'ige/1.6121766' efrig ul akp rm snl:/06.8.7:32 ............... ............... ....... ....... [4Flscpe] 4 ie oid Fl cnitnycek ul ossec hc .......... 1% .......... 0 .......... 2% .......... 0 .......... 3% .......... 0 .......... 4% .......... 0 .......... 5% .......... 0 .......... 6% .......... 0 .......... 7% .......... 0 .......... 8% .......... 0 .......... 9% .......... 0 .......... 10 .......... 0% Dn oe
- 25. Built-in Tools Data Browser Backup Script MySQL $inbcue -ue=BSR-pswr=BSRAS/aht/AKPDR noakpx -srDUE -asodDUEPS pt/oBCU-I/ inbcue:Bcu cetdi drcoy'pt/oBCU-I/030-50-00' noakpx akp rae n ietr /aht/AKPDR21-32_00-9 inbcue:MSLbno psto:flnm 'yq-i.003, noakpx yQ ilg oiin ieae mslbn000' psto 14112 0:05 oiin 96125 00:3 inbcue:cmltdO! noakpx opee K
- 28. Configuration MySQL So many options... msl SO VRALS yq> HW AIBE; +--------------------+-------------+ ----------------------------------|Vral_ae aibenm |Vle au | +--------------------+-------------+ ----------------------------------|at_nrmn_nrmn uoiceeticeet |1 | |at_nrmn_fst uoiceetofe |1 | |atcmi uoomt |O N | |atmtcs_rvlgs uoai_ppiiee |O N | |bc_o aklg |5 0 | |bsdr aei |/oemslbnmsl55| hm/yq/i/yq-. |bgtbe i_als |OF F | |bno_ah_ie ilgccesz |378 26 | |bno_ietnntascinludts|OF ilgdrc_o_rnatoa_pae F | |bno_omt ilgfra |SAEET TTMN | |bno_ttccesz ilgsm_ah_ie |378 26 | |bl_netbfe_ie ukisr_ufrsz |8868 380 | .. . |mxalwdpce a_loe_akt |1456 087 | |mxbno_ah_ie a_ilgccesz |1464030572 8474779450 | |mxbno_ie a_ilgsz |17712 03484 | |mxbno_ttccesz a_ilgsm_ah_ie |1464030572 8474779450 | |mxcneterr a_onc_ros |1 0 | |mxcnetos a_oncin |11 5 | |mxdlydtras a_eae_hed |2 0 | |mxerrcut a_ro_on |6 4 | |mxha_al_ie a_eptbesz |1771 6726 | |mxisr_eae_hed a_netdlydtras |2 0 | |mxji_ie a_onsz |1464030511 8474779565 | .. .
- 29. MySQL Configuration Buffers, Caching & I/O You can optimize dozens of settings like these... ind_ufrpo_ie=1G nobbfe_olsz 2 ind_ufrpo_ntne =8 nobbfe_olisacs ind_diinlmmpo_ie=26 nobadtoa_e_olsz 5M ind_ls_o_ttxcmi =2 nobfuhlga_r_omt ind_ls_ehd=ODRC nobfuhmto _IET ind_o_iesz =18 noblgfl_ie 2M ind_o_ufrsz =6M noblgbfe_ie 4 ind_ieprtbe nobfl_e_al ind_ocpct nobi_aaiy =50 0 ind_edi_hed =6 nobra_otras 4 ind_rt_otras=6 nobwiei_hed 4
- 32. MySQL Configuration Other And these, depending on version & hardware... sr_ufrsz otbfe_ie tptbesz m_al_ie =2 M =3M 2 ji_ufrsz onbfe_ie =18 2k qeyccetp ur_ah_ye qeyccesz ur_ah_ie =1 =6M 4 oe_ie_ii pnflslmt =89 12 .. ..
- 33. neo4j Configuration Tuning Simple Questions How many nodes do you expect? How many relationships do you expect? Average number of properties per node and relationship? Optional: How do you expect to traverse the graph? Long paths and/or large result sets? Short paths and/or small results sets? 3 things to calculate: File Cache Mapped Memory & Object Caches Heap Size RAM for OS
- 34. neo4j Configuration Store file Record size Contents neostore.nodestore.db 9B Nodes neostore.relationshipstore.db 3 B 3 Relationships neostore.propertystore.db 4 B 1 Properties for nodes and relationships neostore.propertystore.db.strings 1 8 B 2 Values of string properties neostore.propertystore.db.arrays 1 8 B 2 Values of array properties Capacity Planning Estimates: Node size (9B) x expected nodes (14 B in 2.0) Relaltionship size (33B) x expected relationships Property size (41B) x expected properties Strings & Arrays
- 37. Configuration neo4j.properties File Caches: Mapped memory Object Caches Indexes HA Backup
- 39. Configuration 21.2. Server Configuration 25. Configuration & Performance
- 40. neo4j: Buffers, Caching & I/O neo4j-wrapper.conf #IiilJv Ha Sz (nM) nta aa ep ie i B waprjv.nteoy12 rpe.aaiimmr=04 #MxmmJv Ha Sz (nM) aiu aa ep ie i B waprjv.ammr=04 rpe.aamxeoy12
- 41. neo4j: Buffers, Caching & I/O neo4j.properties Two types of caches: file buffer and object cache File Buffer Cache: #Dfutvle frtelwlvlgahegn eal aus o h o-ee rp nie notr.oetr.bmpe_eoy2M esoendsoed.apdmmr=5 notr.eainhptr.bmpe_eoy5M esoerltosisoed.apdmmr=0 notr.rprytr.bmpe_eoy9M esoepoetsoed.apdmmr=0 notr.rprytr.bsrnsmpe_eoy10 esoepoetsoed.tig.apdmmr=3M notr.rprytr.bary.apdmmr=3M esoepoetsoed.rasmpe_eoy10 Object Cache: nd_ah_ie26 oeccesz=5M rltosi_ah_ie26 eainhpccesz=5M #otoa pinl nd_ah_ra_rcin5 oeccearyfato= rltosi_ah_ra_rcin5 eainhpccearyfato= #TeG rssatccedsrbdblwi ol aalbei te h C eitn ah ecie eo s ny vial n h #NojEtrrs Eiin e4 nepie dto. #ccetp vle:sf (eal) wa,srn ah_ye aus ot dfut, ek tog ccetp=c ah_yegr
- 42. neo4j: Concurrency neo4j.properties #cnurn HT rqet ta tesre wl srie ocret TP euss ht h evr il evc. ognojsre.esre.atras6 r.e4.evrwbevrmxhed=4
- 43. neo4j: HA neo4j-server.properties ognojsre.aaaemd=A r.e4.evrdtbs.oeH neo4j.properties h.evri= asre_d1 h.nta_ot=evr:01sre250 aiiilhsssre150,evr:01 #adsoeyulht:/xml.o/it h.icvr.r=tp/eapecmls #ot&pr t bn tecutrmngmn cmuiain Hs ot o id h lse aaeet omncto. h.lse_evrsre150 acutrsre=evr:01 #otaeadpr t bn teH sre. Hsnm n ot o id h A evr h.evrm-oancm60 asre=ydmi.o:01 ###Otoa cutrsrtge ### ## pinl lse taeis ## #Itra o pligudtsfo mse. nevl f uln pae rm atr h.ulitra=0 apl_nevl1s #h aon o sae temse wl akt rpiaeacmitd Te mut f lvs h atr il s o elct omte #rnato. tascin h.xps_atr1 at_uhfco= #uhsrtg o atascint asaedrn cmi. Ps taey f rnato o lv uig omt h.xps_taeyfxd#o rudrbn at_uhsrtg=ie r on_oi
- 45. Use File System $AHT_E4 =/p/e4 PT_ONOJ otnoj /p/e4/i otnojbn noj e4 nojbcu e4-akp /p/e4/of otnojcn nojpoete e4.rpris nojsre.rpris e4-evrpoete nojwaprcn e4-rpe.of /p/e4/aa otnojdt /p/e4/aagahd otnojdt/rp.b Teata gahdt h cul rp aa /p/e4/aalg otnojdt/o Allg l os
- 46. Use File System $AHT_E4 =/p/e4 PT_ONOJ otnoj /p/e4/i (urbnmsl otnojbn /s/i/yq) noj e4 nojbcu e4-akp /p/e4/of (ecmsl otnojcn /t/yq) nojpoete e4.rpris nojsre.rpris e4-evrpoete nojwaprcn e4-rpe.of /p/e4/aa(vrlbmsl otnojdt /a/i/yq) /p/e4/aagahd (vrlbmsldt) otnojdt/rp.b /a/i/yq/aa Teata gahdt h cul rp aa /p/e4/aalg(vrlgmsl otnojdt/o /a/o/yq) Allg l os
- 47. Use Indexes The database itself is a natural index Lucene for searches neo4j 2.0: Nodes have labels: Person, Location, etc. that group them into sets CET IDXO :esnnm) RAE NE N Pro(ae Look familiar? CET IDXi_ne O Pro (d; RAE NE didx N esn i)
- 48. Use Indexes neo4j 2.0: Properties can have unique constraints CET CNTAN O (okBo)ASR bo.snI UIU RAE OSRIT N bo:ok SET okib S NQE Look familiar? CET UIU IDXealidxO Pro (mi) RAE NQE NE mi_ne N esn eal;
- 49. Use Indexes Current 1.9.x: Auto indexing (deprecated): one for nodes, one for relationships off by default
- 50. Use Querying msl slc *fo gahlcllmt1; yq> eet rm rp_oa ii 0 +-------------------------------+ --+---------+----+-------+-----|i |gahtmlt_d|hs_d|sm_ur_d|sm_ne | d rp_epaei oti npqeyi npidx +-------------------------------+ --+---------+----+-------+-----| 1| 1 | 2 1| 0| | | 2| 9| 1| 0| | | 3| 1 | 0 1| 0| | | 4| 8| 1| 0| | | 5| 5 | 8 2| 0| | | 6| 6 | 2 2| 0| | | 7| 5 | 3 2| 0| | | 8| 3 | 7 2| 0| | | 9| 6 | 7 2| 0| | |1 | 0 6 | 5 2| 0| | +-------------------------------+ --+---------+----+-------+-----1 rw i st(.0sc 0 os n e 00 e)
- 53. Use Querying via REST PS ht:/oahs:44d/aacpe OT tp/lclot77/bdt/yhr Acp:apiainjo;castUF8 cet plcto/sn hre=TCnetTp:apiainjo otn-ye plcto/sn { } "ur":"tr x =nd:oeat_ne(ae{trNm} qey sat oend_uoidxnm=satae) mthpt =(-r-red ac ah x[]fin) weefin.ae={ae rtr TP(), hr rednm nm} eun YEr" "aas :{ prm" "trNm":"" satae I, "ae :"o" nm" yu } Example response: 20 O 0: K CnetTp:apiainjo;castUF8 otn-ye plcto/sn hre=T{ } "oun":["YEr"] clms TP() , "aa :[["nw ]] dt" ko"
- 54. DBA Perspective Use the best database for the job, or both neo4j ships with great tools neo4j is easier to configure: fewer options, less complex, still flexible for optimization HA more robust and more opaque than basic replication For better or worse, JVM handles a lot for you Authorization - it's up to you Scaling up is easier than changing your data model
- 56. Thank You! Thanks to: Aseem Kishore @aseemk Chris Leishman @cleishm Max De Marzi @maxdemarzi