Methods of NoSQL database systems benchmarking
1 like•1,138 views
Ilya Bakulin presents methods for benchmarking NoSQL database systems. He discusses the Yahoo Cloud Serving Benchmark (YCSB) framework, which allows benchmarking of NoSQL databases using common workloads. YCSB issues simple operations like insert, update, delete and scan without using SQL. It has adapters for popular NoSQL systems and allows custom workloads and databases to be added. Bakulin demonstrates YCSB by benchmarking Cassandra and sharded MySQL under different read/write ratios. Cassandra performs better in a write-heavy workload while MySQL is better for reads.
Methods of NoSQL database systems benchmarking
- 1. Methods of benchmarking NoSQL database systems Ilya Bakulin [email protected], [email protected] SMS Traffic LVEE 2011 Ilya Bakulin (SMS Traffic) NoSQL benchmarking July 2, 2011 1 / 16
- 2. 1 Introduction 2 YCSB benchmarking framework 3 YCSB practical usage 4 Results 5 Where to find further information Ilya Bakulin (SMS Traffic) NoSQL benchmarking July 2, 2011 2 / 16
- 3. Why benchmarking NoSQL is nessesary No guides / FAQs about performance are generally available, or are outdated NoSQL systems are actively developed Nobody wants to end up with crashed DB in production right before 2-week vacation Ilya Bakulin (SMS Traffic) NoSQL benchmarking July 2, 2011 3 / 16
- 4. Why benchmarking NoSQL is complex RDBMS use SQL to provide access to data stored in them, while NOSQL systems don’t Each NoSQL uses different protocol (Thrift, Memcached-style, own protocols) Existing benchmarks require SQL to work with database under inspection. Ilya Bakulin (SMS Traffic) NoSQL benchmarking July 2, 2011 4 / 16
- 5. What is YCSB? YCSB stands for Yahoo Cloud Serving Benchmark Developed by Yahoo! Research group Open Source project, hosted on GitHub (178 watchers, 42 forks) Ilya Bakulin (SMS Traffic) NoSQL benchmarking July 2, 2011 5 / 16
- 6. Architecture Benchmark tool •! Java application Java application –! Many systems have Java APIs Shipped with ready-to-use adapters for several popular Opensource databases systems via HTTP/REST, JNI or some other solution –! Other Command-line parameters •! DB to use •! Target throughput •! Number of threads •! … Workload YCSB client Cloud DB parameter file DB client •! R/W mix •! Record size Client Workload threads •! Data set executor •! … Stats Extensible: define new workloads Extensible: plug in new clients 5 Ilya Bakulin (SMS Traffic) NoSQL benchmarking July 2, 2011 6 / 16
- 7. More on DB interface Benchmark tool Simple operations: INSERT, UPDATE, REPLACE, DELETE, SCAN •! Does not use SQL Java application ... –! Many systems have Java APIs but SQL support is avaible through contributed JDBC driver ... –! Other systems configurations are possible other solution Even sharding via HTTP/REST, JNI or some Command-line parameters •! DB to use •! Target throughput •! Number of threads •! … Workload YCSB client Cloud DB parameter file DB client •! R/W mix •! Record size Client Workload threads •! Data set executor •! … Stats Extensible: define new workloads Extensible: plug in new clients 5 Ilya Bakulin (SMS Traffic) NoSQL benchmarking July 2, 2011 7 / 16
- 8. More on DB interface Benchmark tool Simple operations: INSERT, UPDATE, REPLACE, DELETE, SCAN •! Does not use SQL Java application ... –! Many systems have Java APIs but SQL support is avaible through contributed JDBC driver ... –! Other systems configurations are possible other solution Even sharding via HTTP/REST, JNI or some Command-line parameters •! DB to use •! Target throughput •! Number of threads •! … Workload YCSB client Cloud DB parameter file DB client •! R/W mix •! Record size Client Workload threads •! Data set executor •! … Stats Extensible: define new workloads Extensible: plug in new clients 5 Ilya Bakulin (SMS Traffic) NoSQL benchmarking July 2, 2011 7 / 16
- 9. More on DB interface Benchmark tool Simple operations: INSERT, UPDATE, REPLACE, DELETE, SCAN •! Does not use SQL Java application ... –! Many systems have Java APIs but SQL support is avaible through contributed JDBC driver ... –! Other systems configurations are possible other solution Even sharding via HTTP/REST, JNI or some Command-line parameters •! DB to use •! Target throughput •! Number of threads •! … Workload YCSB client Cloud DB parameter file DB client •! R/W mix •! Record size Client Workload threads •! Data set executor •! … Stats Extensible: define new workloads Extensible: plug in new clients 5 Ilya Bakulin (SMS Traffic) NoSQL benchmarking July 2, 2011 7 / 16
- 10. More on workload d by replacing the value of one Uniform: +(,-./%'&0 ther one randomly chosen field Specifies what DB 1)))2)))333 operations are used by order, starting at a randomly !"#$%&'(")(%*$% 4 number of application records to scan is Zipfian: Also defines request +(,-./%'&0 distribution of scan lengths is distribution oad. Thus, the scan() method number of records to scan.to specify It is possible Of y instead specify a scan interval 1)))2)))333 4 record size February 15th). The number of !"#$%&'(")(%*$% It’s possible to specify to control the size of these in- Latest: termine and number of records and specify meaningful +(,-./%'&0 of the database calls, including tion 5.2.1.) operations 1)))2)))333 4 !"#$%&'(")(%*$% t make many random choices Ilya Bakulin (SMS Traffic) NoSQL benchmarking July 2, 2011 8 / 16
- 11. /* TODO: Remove this crap */ Ilya Bakulin (SMS Traffic) NoSQL benchmarking July 2, 2011 9 / 16
- 12. SMS Traffic: workload construction SMS service provider, several gateways, big clients (such as banks) 15% inserts, 65% updates, 15% reads Request distribution: latest SMS messages are the ”hottest” ones Evaluated Cassandra and sharded MySQL as DB storage for the next generation of SMS sending platform Ilya Bakulin (SMS Traffic) NoSQL benchmarking July 2, 2011 10 / 16
- 13. Testing process 3 instances of DBMS system on one server (Core Quad Q9400, 4GB RAM, SATA-II HDD, FreeBSD 8.2-amd64) Cassandra 0.7.4 (1GB Java heap / instance) MySQL 5.1 + InnoDB engine (1GB InnoDB buffer pool size / instance) Client: separate machine, 1Gb/s connection Should avoid swapping and disk IO saturation Ilya Bakulin (SMS Traffic) NoSQL benchmarking July 2, 2011 11 / 16
- 14. Some resuts: Workload ”A”: 50% read / 50% write Workload A – Upd •! 50/50 Read/update Workload A - Read latency 70 80 60 70 Average read latency (ms) Update latency (ms) 60 50 50 40 40 30 30 20 20 10 10 0 0 0 5000 10000 15000 0 Throughput (ops/sec) Cassandra Hbase Sherpa MySQL Cassan Ilya Bakulin (SMS Traffic) Comment: Cassandra is optimized for writes, and achieves h NoSQL benchmarking July 2, 2011 12 / 16
- 15. oad A – Workload ”A”: 50% read / 50% write Some resuts: Update heavy y Workload A - Update latency 80 70 Update latency (ms) 60 50 40 30 20 10 0 15000 0 5000 10000 15000 ) Throughput (ops/sec) MySQL Cassandra Hbase Sherpa MySQL zed for writes, and achieves higher throughput and lower Ilya Bakulin (SMS Traffic) NoSQL benchmarking July 2, 2011 13 / 16
- 16. Some resuts: Workload ”B”: 95% read / 5% write Workload B – Re •! 95/5 Read/update Workload B - Read latency 20 40 18 35 Average update latency (ms) Average read latency (ms) 16 30 14 12 25 10 20 8 15 6 10 4 5 2 0 0 0 2000 4000 6000 8000 10000 0 2 Throughput (operations/sec) Cassandra HBase Sherpa MySQL Cassan Ilya Bakulin (SMS Traffic) NoSQL benchmarking July 2, 2011 14 / 16
- 17. oad resuts: Workload ”B”:heavy 5% write Some B – Read 95% read / Workload B - Update latency 40 35 Average update latency (ms) 30 25 20 15 10 5 0 8000 10000 0 2000 4000 6000 8000 10000 ec) Throughput (operations/sec) MySQL Cassandra Hbase Sherpa MySQL Ilya Bakulin (SMS Traffic) NoSQL benchmarking July 2, 2011 15 / 16
- 18. Links Yahoo Cloud Serving Benchmark: https://github.com/brianfrankcooper/YCSB google:// [email protected], [email protected] Ilya Bakulin (SMS Traffic) NoSQL benchmarking July 2, 2011 16 / 16