Ruby and Distributed Storage Systems
2 likes13,364 views
This document discusses using Ruby for distributed storage systems. It describes components like Bigdam, which is Treasure Data's new data ingestion pipeline. Bigdam uses microservices and a distributed key-value store called Bigdam-pool to buffer data. The document discusses designing and testing Bigdam using mocking, interfaces, and integration tests in Ruby. It also explores porting Bigdam-pool from Java to Ruby and investigating Ruby's suitability for tasks like asynchronous I/O, threading, and serialization/deserialization.
![Typing?
• Defining APIs
• Rubyists (including me) MAY be using:
[string, integer, boolean, string, ...]
• Rubyists (including me) MAY be using:
{"time": unix_time (but sometimes float)}
• Explicit definition makes nothing bad in designing APIs
• Json schema or something others may help us...](https://image.slidesharecdn.com/rubykaigi2017storagesystems-170920042448/85/Ruby-and-Distributed-Storage-Systems-36-320.jpg)
Ruby and Distributed Storage Systems
- 1. Ruby for Distributed Storage Systems RubyKaigi 2017: Sep 20, 2017 Satoshi Tagomori (@tagomoris) Treasure Data, Inc.
- 2. Satoshi Tagomori (@tagomoris) Fluentd, MessagePack-Ruby, Norikra, Woothee, ... Treasure Data, Inc.
- 4. -45°
- 5. Ruby for Distributed Storage Systems
- 6. Ruby for and Distributed Storage Systems
- 7. Ruby and Performance • Web? or Not? • Disk & Network I/O • "I/O spends most of time on servers"... is it real? • Storages are getting faster and faster (SSD, NVMe, ...) • Networks too (10GbE, fast network in Cloud, ...)
- 8. Storage Systems • Disk I/O • Network I/O • Serialization / Deserialization (json, msgpack, ...) • read/write data from/to disk • parse/generate HTTP request/response • Indexing (update, search) • Timer • Threads + Locks
- 9. Distributed Storage Systems • Data replication • Checksum • Asynchronous network I/O • Quorum • More Threads + Locks
- 10. Replication w/ 3 replicas • Create 3 replica of data, including local storage accept request to write data write the data into local storage (1) receive responses to replicate data (3) send response to write data input input input input input input input input input input input input input input input send requests to replicate data
- 11. Replication in Quoram Systems: In Action • Create 2 replica of data at least (max 3), including local storage accept request to write data, and write it locally (1) send response to write data input input input input input ? ? input input input input input input create 2 threads to send requests to replicate data input input receive a successful response to replicate data (2) ? ? ? Discard a thread for another node
- 12. Bigdam
- 13. Bigdam • Brand new data ingestion pipeline in Treasure Data • Huge data • Extraordinary large number of connections / requests • Many edge endpoints on the planet
- 14. Bigdam: Edge locations on the earth + the Central location
- 16. Bigdam-pool • OSS (in future... not yet) • Distributed key-value storage • for buffer pool in Bigdam • to build S3 free data ingestion pipeline
- 17. Bigdam-pool: Small Buffers • Small buffers (MBs) • Write: append support for many small chunks (KBs) • Read: secondary index to query/read many buffers at once • Short buffer lifetime: minutes (create - append - read - delete) • Buffers store ids of chunks (for deduplication) buffer buffer buffer buffer chunk chunk chunk chunk chunk chunk chunk chunk chunk chunk account_id, database, table
- 18. Bigdam-pool: Replication • Replication in a cluster • without maintaining replica factor • Clients send requests to all living nodes
- 19. Bigdam-pool: Buffer Transferring over Clusters Edge location Central location Over Internet Using HTTPS or HTTP/2 Buffer committed (size or timeout)
- 20. written in Java
- 21. Designing Bigdam • Architecture Design - split a system to 5 microservices • consistency, availability • performance (how to scale it out?) • deployment, cost • API Design • Mocking • Interface Test • Integration Test
- 22. Mocking Bigdam using Ruby • Mocking • build mock servers of all components • implement all public APIs between components • Find/add missing parameters required • Prepare to develop components in parallel • Mocked using Ruby, Sinatra • public APIs - it's just a Webapp • fast and easy to do :D
- 23. Interface/Integration Tests of Bigdam using Ruby • Interface tests: • verify all public APIs are implemented correctly • Integration tests • verify the whole pipeline can import data correctly • Written in Ruby, test-unit • less code to serialize/deserialize various req/res • readable test cases • fast and easy to do :D
- 24. And,
- 25. Bigdam-pool-ruby • Port bigdam-pool from Java to Ruby • Experiment to know Ruby is good enough or not
- 27. Bigdam-pool-ruby • Perfectly compatible with Java implementation • Public API, Private API • Data formats on local storage, of secondary index • Under development • only supports stand alone mode, for now
- 28. Studies: Serialization / Deserialization • All network API call requires it • parsing HTTP request • parsing request content body (json/msgpack) • building response content body (json/msgpack) • building HTTP response • Should be parallelized on CPU cores
- 29. Studies: Asynchronous Network I/O • EventMachine? Cool.io? Celluloid::IO? • 🤔 • I want to use only async network I/O at a time! (not disk, not timer) • Event driven I/O library? • Thread pools + callback? • or any idea?
- 30. Threading / Timers • ExecutorService in Java is very useful... • Fixed / non-fixed thread pools with Queue • (and some other executor models) • Runner of Runnable tasks • "Runnable task" is just like a lambda w/o args • To be implemented as Gem? • Queue and SizedQueue look useful for it
- 31. Queue#peek Get the head object w/o removing it from queue https://github.com/ruby/ruby/pull/1698
- 32. Queue#peek Get the head object w/o removing it from queue https://github.com/ruby/ruby/pull/1698
- 33. MonitorMixin#mon_locked? and #mon_owned? • Mutex#owned? exists https://github.com/ruby/ruby/pull/1699
- 34. Resource Control Make sure to release resources: try-with-resources in Java
- 35. Resource Control Make sure to release resources: try-with-resources in Java
- 36. Typing? • Defining APIs • Rubyists (including me) MAY be using: [string, integer, boolean, string, ...] • Rubyists (including me) MAY be using: {"time": unix_time (but sometimes float)} • Explicit definition makes nothing bad in designing APIs • Json schema or something others may help us...
- 37. Typing: in logging and others https://bugs.ruby-lang.org/issues/13913
- 38. Process Built-in Application Servers • Distributed Storage Systems: • Background worker threads • Timers • Communication workers to other nodes • Various async operation workers • Public API request handlers • Private API request handlers (inter-nodes) • Startup/Shutdown hooks • It's NOT just web application, but handles HTTP requests
- 41. "Why Do You Want to Write Such Code in Ruby?"
- 42. "Why Do You Want to Write Such Code in Ruby?" "Because I WANT TO DO IT!"
- 43. "Why Do You Want to Write Such Code in Ruby?" "Because I WANT TO DO IT!" "... And we already have Fluentd :P" Thank you. @tagomoris