Dynomite - PerconaLive 2017
0 likes661 views
Dynomite is a framework that converts non-distributed data stores into distributed systems, offering features like high availability, automatic failover, and tunable consistency. It has been deployed in production at Netflix for around 2.5 years, utilizing 70 clusters and approximately 1000 nodes to support internal microservices. The Dyno client provides advanced capabilities such as connection pooling, load balancing, and effective failover options, while the ecosystem includes components like Dynomite Manager for orchestration and Dynomite Explorer for a user interface.
Dynomite - PerconaLive 2017
- 1. Making Non-Distributed Databases, Distributed Ioannis Papapanagiotou, PhD Shailesh Birari
- 2. Dynomite Ecosystem ● Dynomite - Proxy layer ● Dyno - Client ● Dynomite-manager - Ecosystem orchestrator ● Dynomite-explorer - UI
- 3. ● Needed a data store: o Scalable & highly available o High throughput, low latency o Netflix use case is active-active ● Master-slave storage engines: o Do not support bi-directional replication o Cannot withstand a Monkey attack o Cannot easily perform maintenance Problems & Observations
- 4. What is Dynomite? A framework that makes non-distributed data stores, distributed. Can be used with many key-value storage engines Features: highly available, automatic failover, node warmup, tunable consistency, backups/restores
- 5. Dynomite @ Netflix ● Running around 2.5 years in PROD ● 70 clusters ● ~1000 nodes used by internal microservices ● Microservices based on Java, Python, NodeJS
- 6. Pluggable Storage Engines RESP ● Layer on top of a non- distributed key value data store ○ Peer-peer, Shared Nothing ○ Auto-Sharding ○ Multi-datacenter ○ Linear scale ○ Replication ○ Gossiping RESP
- 7. ● Each rack contains one copy of data, partitioned across multiple nodes in that rack ● Multiple Racks == Higher Availability (HA) Topology
- 8. Replication ● A client can connect to any node on the Dynomite cluster when sending requests. o If node owns the data, ▪ data are written in local data-store and asynchronously replicated. o If node does not own the data ▪ node acts as a coordinator and sends the data in the same rack & replicates to other nodes in other racks and DC.
- 9. Dyno Client - Java API ● Connection Pooling ● Load Balancing ● Effective failover ● Pipelining ● Scatter/Gather ● Metrics, e.g. Netflix Insights
- 10. Dyno Load Balancing ● Dyno client employs token aware load balancing. ● Dyno client is aware of the cluster topology of Dynomite within the region, can write to specific node using consistent hashing.
- 11. Dyno Failover ● Dyno will route requests to different racks in failure scenarios.
- 12. Dynomite on the Cloud RESP
- 13. Moving across engines Rack A Rack B
- 14. Dynomite-manager: Warm up 1. Dynomite-manager identifies which node has the same token in the same DC 2. Leverage master/slave replication 3. Checks for peer syncing a. difference between master and slave offset 4. Once master and slave are in sync, Dynomite is set to allow write only 5. Dynomite is set back to normal state 6. Checks for health of the node - Done!
- 15. Dynomite-Explorer (UI) • Node.js web app with a Polymer-based user-interface • Support Redis’ rich data types • Avoid operations that can negatively impact Redis server performance • Extended for Dynomite awareness • Allow extension of the server to integrate with the Netflix ecosystem
- 17. Roadmap ● Data reconciliation & repair v2 ● Optimizations of RocksDB configuration ● Optimizing backups through SST ● Others….
- 18. More information • Netflix OSS: • https://github.com/Netflix/dynomite • https://github.com/Netflix/dyno • https://github.com/Netflix/dynomite- manager • Chat: https://gitter.im/Netflix/dynomite