Distributed Systems explained (with NodeJS) - Bruno Bossola, JUG Torino
- 1. Distributed Systems + NodeJS Bruno Bossola TEL AVIV 30 MARCH 2017 @bbossola
- 2. @bbossola Whoami ● Developer since 1988 ● XP Coach 2000+ ● Co-founder of JUG Torino ● Java Champion since 2005 ● I live in London, love the weather...
- 3. @bbossola Agenda ● Distributed programming ● How does it work, what does it mean ● The CAP theorem ● CAP explained with live demos! – CA system using two phase commit – AP system using sloppy quorums – CP system using majority quorums ● Q&A This is a 30 minutes version of the original 120mins presentation!! 0 min. 10 mins. 20 mins.
- 4. @bbossola Distributed programming ● Do we need it?
- 5. @bbossola Distributed programming ● Any system should deal with two tasks: – Storage – Computation ● We will look here at the storage part of the equation. We need. three basic properties: – Scalability – Availability – Consistency
- 6. @bbossola Scalability ● The ability of a system/network/process to: – handle a growing amount of work – be enlarged to accommodate new growth A scalable system continue to meet the needs of its users as the scale increase clipart courtesy of openclipart.orgclipart courtesy of openclipart.org
- 7. @bbossola How do we scale? partitioning ● Slice the dataset into smaller independent sets ● reduces the impact of dataset growth – improves performance by limiting the amount of data to be examined – improves availability by the ability of partitions to fail indipendently
- 8. @bbossola How do we scale? partitioning ● But can also be a source of problems – what happens if a partition become unavailable? – what if It becomes slower? – what if it becomes unresponsive? clipart courtesy of openclipart.org
- 9. @bbossola How do we scale? replication ● Copies of the same data on multiple machines ● Benefits: – allows more servers to take part in the computation – improves performance by making additional computing power and bandwidth – improves availability by creating copy of the data
- 10. @bbossola How do we scale? replication ● But it's also a source of problems – there are independent copies of the data – need to be kept in sync on multiple machines ● Your system must follow a consistency model v4 v4 v8 v8 v4 v5 v7 v8 clipart courtesy of openclipart.org
- 11. @bbossola Availability ● The proportion of time a system is in functioning conditions ● The system is fault-tolerant – the ability of your system to behave in a well defined manner once a fault occurs ● All clients can always read and write – In distributed systems this is achieved by redundancy clipart courtesy of openclipart.org
- 12. @bbossola Consistency ● Any read on a data item X returns a value corresponding to the result of the most recent write on X. ● Each client always has the same view of the data ● Also know as “Strong Consistency” clipart courtesy of cliparts.co
- 13. @bbossola Consistency flavours ● Strong consistency – every replica sees every update in the same order. – no two replicas may have different values at the same time. ● Weak consistency – every replica will see every update, but possibly in different orders. ● Eventual consistency – every replica will eventually see every update and will eventually agree on all values.
- 14. @bbossola The CAP theorem CONSISTENCY AVAILABILITY PARTITION TOLERANCE
- 15. @bbossola The CAP theorem ● You cannot have all :( ● You can select two properties at once Sorry, this has been mathematically proven and no, has not been debunked.
- 16. @bbossola The CAP theorem CA systems! ● You selected consistency and availability! ● Strict quorum protocols (two/multi phase commit) ● Most RDBMS Hey! A network partition will f**k you up good!
- 17. @bbossola The CAP theorem AP systems! ● You selected availability and partition tolerance! ● Sloppy quorums and conflict resolution protocols ● Amazon Dynamo, Riak, Cassandra
- 18. @bbossola The CAP theorem CP systems! ● You selected consistency and partition tolerance! ● Majority quorum protocols (paxos, raft, zab) ● Apache Zookeeper, Google Spanner
- 19. @bbossola NodeJS time! ● Let's write our brand new key value store ● We will code all three different flavours ● We will have many nodes, fully replicated ● No sharding ● We will kill servers! ● We will trigger network partitions! – (no worries. it's a simulation!) clipart courtesy of cliparts.co
- 20. @bbossola CA key-value store ● Uses classic two-phase commit ● Works like a local system ● Not partition tolerant ● Classic RDMBS (strict quorum)
- 21. @bbossola AP key-value store ● Eventually consistent design ● Prioritizes availability over consistency ● Dynamo, Riak, Cassandra (sloppy quorum)
- 22. @bbossola CP key-value store ● Uses majority quorum (raft) ● Guarantees eventual consistency ● Zookeper, Spanner (majority quorum)
- 23. @bbossola Q&A Amazon Dynamo: http://www.allthingsdistributed.com/2007/10/amazons_dynamo.html The RAFT consensus algorithm: https://raft.github.io/ http://thesecretlivesofdata.com/raft/ The code used into this presentation: https://github.com/bbossola/sysdist