Glusterd_thread_synchronization_using_urcu_lca2016
Download as ODP, PDF0 likes409 views
This document discusses the thread synchronization methods used in glusterd, focusing on the transition from single-threaded to multi-threaded design and the challenges associated with the 'big lock' approach. It introduces Read-Copy-Update (RCU) as a more efficient synchronization mechanism that allows concurrent reads and updates, avoiding the pitfalls of deadlocks and writer starvation. The document also explores various RCU flavors and APIs, highlighting their practical applications and benefits.
Glusterd_thread_synchronization_using_urcu_lca2016
- 2. 2 Glusterd Thread Synchronization using user space RCU Atin Mukherjee SSE-Red Hat Gluster Maintainer IRC : atinm on freenode Twitter: @mukherjee_atin
- 3. 3 Agenda ● Introduction to GlusterD ● Big lock in thread synchronization in GlusterD ● Issues with Big Lock approach ● Different locking primitives ● What is RCU ● Advantage of RCU over read-write lock ● RCU mechanisms – Insertion, Deletion, Reader ● URCU flavors ● URCU APIs ● URCU use cases ● Q&A
- 4. 4 What is GlusterD ● Manages the cluster configuration for Gluster ● Responsible for – Peer membership management – Elastic volume management – Configuration consistency – Distributed command execution (orchestration) – Service management (manages GlusterFS daemons)
- 5. 5 Thread synchronization in GlusterD ● GlusterD was initially designed as single threaded ● Single threaded → Multi threaded to satisfy usecases like snapshot ● Big lock – A coarse grained lock – Only one transaction can work inside big lock – Protects all the shared data structures
- 6. 6 Issues with Big Lock ● Threads contend for even unrelated data ● Can end up in a deadlock – RPC request's callback also needs big lock ● Shall we release big lock in between a transaction to get rid of above deadlock? Yes we do, but…. ● Here come's the problem - a small window of time when the shared data structures are prone to updates leading to inconsistencies
- 7. 7 Different locking primitives ● Fine grained locks – Mutex – Read-write lock – Spin lock – Seq lock – Read-Copy-Update (RCU)
- 8. 8 What is RCU ● Synchronization mechanism ● Not new, added to Linux Kernel in 2002 ● Allows reads to occur concurrently with update ● Maintains multiple version of objects for read coherency ● Almost zero over heads in read side critical section
- 9. 9 Advantages of RCU over read-write lock ● Concurrent readers & writers – writer writes, readers read ● Wait free reads – RCU readers have no wait overhead. They can never be blocked by writers ● Existence guarantee – RCU guarantees that RCU protected data in a readers critical section will remain in existence till the end of the critical section ● Deadlock immunity – RCU readers always run in a deterministic time as they never block. This means that they can never become a part of a deadlock. ● No writer starvation – As RCU readers don't block, writers can never starve.
- 10. 10 RCU mechanism ● RCU is made up of three fundamental mechanisms – Publish-Subscribe Mechanism (for insertion) – Wait For Pre-Existing RCU Readers to Complete (for deletion) – Maintain Multiple Versions of Recently Updated Objects (for readers)
- 11. 11 Publish-Subscribe model ● rcu_assign_pointer () for publication 1 struct foo { 2 int a; 3 int b; 4 int c; 5 }; 6 struct foo *gp = NULL; 7 8 /* . . . */ 9 10 p = malloc (...); 11 p->a = 1; 12 p->b = 2; 13 p->c = 3; 14 gp = p; 1 struct foo { 2 int a; 3 int b; 4 int c; 5 }; 6 struct foo *gp = NULL; 7 8 /* . . . */ 9 10 p = malloc (...); 11 p->a = 1; 12 p->b = 2; 13 p->c = 3; 14 rcu_assign_pointer(gp, p); ● rcu_dereference () for subscription 1 p = gp; 2 if (p != NULL) { 3 do_something_with(p->a, p->b, p->c); 4 } 1 rcu_read_lock(); 2 p = rcu_dereference(gp); 3 if (p != NULL) { 4 do_something_with(p->a, p->b, p->c); 5 } 6 rcu_read_unlock();
- 12. 12 Publish-Subscribe Model (ii) ● rcu_assign_pointer () & rcu_dereference () embedded in special RCU variants of Linux's list-manipulation API ● rcu_assign_pointer () → list_add_rcu () ● rcu_dereference () → list_for_each_entry_rcu ()
- 13. 13 Wait For Pre-Existing RCU Readers to Complete ● Approach used for deletion ● Synchronous – synchronize_rcu () ● Asynchronous – call_rcu () q = malloc(...); *q = *p; q->b = 2; q->c = 3; list_replace_rcu(&p->list, &q->list); synchronize_rcu(); free(p) q = malloc(...); *q = *p; q->b = 2; q->c = 3; list_replace_rcu(&p->list, &q->list); call_rcu (&p->list, cbk); /* cbk will free p */
- 14. 14 Maintain multiple version objects ● Used for existence gurantee 1. p = search(head, key); 2. list_del_rcu(&p->list); 3. synchronize_rcu(); 4. free (p); 1. p = search(head, key); 2. list_del_rcu(&p->list); 3. synchronize_rcu(); 4. free (p); 1. p = search(head, key); 2. list_del_rcu(&p->list); 3. synchronize_rcu(); 4. free (p); Maintain multiple version objects ● Used for existence gurantee 1. p = search(head, key); 2. list_del_rcu(&p->list); 3. synchronize_rcu(); 4. free (p); 1. p = search(head, key); 2. list_del_rcu(&p->list); 3. synchronize_rcu(); 4. free (p); 1. p = search(head, key); 2. list_del_rcu(&p->list); 3. synchronize_rcu(); 4. free (p);
- 15. 15 URCU flavors ● QSBR (quiescent-state-based RCU) – each thread must periodically invoke rcu_quiescent_state() – Thread (un)registration required ● Memory-barrier-based RCU – Preemptible RCU implementation – Introduces memory barrier in read critical secion, hence high read side overhead ● “Bullet-proof” RCU (RCU-BP) – Similar like memory barrier based RCU but thread (un)registration is taken care – Primitive overheads but can be used by application without worrying about thread creation/destruction
- 16. 16 URCU flavors (ii) ● Signal-based RCU – Removes memory barrier – Can be used by library function – requires that the user application give up a POSIX signal to be used by synchronize_rcu() in place of the read-side memory barriers. – Requires explicit thread registration ● Signal-based RCU using an out-of-tree sys_membarrier() system call – sys_membarrier() system call instead of POSIX signal
- 17. 17 URCU APIs ● Atomic-operation and utility APIs – caa_: Concurrent Architecture Abstraction. – cmm_: Concurrent Memory Model. – uatomic_: URCU Atomic Operation. – https://lwn.net/Articles/573435/ ● The URCU APIs – https://lwn.net/Articles/573439/ ● RCU-Protected Lists – https://lwn.net/Articles/573441
- 18. 18 When is URCU useful
- 19. 19 References ● https://lwn.net/Articles/262464/ ● https://lwn.net/Articles/263130/ ● https://lwn.net/Articles/573424/ ● http://www.efficios.com/pub/lpc2011/Presentation- lpc2011-desnoyers-urcu.pdf ● http://www.rdrop.com/~paulmck/RCU/RCU.IISc- Bangalore.2013.06.03a.pdf ● http://urcu.so/