About sync.Pool and what changed in go1.13

Today goal

• sync.Pool interesting history
• What changed in sync.Pool from go1.13

Do you know about sync.Pool?

• https://golang.org/pkg/sync/#Pool
• A Pool is a set of temporary objects that may be individually saved and retrieved.

type Pool struct {
        New func() interface{}
}

func (p *Pool) Put(x interface{}) {}

func (p *Pool) Get() interface{} {}

• An appropriate use of a Pool is to manage a group of temporary items silently shared among and potentially reused by concurrent independent clients of a package. Pool provides a way to amortize allocation overhead across many clients.

"Appropriate" use

"Appropriate" use

"Appropriate" use

A little bit of history

• Proposed at 2013 by @bradfitz

A little bit of history

• First implement by @dvyukov
• First implement is a little more complex

func (p *Pool) Get() interface{} 

func (p *Pool) Put(v interface{}) bool 

func (p *Pool) SetCapacity(local, global int) 

func (p *Pool) Drain() []interface{} 

A little bit of history

• Many concerns by @russ
• Should users care about Drain?
• If we're going to have Drain, it should be allowed to be concurrent.
• Should users care about Capacity?
• Naming? (Cache vs Pool)

Finally

• Minimize into 2 interfaces only (same with current)
• Users don't have to care about capacity (because GC will care)
• Users don't have to care about drain (GC will do)

groups.google.com/forum/#!searchin/golang-dev/pool/golang-dev/QtSIOFsNFrQ/N89_5dZWsdIJ

A little bit of history

• First accepted version by @bradfitz

github link

• Very throughout blog http://jxck.hatenablog.com/entry/sync.Pool which explained how it works

A little bit of history

• The idea is simple

type Pool struct {
    list []interface{} // offset known to runtime
    mu   Mutex         // guards list
}
func runtime_registerPool(*Pool)
func (p *Pool) Put(x interface{}) {
    ...    
    p.mu.Lock()
    p.list = append(p.list, x)
    p.mu.Unlock()
}
func (p *Pool) Get() interface{} {
    ...
    p.mu.Lock()
    var x interface{}
    x = p.list[n-1]
    p.list = p.list[:n-1]
    p.mu.Unlock()
    ...    
    return x
}

A little bit of history

• The idea is simple
• Pool cleanup is supported by run time

// pool.go
func init() {
   runtime_registerPoolCleanup(poolCleanup)
}

func clearpools() {
     poolcleanup()
}

func poolCleanup() {
    // assign pool to nil, size to 0
}

// mgc.go
func gcStart(trigger gcTrigger) {
   [...]
   // clearpools before we start the GC
   clearpools()
     [...]
 }

What's wrong

• Mutex every single Put/Get
• Mutex is expensive
• Put/Get is very hot path normally (will be called many many times)

A little bit of history

• Improvement patch by @dvyukov

A little bit of history

• Idea is ... not simple :(

github.com/golang/go/commit/f8e0057bb71cded5bb2d0b09c6292b13c59b5748

• Overall idea: split into "internal pool" and "shared pool"
• Local pool = per "P" (do you remember about P)
• Shared pool = global pool, protect by mutex
• Get path: if local pool exhaust, "steal" from global
• Put path: if local pool's capacity > threshold, "share" to global
• In short: work stealing model
• Work stealing work by shard slice by hashing parts for each P, and lock for each shard when steal

How much it improved?

What we learned

• With help from runtime, performance could be improved by rocket speed..

Problem (before 1.13)

• GC problem (spike when GC happen for very big pool)
• Pool stealing still require some lock

Now come 1.13 changes!

1. Make Pool stealing lock free!!

• sync: use lock-free structure for Pool stealing
• sync: internal dynamically sized lock-free queue for sync.Pool
• sync: internal fixed size lock-free queue for sync.Pool

go-review.googlesource.com/c/go/+/166960

go-review.googlesource.com/c/go/+/166957/7

go-review.googlesource.com/c/go/+/166958/8

1. Make Pool stealing lock free!!

• Idea: use lock-free double linked list with atomic CAS
• Each processor (P) with push and pop at the head of its queue while the shared access will pop from the tail
• Note: lock free != contention free (CAS is not free)

1. Make Pool stealing lock free!!

2. Make GC less spiky

• sync: smooth out Pool behavior over GC with a victim cache

go-review.googlesource.com/c/go/+/166961/10

2. Make GC less spiky

> Currently, every Pool is cleared completely at the start of each GC.
This is a problem for heavy users of Pool because it causes an
allocation spike immediately after Pools are clear, which impacts both
throughput and latency.

• We could make problem better by NOT clear data at once

2. Make GC less spiky

• Ideas: Split into 2 pools, victim and primary

func poolCleanup() {
[...]
// Drop victim caches from all pools.
for _, p := range oldPools {
   p.victim = nil
   p.victimSize = 0
}

// Move primary cache to victim cache.
for _, p := range allPools {
   p.victim = p.local
   p.victimSize = p.localSize
   p.local = nil
   p.localSize = 0
}

// The pools with non-empty primary caches now have non-empty
// victim caches and no pools have primary caches.
oldPools, allPools = allPools, nil
[...]
}

2. Make GC less spiky

• Why it helped?
• Each GC, drop victim, and copy primary to victim
• Result: active objects (frequently be Get) will remain in the heap, and less allocation
• Interesting: don't have to sacrify Get/Put performance but improve Pool impact when a GC happens

Finally

• Design process from go team show us how to make good software
• Go team were obsessed to final simplest interface/solution
• Run time trick (per P local + work stealing) to improve performance is very interesting

Thank you