![m[un]map
✤ Log-structured storage wants to remove old files post-
compaction; some platforms disallow deleting open files
✤ Old workaround (pre-1.0):
✤ use PhantomReference to tell when mmap’d file is GC (hence
unmapped)
✤ Poor user experience and messy corner cases
✤ New workaround:
✤ Class.forName("sun.nio.ch.DirectBuffer").getMethod("cleaner")](https://image.slidesharecdn.com/fosdem2012-120205110624-phpapp01/85/Dealing-with-JVM-limitations-in-Apache-Cassandra-Fosdem-2012-18-320.jpg)
Dealing with JVM limitations in Apache Cassandra (Fosdem 2012)
- 1. Dealing with JVM limitations in Apache Cassandra Jonathan Ellis / @spyced
- 2. Pain points for Java databases ✤ GC ✤ GC ✤ GC
- 3. Pain points for Java databases ✤ GC ✤ Platform specific code
- 4. GC ✤ Concurrent and compacting: choose one ✤ G1 ✤ Azul C4 / Zing?
- 5. Fragmentation ✤ Bloom filter arrays ✤ Compression offsets
- 6. Automatic mitigation? ✤ http://www.research.ibm.com/people/d/dfb/papers/Bacon03Controlling.pdf ✤ http://researcher.ibm.com/files/us-hirzel/pldi10-arraylets.pdf
- 7. Fragmentation, 2 ✤ Arena allocation for memtables
- 8. (Memtables?) write( k1 , c1:v1 ) Memory Memtable Commit log Hard drive
- 9. write( k1 , c1:v ) Memory k1 c1:v Memtable k1 c1:v Commit log Hard drive
- 10. write( k1 , c2:v ) Memory k1 c1:v c2:v k1 c1:v k1 c2:v Hard drive
- 11. write( k2 , c1:v c2:v ) Memory k1 c1:v c2:v k2 c1:v c2:v k1 c1:v k1 c2:v k2 c1:v c2:v Hard drive
- 12. write( k1 , c1:v c3:v ) Memory k1 c1:v c2:v c3:v k2 c1:v c2:v k1 c1:v k1 c2:v k2 c1:v c2:v k1 c1:v c3:v Hard drive
- 13. Memory flush index cleanup k1 c1:v c2:v c3:v k2 c1:v c2:v SSTable Hard drive
- 14. “Java is a memory hog” ✤ Large overhead for typical objects and collections ✤ How large? ✤ java.lang.instrument.Instrumentation ✤ JAMM: Java Agent for Memory Measurements ✤ https://github.com/jbellis/jamm
- 15. org.apache.cassandra.cache.SerializingCache ✤ Live objects are about 85% JVM bookeeping ✤ org.apache.cassandra.cache.FreeableMemory using reference counting ✤ Considering doing reference-counted, off-heap memtables as well
- 16. Don’t forget about young gen ✤ Always stop-the-world for ~100ms
- 17. Platform-specific code ✤ OS ✤ JVM
- 18. m[un]map ✤ Log-structured storage wants to remove old files post- compaction; some platforms disallow deleting open files ✤ Old workaround (pre-1.0): ✤ use PhantomReference to tell when mmap’d file is GC (hence unmapped) ✤ Poor user experience and messy corner cases ✤ New workaround: ✤ Class.forName("sun.nio.ch.DirectBuffer").getMethod("cleaner")
- 19. mmap part 2 ✤ 2GB limit via ByteBuffer: public abstract byte get(int index) ✤ Workaround: MmappedSegmentedFile public Iterator<DataInput> iterator(long position)
- 20. link ✤ Used for snapshots ✤ Old workaround: JNA ✤ New workaround: supported directly by Java7
- 21. mlockall ✤ swappiness: pissing off database developers since 2001 (?) ✤ mlockall(MCL_CURRENT)
- 22. Low-level i/o ✤ posix_fadvise ✤ mincore/fincore ✤ fctl ✤ ... JNA
- 23. A plug for JNA ✤ https://github.com/twall/jna static { try { Native.register("c"); ... private static native int mlockall(int flags) throws LastErrorException;
- 24. The fallacy of choosing portability over power ✤ Applets have been dead for years ✤ Python gets it right ✤ import readline
- 25. The fallacy of choosing safety over power ✤ Allowing munmap would expose developers to segfaults ✤ But, relying on the GC to clean up external resources is a well-known antipattern ✤ File.close ✤ We need munmap badly enough that we resort to unnatural and unportable code to get it ✤ You haven’t kept us from risking segfaults, you’ve just made us miserable
- 26. Compatibility through obscurity? ✤ sun.misc.Unsafe ✤ Used by high-profile libraries like high-scale-lib
- 27. ... even public options http://blogs.oracle.com/dave/entry/false_sharing_induced_by_card
- 28. Too negative?
- 29. Still true ✤ "Many concurrent algorithms are very easy to write with a GC and totally hard (to down right impossible) using explicit free." -- Cliff Click