![www.mapdb.org
TreeMap
● Full ConcurrentNavigableMap implementation
● (only alternative implementation I know about)
● Transparent specialized keys
● Long → long
● Strings → byte[] or char[] depending on
encoding
● B-Link Tree ( Yehoshua Sagiv 1986)
● Highly concurrent
● No locks on read, one Lock per node on
update
● On delete empty nodes are not removed :-(
● Modification listeners](https://image.slidesharecdn.com/mapdb-intro-160108210808/85/MapDB-taking-Java-collections-to-the-next-level-9-320.jpg)
![www.mapdb.org
TreeMap – Key Representation
● class LeafNode{ Object[] keys, Object[] values}
● Data interpretation such as array size, binary search, update,
split... is done in plugable serializer
● Array of keys and values can be represented in many ways
● Object[]
● long[]
● char[][] → char[]
● Delta compression
● [ 6001, 6001, 6002 ] stored as [ 6000, 1, 1 ]
→ only 4 bytes in packed longs
● Common prefix compression
● [ “New Orleans, “New York” ] stored as
[ “New “, ”Orleans”, ”York”]](https://image.slidesharecdn.com/mapdb-intro-160108210808/85/MapDB-taking-Java-collections-to-the-next-level-11-320.jpg)
MapDB - taking Java collections to the next level
- 1. www.mapdb.org MapDB Taking java collections to next level www.mapdb.org
- 2. www.mapdb.org Me ● [email protected] ● @JanKotek ● Independent consultant ● I took over MapDB in 2010 (it started in 1999 under name JDBM) ● Last 3 years I work on MapDB full time
- 3. www.mapdb.org MapDB ● MapDB is embedded database engine ● ACID, isolation etc..., this talks only covers in-memory mode with transactions disabled ● Apache 2 licensed ● Maps and other Java collections ● Flexible component architecture ● Very fast ● Speed comporable to Java Collections ● No degradation by Garbage Collection ● 600 KB jar, no deps, pure java
- 4. www.mapdb.org Hello World DB db = DBMaker .memoryDB() .make(); Map<Long,UUID> map = db .treeMapCreate("map") .keySerializer(Serializer.LONG) .valueSerializer(Serializer.UUID) .nodeSize(64) .make();
- 5. www.mapdb.org Random Map<Long,UUID>.put() performance Time to update 100M random keys on Map with 100M entries (smaller is better)
- 6. www.mapdb.org Random Map<Long,UUID>.get() performance Time to get 100M random keys on Map with 100M entries (smaller is better)
- 7. www.mapdb.org HashMap ● Hashtable is tree with upto 3 levels ● (no fixed sized array) ● No rehashing and no reinsert on grow or shrink ● Empty hash position does not use space ● Concurrent, 16 segments with separate locking ● Expiration ● Modification Listeners
- 8. www.mapdb.org HashMap - expiration ● Expiration based on TTL, memory limit and number of entries ● Aproximate expiration (+- 100 entries) for better performance ● Concurrent, 16 separate segment, each with separate queue and lock Map map = db .hashMapCreate("cache") .expireMaxSize(1000000) .expireAfterWrite(30, TimeUnit.HOURS) .expireAfterAccess(10, TimeUnit.HOURS) .make();
- 9. www.mapdb.org TreeMap ● Full ConcurrentNavigableMap implementation ● (only alternative implementation I know about) ● Transparent specialized keys ● Long → long ● Strings → byte[] or char[] depending on encoding ● B-Link Tree ( Yehoshua Sagiv 1986) ● Highly concurrent ● No locks on read, one Lock per node on update ● On delete empty nodes are not removed :-( ● Modification listeners
- 10. www.mapdb.org TreeMap<Long,UUID> – number of entries in 1GB memory
- 11. www.mapdb.org TreeMap – Key Representation ● class LeafNode{ Object[] keys, Object[] values} ● Data interpretation such as array size, binary search, update, split... is done in plugable serializer ● Array of keys and values can be represented in many ways ● Object[] ● long[] ● char[][] → char[] ● Delta compression ● [ 6001, 6001, 6002 ] stored as [ 6000, 1, 1 ] → only 4 bytes in packed longs ● Common prefix compression ● [ “New Orleans, “New York” ] stored as [ “New “, ”Orleans”, ”York”]
- 12. www.mapdb.org TreeMap - Data Pump ● Imports TreeMap very fast ● First creates Leaf Nodes, than builds Tree Nodes on top ● Insert only operation, no random updates ● Inserts millions of entries per second ● Insert speed is constant, no degradation with large sets ● 1B (1e9) entry map is created overnight on slow rotating HDD
- 13. www.mapdb.org Bind ● Collections provide Modification Listeners ● Bind is utility on top of listeners which binds two collections together ● Secondary collection is automatically modified by changes in first
- 14. www.mapdb.org Bind – secondary Map HTreeMap<Long, String> primary = DBMaker.memoryDB().make().hashMap("test"); // secondary map will hold String.size() Map<Long,Integer> secondary = new HashMap(); //Bind maps together Bind.secondaryValue(primary, secondary, { (Long key, String value) => return value.lenght() }); primary.put(111L, "just some chars"); secondary.get(111L) => 15
- 15. www.mapdb.org Bind – overflow to disk after expiration // slow large collection on disk HTreeMap onDisk = db.hashMap("onDisk").make(); // fast in-memory collection with limited size // its content is moved to disk after it expires HTreeMap inMemory = db .hashMapCreate("inMemory") .expireAfterAccess(1, TimeUnit.SECONDS) // register overflow .expireOverflow(onDisk, true) .executorEnable() .make();
- 16. www.mapdb.org Current status ● MapDB 1.0 is out ● MapDB 2.0 release is coming soon ● At this point its faster and more stable than 1.0 ● Issues with on-disk mode and recovery after crash ('kill -9' unit test fails now) ● MapDB 2.1 will follow soon ● It will improve concurrency ● Will have long term support for couple of years (API, format) ● Java 8 Streams support
- 17. www.mapdb.org Conclusion ● Better memory usage ● Reasonable performance ● Extra features such as expiration ● I hope you will find it useful :-) ● Resources ● www.mapdb.org ● github.com/jankotek/mapdb