Introduction to PalDB
1 like1,409 views
PalDB is an embeddable, write-once Java key-value store with no dependencies. It provides very fast read performance of over 2 million reads per second from a single binary file. PalDB is simple to use and works like an immutable un-typed hash map.
![Code: Write store
Java
StoreWriter writer = PalDB.createWriter(new File("store.paldb"));
writer.put("foo", "bar");
writer.put(1213, new int[] {1, 2, 3});
writer.close();
Scala
val writer: StoreWriter = PalDB.createWriter(new File("store.paldb"));
writer.put("foo", "bar");
writer.put(1213, Array(1, 2, 3));
writer.close();](https://image.slidesharecdn.com/introductiontopaldb-151026164440-lva1-app6892/85/Introduction-to-PalDB-6-320.jpg)
![Code: Read store
Java
StoreReader reader = PalDB.createReader(new File("store.paldb"));
String val1 = reader.get("foo");
int[] val2 = reader.get(1213);
reader.close();
Scala
val reader: StoreReader = PalDB.createReader(new File("store.paldb"));
val val1: String = reader.get("foo");
var val2: Array[Int] = reader.get(1213);
reader.close();](https://image.slidesharecdn.com/introductiontopaldb-151026164440-lva1-app6892/85/Introduction-to-PalDB-7-320.jpg)
Introduction to PalDB
- 1. PalDB Introduction to PalDB Mathieu Bastian - October 2015
- 2. Summary ❖ PalDB is an embeddable write-once key-value store ❖ Written in Java, no dependencies and only 110K JAR ❖ Very fast read performance, 2M+ reads/second ❖ Simple, works like an immutable un-typed HashMap ❖ Compact, holds in a single binary file ❖ Open-sourced at LinkedIn in 2015
- 3. Why PalDB? ❖ Need for an efficient solution to package side-data ❖ Inappropriate existing solutions ‣ Raw data files (CSV, JSON, Avro, Thrift) require complex parsing code and in-memory data structures ‣ Embeddable key-value stores (LevelDB, RocksDB) have large overhead due to read/write capabilities ‣ Traditional in-memory data structures (List, HashSet, HashMap) take too much memory and require load time
- 4. Features ✓ All primitives and arrays, no schema needed ✓ Random read & iteration (unsorted) ✓ No load time, and uses off-heap memory ✓ Custom serializers can be defined ✓ Read from store file, stream or resources within JAR ✓ Holds in a single binary file
- 5. Write-once ❖ Write-once, read many ❖ Once a store has been written and closed, it can’t be modified ❖ Typical use-case is to transport pre-created datasets ❖ Principal benefit is a more compact store size
- 6. Code: Write store Java StoreWriter writer = PalDB.createWriter(new File("store.paldb")); writer.put("foo", "bar"); writer.put(1213, new int[] {1, 2, 3}); writer.close(); Scala val writer: StoreWriter = PalDB.createWriter(new File("store.paldb")); writer.put("foo", "bar"); writer.put(1213, Array(1, 2, 3)); writer.close();
- 7. Code: Read store Java StoreReader reader = PalDB.createReader(new File("store.paldb")); String val1 = reader.get("foo"); int[] val2 = reader.get(1213); reader.close(); Scala val reader: StoreReader = PalDB.createReader(new File("store.paldb")); val val1: String = reader.get("foo"); var val2: Array[Int] = reader.get(1213); reader.close();
- 8. Benchmark summary ❖ When compared to embeddable key-value stores (LevelDB, RocksDB) ‣ PalDB has 5X to 15X higher throughput on datasets fitting in memory* ❖ When compared to in-memory Java HashSet/HashMap ‣ PalDB has 2X to 5X lower throughput ‣ Uses 6X less memory * PalDB does not intend to scale to very large disk indices like RocksDB or LevelDB
- 9. Throughput ❖ Throughput benchmark between PalDB, LevelDB and RocksDB (higher is better)
- 10. Memory ❖ Memory usage benchmark between PalDB and a Java HashSet (lower is better)
- 11. PalDB © 2015 LinkedIn Corp. Licensed under the terms of the Apache License, Version 2.0. Code & documentation available on GitHub https://github.com/linkedin/PalDB PalDB