October 2016 HUG: Architecture of an Open Source RDBMS powered by HBase and Spark 

Editor's Notes

• #3: The Hadoop RDBMS is designed to scale-out from a single server to thousands of machines, with a high degree of fault tolerance. Rather than relying on high-end hardware, Splice Machine uses the proven scale-out and high availability of Hadoop, proven in production clusters of dozens of petabytes at large scale leaders like Yahoo, Facebook, and Twitter. The Hadoop RDBMS benefits include: Affordability – scale-out -- using commodity hardware Elasticity -- expand or scale back easily Transactional – execute real time updates and ACID transactions ANSI SQL -- leverage existing SQL code, tools, and skills Flexibility -- support both operational and analytical workloads Notes: SQL: Structured Query Language. SQL is a special-purpose programming language designed for managing data held in a relational database management system (RDBMS).
• #13: Splice Machine has focused on the orange blocks to maximize the value of our R&D investment Derby's database engine, is a full-functioned relational embedded database-engine, supporting JDBC and SQL as programming APIs. It uses IBM DB2 SQL syntax. Apache Derby originated at Cloudscape Inc, an Oakland, California, start-up founded in 1996 In 1999 Informix Software, Inc., acquired Cloudscape, Inc. In 2001 IBM acquired the database assets of Informix Software, including Cloudscape In August 2004 IBM contributed the code to the Apache Software Foundation as Derby Splice Machine has focused the middle of the stack to maximize the value created by our R&D Our parallelization engine to execute SQL Secondary indexes Join strategies Query optimizers Performance High concurrency, lockless programming
• #14: HBase is a “distributed, versioned, non-relational database modeled after Google's Bigtable, a distributed storage system for structured data”. HBase can handle very high throughput scaling. Fully leverage Hbase as a storage engine for horizontal scale-out Auto-sharding in Hbase is based on regions with regions assigned to region servers Hbase does region balancing/re-balancing Recall that regions are ranges of rowkeys in sorted order RDBMS primary key uses Hbase rowkeys Fast single row selects Fast range based scans Dense secondary indices stored in separate tables Write pipeline Writes to WAL first (not shown) for durability The writes to memstore Memstores are eventually flushed to disk to storefiles Read pipeline Blocks are read from storefiles and memstores Blocks are cached in Block Cache (not shown) Remember that HDFS is an immutable file system Storefiles are written and never updated Updates are really inserts (upserts)
• #17: Key points A theme in Distributed computing is moving the code to where the data is because the data is big Splice Machine has its own query execution and task parallelization engine Secret sauce Not based on map/reduce Predicates pushed to shards and locally applied Each region executes local HBase operations Results are returned to the controlling node and “spliced together” hence the “splice” in the company name Serialization Highly compressed storage format for table row Snappy compression Reduces network traffic! Join strategies: Nested Loop, SortMerge, Merge, Broadcast Rely on HBase co-processors: end-points and observers Performance Speed of the read and async write pipelines 20 msecs for read 30 msecs for write
• #34: This is big area of focus for us Based on MVCC “snapshot isolation” Lockless is key here Patent pending in this area Based on timestamps Transaction C will see changes from transaction A Transaction C won’t see changes from transaction B Here's an explanation of what is depicted in Figure 1: Transaction T1 bumps up the Qty for A by 10 twice, then commits at time t6. At the commit, A's Qty is 30, which is now visible to other transactions. Note, however, that T2 started at time t4 before T1's commit, so its value for A is still 10. Thus when it computes C = A+10, this results in 20. T3 starts at t7, as an overlap to T2, and attempts to update B just as T2 did, resulting in a write-write conflict. T3 rolls back, and attempts a reissue with T3'. This succeeds with the previously committed value from T2.