Hybrid Transactional/Analytics Processing: Beyond the Big Database Hype

Editor's Notes

• #8: We’re talking today about HTAP, and analytics in general, because the economic value of insight-driven transformation is undeniable Recent research shows really interesting numbers for what you might call insight-driven businesses From an ROI perspective, firms are seeing a %1300 ROI The majority of those who haven’t become fully insight-driven, about 74%, already have plans for introducing analytics at every corner for their business This is mainly due to the recognition that, having analytics, not only as means of differentiation, but as a fast innovation engine, to be twice as innovate and ourperform their peers.
• #9: Which brings us to the business value of analytics:…. Recent years have seen the need for more real-time analytics. In addition, mobile and IoT have given rise to a new generation of applications that are characterized by heavy ingest rates, i.e. they produce large amounts of data in a short time, as well as their need for more realtime analysis. Enterprises are pushing for more real-time analysis of their data to drive competitive advantage, and as such they need the ability to run analytics on their operational data as soon as possible. In order to become truly insight driven and innovate like amazon, this requires a departure from traditional analytics infrastructures.
• #10: Speaking of Amazon, one interesting use case we see quite often in retail is the ability to become an omni-channel retailer. Which requires what we call “hyperscale analytics” Let’s take a look
• #11: NOW FORTUNATELY, there has been advances in distributed computing that help us realize this vision. Thanks to the declining price of RAM and advancements in SSD storage, in-memory computing is becoming mainstrema. For those not familiar, in-memory computing means using RAM as the primary storage medium for business and analytics. There by eliminating any form of Disk I/O or network I/O latency, therefore operating at millisecond latencies at very high throughput. To understand HTAP, we first need to look into OLTP and OLAP systems and how they progressed over the years. Relational databases have been used for both transaction processing as well as analytics. However, OLTP and OLAP systems have very different characteristics. OLTP systems are identified by their individual record insert/delete/update statements, as well as point queries that benefit from indexes. One cannot think about OLTP systems without indexing support. OLAP systems, on the other hand, are updated in batches and usually require scans of the tables. Batch insertion into OLAP systems are an artifact of ETL (extract transform load) systems that consolidate and transform transactional data from OLTP systems into an OLAP environment for analysis.
• #12: If you read Gartner’s report on HTAP, you’ll see that most are actually classic database vendors. Now what does it mean to have an HTAP architecture? We see quite a lot people fall into the trap of thinking about HTAP as a acquiring a large vertically scalable database (like SAP HANA, Oracle) or others.
• #13: To understand HTAP, we first need to look into how databases evolved from the traditional OLTP vs OLAP world to the modern HTAP. As HTAP and realtime analytics became a necessity, we started seeing database vendors go outside their swim-lanes to introduce built-in LRU caching mechanisms (using In-Memory).
• #14: The reality of insight-driven transformation is that it requires a wide scope of analytics HTAP databases are simply focused on BI type of workloads (reporting queries)
• #15: At the same time, the last decade seen an explosion of many big data and in-memory computing technologies, driven by new generation applications. NoSQL or key-value stores, such as Voldemort, Cassandra, RocksDB, offer fast inserts and lookups, and very high scale out, but lack in their query capabilities, and offer only loose transactional guarantees (see Mohan’s tutorial[25]). There have been also many SQL-on-Hadoop offerings, including Hive [36], Big SQL[15], Impala[20], and Spark SQL[3], that provide analytics capabilities over large data sets, focusing on OLAP queries only, and lacking transaction support. Although all these systems support queries over text and CSV files, their focus have been on columnar storage formats
• #16: HTAP solutions today follow a variety of design practices. Now one of the major design decisions HTAP systems have to make is whether or not to use the same engine for both OLTP and OLAP requests. One approach is to decouple OLTP and an OLAP systems together for HTAP. It is up to the applications to maintain the hybrid architecture. The operational data in the OLTP system are aged to the OLAP system using standard ETL process. In fact, this is very common in the big data world, where applications use a fast key-value store like Cassandra for transactional workloads, and the operational data are groomed into Parquet or ORC files on HDFS for a SQL-on-Hadoop system for queries. BUT as a result, there is a lag between what data the OLAP system can query and what data the OLTP system sees.
• #20: all common API tap into other data stores on demand Data science is in high demand, but short supply – so the ability to leverage the know how, capability, and production readiness eliminates a lot of pain points.
• #21: First reason is speed: “In memory computing (IMC) … provides transformational opportunities. The execution of certain-types of hours-long batch processes can be squeezed into minutes or even seconds …Millions of events can be scanned in a matter of a few tens of millisecond to detect correlations and patterns pointing at emerging opportunities and threats "as things happen.” Besides that, in-memory data grids are proving to be a very mature containers for real-time application. While they started in finance and
• #25: Goal is to provide a unified environment where application developers and data scientists can collaborate. Data science by itself is an iterative activity which requires a lot of trial and error