Stream your Operational Data with Apache Spark & Kafka into Hadoop using Couchbase

Editor's Notes

• #5: KEY POINT: COUCHBASE HAS YOU COVERED FOR YOUR GENERAL PURPOSE DB NEEDS. FROM CACHING TO KV STORE, TO JSON DOCUMENT STORE, TO MOBILE APPS. NO OTHER NOSQL DB VENDOR HAS THIS BREADTH AND DEPTH OF TECHNOLOGY The purpose of this slide is to discuss the high level concepts of Couchbase, and if the SE wants to discuss what parts of Couchbase make up each concept. It is not to go over specific technologies like N1QL, ODBC, etc
• #6: KEY POINT: Frame the conversation
• #7: KEY POINT: GROUND THE USER IN HOW OBJECTs RELATE TO BUCKETS AND THOSE ARE SPREAD ACROSS THE CLUSTER; AS WELL AS HOW THINGS IN THE CLUSTER ARE STACKED PHYSICALLY. Talk to the audience about how documents move in the application at a high level and the relation between data buckets and how they are spread evenly across the cluster. Remember that vBuckets are not in this diagram, but that is on purpose. That comes later and might confuse people at this point. Going over this slide now sets you up for the vBucket discussion later in the presentation. Convey what a bucket is. That it is a logical key space, with its own set of server resources, queues, etc. Make sure to stress that one can have multiple buckets, but to not just create them like you would tables or schemas in a relational database An example of when to split data into different buckets is using Views across different object types. For example, if you have JSON documents and base64 encoded XML documents in the same bucket, a view or index will have to interact with that object even though it will never need it. So it would be better to put the XML into another bucket so the views and indexes are only looking at the JSON data they actually will be indexing.
• #8: Application has single logical connection to cluster (client object) Multiple nodes added or removed at once One-click operation Incremental movement of active and replica vbuckets and data Client library updated via cluster map Fully online operation, no downtime or loss of performance
• #13: Use cases are totally different – Spark is an execution engine, not a database. A prime use case for Hadoop is as a low cost data warehouse, which is not a good use case for Couchbase or Spark Latency - Everyone says real time, but what do mean? For an operational system, this means: Extremely fast (in-memory) reads Extremely fast (log append) writes For Couchbase, complete millions of ops / second (these are gets / sets) at latencies of under 1ms, compare LinkedIn figures from Jerry Franz’s session Tuned to LinkedIn’s specific workload: 75% writes (sets + incr) / 25% reads – 13 byte values, 25 byte keys on average 2.5 billion items (+ 1 replica) 600 Gbytes of RAM /  3 Tbytes of disk in use on average Average store latency ~ 0.4 milliseconds 99th percentile store latency ~ 2.5 milliseconds Average get latency ~ 0.8 milliseconds 99th percentile get latency ~ 8 milliseconds In general, Spark is just better at Hadoop’s core use cases than Hadoop (note, I’m not talking about HDFS) Spark is much better for highly iterative algorithms and interactive queries – which is important, given that the majority of jobs work on 100GB or less of data (small big data) Spark scale – less than Map Reduce based solutions on Hadoop, but that’s OK – “[T]he majority of real-world analytics jobs process less than 100GB of input, but popular infrastructures such as Hadoop/MapReduce were originally designed for petascale processing.” http://www.msr-waypoint.com/pubs/204499/a20-appuswamy.pdf This is especially convenient for people with development background who like to run "stuff" (ad-hoc queries) on data in hadoop/hdfs. This remove the need to know about the underlying hadoop layer and just think of it as data.
• #15: - Improve the animation for data handling – refresh existing sales deck and deep dives.
• #16: Using Couchbase as the high performance, low latency, scalable data store to support personalized interactions Couchbase, as real-time operational database may be generating real-time data to feed into both the batch and real-time layers In some use cases, Couchbase is used to perform real-time processing and analytics – M/R views Some customers are using Couchbase as the data store to for stream processing Email from Michael on streaming data from CB to Spark via Kafka: well, Kafka is one way but it also uses DCP under the covers. I actually need to make some changes to the DCP implementation in the java client and my plan is to have DCP support in dp2 (a month later or so). So once we are GA, there will be a 100% way to stream data directly into a DStream (spark streaming). And of course you can easily implement simple polling of let’s say a view, and grab the full docs that match for example a time interval. 
• #17: Work people do in these systems - Training ML models ETL / Data wrangling Aggregations Reporting / BI Kafka is a data multiplexer – some people are still going to want to do this, but it’s designed for higher latency applications with a known high complexity (e.g. ebay – many different consumers for information) Traditional data warehouse – definitely will be a different programming language – how do you make sense of the data feed? You get into the problems that making changes on one side introduces tons of complexity on the other Downsides – maturity is not 100% on the Spark side, still in active development in the Couchbase side KV / N1QL
• #18: The data generated by users is published to Apache Kafka. Next, it’s pulled into Apache Storm for real time analysis and processing as well as into Hadoop. Finally, Storm writes the data to Couchbase Server for real-time access by LivePerson agents while the data in Hadoop is eventually accessed via HP Vertica and MicroStrategy for offline business intelligence and analysis.
• #23: High throughput distributed messaging system Massively scalable Simple, elegant – basically a giant distributed ordered commit log Like a manifold for your data processing – decouples the data producers from the data consumers, allows for asynchronicity Automatic recovery from failures Use cases: Offline & online processing of activities, events, monitoring, and sensor data Couchbase can be either a producer or a consumer in Kafka terms. Couchbase as the Master Database React to changes happening in the bucket by updating data somewhere else Triggers/Event Handling Handle events like Deletions/Expirations externally (E.G. Expiration & replicated session tokens) Real-time Data Integration Extract from Couchbase , transform and load data in real time Real-time Data Processing Extract from a Bucket, process in real-time and load to another Bucket
• #25: Fully transparent cluster and bucket management, including direct access if needed
• #26: Fully transparent cluster and bucket management, including direct access if needed
• #29: Spark Core Automatic Cluster and Resource Management Creating and Persisting RDDs Java, Scala APIs Spark SQL Easy JSON handling and querying Tight N1QL Integration Spark Streaming Persisting DStreams DCP source (experimental)
• #30: 29
• #31: 30
• #34: 33
• #35: Analyze other
• #37: Analyze other
• #38: Fully transparent cluster and bucket management, including direct access if needed
• #39: Fully transparent cluster and bucket management, including direct access if needed
• #42: Spark Application = a user program built on Spark. It includes the driver program and the executors on the cluster. Application jar – a jar containing the users application along with its dependencies. Driver program – The process running the main() function of the application and creating the SparkContext Executor - A process launched for an application on a worker node, that runs tasks and keeps data in memory or disk storage across them. Each application has its own executors.