SAP Lambda Architecture Point of View
1 like2,229 views
The document discusses the concept of lambda architecture in the context of SAP's digital enterprise platform, emphasizing its three components: batch layer, speed layer, and serving layer. It highlights how SAP HANA integrates with lambda architecture to provide real-time data processing capabilities, ensuring flexibility, fault tolerance, and scalability for various data analytics needs. Additionally, the document mentions the role of Vora and the structural advantages offered by the architecture in adapting to changing business requirements.
SAP Lambda Architecture Point of View
- 1. SAP Lambda Architecture Point Of View SAP Digital Enterprise Platform Doug Martino January 2016 External
- 2. Inside cover
- 3. SAP External | 3 Introduction New Architectures for Big Data Whether MVC for user interfaces, or Spine and Leaf for data centers, new architecture patterns in our industry act as sort of historical markers of the effectiveness and acceptance of new technologies. Practical techniques push the bounds resulting in a shift. Application of distributed storage and streaming capabilities such as Kafka and of course Hadoop are shifting Big Data architectures from a layer cake concept, or North/South oriented approach to one which can be thought of as an East/West architectural concept. Recently popular is Lambda Architecture, this article presents an SAP HANA based rendering of the Lambda Architecture. Lambda Architecture Before discussing how SAP products can be used in a Lambda fashion, a brief overview of Lambda is in order. Attribution must go to Nathan Marz for his work. Typically consisting of three components, a batch layer, a speed layer, and a serving layer, Lambda rethinks how data flows through an analytics system. Data flows left to right in a multi speed architecture; with all data stored in the batch layer and the speed layer simultaneously, while the serving layer is used to combine and analyze that data as required. Advantages are fault tolerance, scalability, while at the same time offering multiple latencies to supporting a a variety of performance requirements. With Lambda some business questions are answered using the speed layer, others answered by the batch layer, and still others answered by combing both. As algorithms in the speed layer decide which data to keep, answer immediate questions such as which ad to display or how much discount to offer, it also stores an immutable copy of the data into the batch layer. In addition to acting as a replay source, data in the batch layer is suited for use cases such as long tail analytics, machine learning, or propensity to ‘act’ types of endeavors. Fault tolerance, resilience, ultimate scale, and back testing are derived from having the immutable copy of the original data for replay through the downstream components. The serving layer may persist a copy of the data to be used for additional downstream use of the data such as dashboards or feeds to another system. Shown here are several open source options for the components which make up a Lambda Architecture.
- 4. SAP External | 4 Traditional Lambda Architecture Lambda with SAP HANA Platform At SAP we are rapidly delivering value with our customers by employing these types of architecture patterns. In 2012 we started moving the HANA product towards the SAP Real-time data architecture where one already finds the notions of Lambda. Smart Data Streaming is the obvious choice for the real-time layer. Born on Wall Street, this component is capable of ingesting millions of rows per second into HANA. Complete with a rich set of API’s, stream and window compute constructs, and scale out capability, it can send the right data into HANA and all the data into HDFS as necessary. As the leading in-memory database platform, HANA meets the needs of the serving component of Lambda rather well. The data can be stored once, then transformed, aggregated, and calculated dynamically. Easy to understand in-memory compute engines such as graph, spatial, OLAP, and predictive analytics are available to manipulate the data where it is stored. Because the immutable data is kept in an append only columnar SQL database and the serving is done on the fly, our customer’s experience a valuable and unique combination of expressive, declarative programming capability on data which is managed in a SQL framework. This allows data access via everyday BI tools via standard ODBC or JDBC SQL as well as ODBO MDX. Based on Node.js, the HANA app server XS Advanced provides amongst other things noSQL access to the data via restful ODATA.
- 5. SAP External | 5 Often times requirements change such that new algorithms or new derivations of the immutable data are required. Because of the dynamic nature of materialization employed by the HANA core architecture, these changes can be made almost instantly. This flexibility changes the logistics of replay, allowing for rapid experimentation and moves into a production environment. In the case that a schema needs to be extended HANA does have schema flexibility whereby columns may be added to an existing table structure, leaving any existing code intact. The amount of flexibility offered by HANA once the immutable data has been stored is a fundamental differentiator of HANA. This flexibility offers reduced time to value and increased data agility. While the HANA platform is working extremely well for real-time applications, Lambda is complete by incorporating Vora. From an architecture standpoint, there are several aspects to add to the discussion. First is the aspect of scalability.
- 6. SAP External | 6 One of the beauties of the lambda is scaling each component dynamically and independently as latency requirements shift to meet changing business requirements. This notion of independent scaling allows appropriate resources allocation in a fit-for purpose model. Vora’s use of Hadoop moves the scale of the overall system up to multi petabyte range. More importantly, addressing long tail analytics and other big data compute problems are now possible. This leads to the next concept, which is one of throughput and algorithmic capability. Each of the real- time/batch/serving layers are best suited for a particular algorithm at a given throughout. All of them can run ‘out of band’ code; the question is at which level of concurrency and complexity. Rather than have JAVA JVM as the foundation of each, SAP have the appropriate engines with well-documented and supported extension capabilities. One may argue that open source provides the ultimate extension capability but only as a consuming organization is willing to merge code. In this case, SAP is providing high performance, fit for purpose engines to achieve the desired results more efficiently. This leaves us with the replay topic, an oft forgotten concept, especially in an operational batch context. Usually left for that – operations, it is often the case that re-running loads into a system to catch up with the real world, or to modify results based on new algorithms cause over provisioning of hardware in today's ETL and serving layers. By accounting for this capability up front, the opportunity to make transforms happen at the right time reduce the time and energy required to keep things in phase. HANA is a fantastic example of this given its powerful late materialization capability. Eventually elastic capabilities will further reduce replay workload stress by optimal allocation of compute resources with finer grain and lower latency. By employing lambda and other emerging big data architecture patterns, our customers remain well positioned as the state of the art continues forward.
- 7. www.sap.com/contactsap © 2016 SAP SE or an SAP affiliate company. All rights reserved. No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP SE or an SAP affiliate company. SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP SE (or an SAP affiliate company) in Germany and other countries. Please see http://www.sap.com/corporate- en/legal/copyright/index.epx#trademark for additional trademark information and notices. Some software products marketed by SAP SE and its distributors contain proprietary software components of other software vendors. National product specifications may vary. These materials are provided by SAP SE or an SAP affiliate company for informational purposes only, without representation or warranty of any kind, and SAP SE or its affiliated companies shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP SE or SAP affiliate company products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty. In particular, SAP SE or its affiliated companies have no obligation to pursue any course of business outlined in this document or any related presentation, or to develop or release any functionality mentioned therein. This document, or any related presentation, and SAP SE’s or its affiliated companies’ strategy and possible future developments, products, and/or platform directions and functionality are all subject to change and may be changed by SAP SE or its affiliated companies at any time for any reason without notice. The information in this document is not a commitment, promise, or legal obligation to deliver any material, code, or functionality. All forward- looking statements are subject to various risks and uncertainties that could cause actual results to differ materially from expectations. Readers are cautioned not to place undue reliance on these forward- looking statements, which speak only as of their dates, and they should not be relied upon in making purchasing decisions.