Bringing the Power of Big Data Computation to Salesforce
3 likes1,007 views
The document discusses how Model N leverages big data computation within Salesforce to enhance revenue management for various industries. It emphasizes the importance of metadata integration and the use of big data technologies to gain insights and drive decision-making in business processes. Key topics include building flexible cloud applications, data-driven strategies, and the collaborative use of data pipelines and big objects to support large-scale data management.
Bringing the Power of Big Data Computation to Salesforce
- 1. Bringing the Power of Big Data Computation to Salesforce Arun Bhat Chief Architect – Model N Inc. [email protected] @parunbhat Krishna Shekhram Software Architect – Model N Inc. [email protected] @kshekhram
- 2. Speaker Introduction Little bit about us
- 3. • Model N is the leading provider of Revenue Management solutions for the life sciences and technology industries. • The company helps customers maximize revenues, drive growth and reduce compliance risk by transforming the revenue lifecycle from inefficient disjointed operation into a strategic end to end process. Why do we care about big data Model N – The Pioneer in Revenue Management Founded in 1999$120+B Revenue under management 2+M Sales lines processed daily 100+ Companies maximizing revenue with Model N 50,000+ Sales, Sales Ops, FAE’s, Finance, Marketing, Manufacturing reps and Distributor users 100+ Countries where Model N Revenue Management is used 1,000+ Distributors in 50 Countries
- 4. Arun Bhat Chief Architect, Revvy Products 15 years in Model N 19 years in Software Industry Led Architecture of Model N products Responsible for architecture of multi-tenant Revvy products on Salesforce Passionate about technology but likes to read comics Krishna Shekhram Architect, Revvy Products 6 years in Model N 14 years in Software Industry Architected Model N Analytics Products Lead for Revvy Big Data Architecture Enjoys exploring new technologies. Love to watch documentaries to learn more about world. Model N – The Pioneer in Revenue Management
- 5. Overview What we will be discussing over this talk
- 6. Leveraging Salesforce Computing using Big Data Metadata as a common fabric Integrating into a Cohesive Architecture Building a Data Driven Application Demo Data Pipeline and BigObjects Summary Agenda Big Data
- 7. Leveraging Salesforce To build flexible cloud applications
- 8. Availability Deployment Elasticity Customization Security Upgradeability Integration Device Independence Multi Tenancy Metadata Cloud Computing Force.com Stack Enabling Technology Leveraging Salesforce Power User Interface Logic Integration Database Infrastructure DeveloperTools
- 9. Computing using Big Data Realize valuable insights, actions and faster decisions from your data at scale
- 10. Source: logs, social media, mobile, IOT, POS Format: structured, text, picture, video, binary, document Speed: real-time streams, transactions, batch upload Rapid Ingestion Bigger Storage Faster Processing Quicker Retrieval Better Visualization Hidden insights discovery Facts based decision making Business process automation Ecosystem engagement Growth & monetization of data Data Explosion Technology Evolution Business Opportunities Why “Big Data” is a Big Deal Competitive advantage for today, Survival for tomorrow
- 11. Big data technology is going through innovation spurt Big Data Technology Landscape
- 12. Components • HDFS, Map/Reduce, YARN • Provides fault tolerant and scalable cluster HDFS as storage • Supports variety of data formats • Metadata driven schema evolution YARN as cluster manager • Supports Security, Resource Isolation, Multi-tenancy • Highly available and elastic scaling Components • Spark Core, SQL, MLib, Streaming, GraphX • Can run in variety of clusters (YARN, Mesos, Standalone) Data Access • Data access from HDFS, S3, Cassandra, HBase, JDBC, Streaming source like Kafka • Supports multiple formats like Parquet, json, csv, etc. Compute • General purpose low latency compute engine • Batch, Interactive, Query, Predictive, Graph and Stream processing Hadoop and Spark Advantage Data driven, flexible, multi-tenant applications at scale Hadoop Spark
- 14. Sales Data Sales Metadata URL: /tx/sales/Sales.parquet Columns: Sale ID: ID Customer : Relationship (Customer) Product : Relationship (Product) Invoice Date: Date Qty : Integer Price : Decimal Metadata Example Metadata describes data Sale ID Customer Product Invoice Date Qty Price Product ID Product # BU Customer ID Name Type Customer Sales Product
- 15. Calculation Unit Calculation Model Flexibility & Extensibility Key for multi tenant cloud applications Calc Op Input Dataset Output Dataset Define Metadata Define Metadata Input Dataset Input Dataset Input Dataset Output Dataset Output Dataset Output Dataset Calculation Model Metadata MetadataConfiguration
- 16. • Metadata Capture & Synchronization • Define all dataset as objects in Salesforce to capture metadata. Example: Sales, Inventory, Order • Load actual data in HDFS • Synchronize metadata on change • Master Data Sync • Synchronize the master data from SFDC to HDFS. Example: Accounts, Catalog • HDFS Schema using metadata • Use HDFS file formats which supports schema evolution(e.g. Parquet, Avro) • Use the dataset metadata to read/write HDFS file • Configure Calculation • Define Variability in calculation as configuration using Salesforce custom object Leverage Salesforce to capture metadata Flexibility & Extensibility using metadata
- 17. Integration Building a cohesive architecture
- 18. • Exposes all the REST APIs needed for application. • Stores application and object metadata • Provides support for multi-tenancy, error handling and recovery • Provides secure API for • Metadata synchronization • Data Loads • Batch calculation • Querying the aggregated results • Real time calculation/prediction Exposes big data computation as service Web Service as Middleware Compute Cluster Cluster Web Service
- 19. • Abstracts out complexity of big data technology • Translates business specific service calls to calculation jobs • Uses metadata to build calculation model • Handles connection to cluster • Manages multi-tenancy context to submit jobs to cluster • Interacts with Various cluster components • HDFS • YARN • Spark Acts as client for cluster Web Service as Middleware Compute Cluster Cluster Web Service
- 20. Building a Data Driven Application Getting best of both world to realize business value
- 21. • Unified transactional and analytics application • Provides real time insights from data in business context • Calculates KPIs and processes data for business • Evaluate performance against goal based on data • Combines intelligence with Action • Facilitate business process automation • Learn from data to support fast and accurate decision Key Concepts What is a data driven application
- 22. Contextual Discovery Measuring KPIs and triggering workflow actions, alerts or notifications based on KPI. Claim processing Fraud detection Processing large amount of data and running business calculation on it to generate results critical for business operation. Tax report generation Stock portfolio valuation Intelligent decisions and actions based on learning from data. Prediction, Optimization, Anomaly detection, AI, Recommendation. Google Now, Price Optimization Business Process Automation Data Processing Decision Intelligence Interactive dashboards and analysis in the transactional application business context. Account performance dashboard in CRM application Data Driven Application Examples
- 23. Guideline for building data driven application Reference Architecture Metadata Manager Common Library Data Manager Job Manager Config Manager Application Account Catalog Opportunity Sales Segment Big Data Cluster Web App Middleware Cluster Client Metadata Service Data Service Application Service Data Storage Calculation Runtime
- 25. User enters segment definition See Sales metadata in Salesforce Show Sales lines loaded in Hadoop Trigger segmentation from Salesforce Show dashboards with segmented customers in Salesforce Segmenting customers based on revenue Demo Overview
- 26. Data Pipelines BigObjects Collaborating with Salesforce on the big data roadmap
- 27. Data Pipelines Brings batch processing using Hadoop to the Salesforce Platform Apache Pig for data flow control and evaluation BigObjects Storage of large amounts of data Data Pipelines and BigObjects (Pilot)
- 28. Features that can be leveraged BigObjects to store POS, Order and line items Apache Pig Script and Hadoop through the Data Pipeline API Features that need to be incorporated Support Data Pipeline API through Apex (instead of the Metadata API) Support for low latency jobs e.g. Spark (as compared to batch processing) To get big data computation in Salesforce Collaborate with Salesforce on big data roadmap
- 29. Reference Architecture Metadata Manager Common Library Data Manager Job Manager Config Manager Application Account Catalog Opportunity Sales Segment Big Data Cluster Web App Middleware Cluster Client Metadata Service Data Service Application Service Data Storage Calculation Runtime Data Pipeline Bulk SOQL Apex SObjects BigObjects Files SObjects BigObjects Files SObjects BigObjects Files SObjects BigObjects Files Job Manager Config Manager
- 31. • How to leverage Salesforce to build flexible cloud applications • How to use big data computation to realize valuable insights, actions and faster decisions from your data at scale • How to fuse Salesforce and Big Data technologies together using metadata and integrations • How to unlock your business potential using data driven application • How Salesforce and Big Data technologies can coexist well What we learnt Summary
- 32. Thank you