Hadoop Infrastructure and SoftServe Experience by Vitaliy Bashun, Data Architect
Download as PPTX, PDF1 like703 views
This document discusses Hadoop infrastructure and SoftServe's experience with it. It provides an overview of various Hadoop components like HDFS, YARN, Pig, Hive, Sqoop and HBase. It also discusses popular Hadoop distributions and the Lambda architecture. Finally, it shares three case studies where SoftServe implemented Hadoop solutions for clients in log analysis, web analytics and an online analytics platform.
Hadoop Infrastructure and SoftServe Experience by Vitaliy Bashun, Data Architect
- 1. HADOOP INFRASTRUCTURE AND SOFTSERVE EXPERIENCE Pacemaker BigData, Lviv
- 2. Agenda •Business needs •Hadoop Infrastructure •Hadoop Distributives •SoftServe Experience
- 3. Presentation drivers • Hadoop competence development • Hadoop isn’t MapReduce only • Components for solution building • Case studies
- 4. Big Analytics Engineering Challenges Data Discovery Business Reporting Real Time Intelligence Business Users Intelligent AgentsConsumers How to achieve Low Latency for personalized customer experience in real-time? Data Scientists/ Analysts How to improve System Performance for Data Science/ Analytics team? How to implement Self-Service with high Data Quality over terabytes and petabytes?
- 6. A distributed file system • Files are split into blocks • Each block has 3 replicas minimum
- 7. A distribute computing framework
- 8. Apache YARN A resource manager (Yet Another Resource Manager)
- 9. A more complex resource management
- 10. An SQL interpreter for MapReduce
- 11. Apache Pig A script language to query HDFS
- 12. Real-Time Queries in Apache Hadoop
- 13. Runs Everywhere Engine for large scale data processing. Could be used with Java, Scala and Python
- 14. Apache Sqoop SQL to HADOOP – data load tool for RDBMS
- 16. Other Databases on top of Hadoop Column oriented Key-Value datastore Graph oriented Database
- 17. A distributed service for collecting, aggregating, transformation and moving large amount of log data
- 18. Distributed, real time computation service. Could be used for real time analytics, online machine learning, continuous computation, distributed RPC, ETL, and more
- 19. Apache Zookeeper Distributed Service for: • maintaining configuration information • naming • providing distributed synchronization • providing group services Service is fault tolerant: • Zookeeper cluster is called “ensemble” • There is one “leader” in an “ensemble” • If “leader” is down a new “leader” is elected with quorum
- 20. Distributed messaging service • Large amount of data • Scalable • Durable (messages are persisted on disc)
- 22. The last architecture trends
- 23. Lambda Architecture http://lambda-architecture.net/
- 24. SoftServe Lambda Architecture Accelerator • Lambda Architecture – is a highly scalable and reliable data processing architecture based on Twitter successful experience in Big Data and Analytics • Supports majority of use cases: Real-time analytics, data discovery and business reports • SoftServe’s pre-built Lambda Architecture stack accelerates customer’s Time to Market to 15-20+ man/month
- 25. 25 Business Goals: Build a centralized platform for log data analysis which collects data from ~270-300 Web Servers Provide Online Monitoring to answer the question: “What is going on with systems now?” Provide Retrospective Analytics – strategic management, capacity management/planning, route cause analysis, ad-hoc analysis Business Area: Retail industry. A leading travel site in a world Big Data Lab: Log Management
- 26. Log Data Analysis Platform Details 26 Key Facts: • ~270-300 Web Servers • Log Types: HTTPD Access logs, Error logs, Application Server Servlet, OS Service Logs • ~500K events per minute • 150GB of data per day Technologies: • Flume • Hadoop/HDFS, MapReduce • Hive, Impala • Oozie • Elasticsearch, Kibana • MicroStrategy Analytics platform
- 28. 28 Business Goals: Build in-house Analytics Platform for ROI measurement and performance analysis of every product and feature delivered by the e-commerce platform; Provide the ability to understand how end-users are interacting with service content, products, and features on sites; Do clickstream analysis; Perform A/B Testing Business Area: Retail. A platform for e-commerce and collecting feedbacks from customers Case Study #1: Clickstream for retail website
- 29. Architectural Decisions 29 ▪ Volume (45 TB) ▪ Sources (Semi-structured - JSON) ▪ Throughput (> 20K/sec) ▪ Latency (1 hour/real-time) ▪ Extensibility (Custom tags) ▪ Data Quality (Not critical) ▪ Reliability (24/7) ▪ Security (Multitenancy) ▪ Self-Service (Canned reports, Data science) ▪ Cost (The less the better ) ▪ Constraints (Public Cloud) Architecture Drivers: Technology Stack: Lambda Architecture • Apache Kafka • Apache Storm • Amazon S3 • Hadoop/HDFS, MapReduce (CDH 5) • HBase • Oozie, Zookeper • Cloudera Manager
- 31. 31 Business Goals: In-house Web Analytics Platform for Conversion Funnel Analysis, marketing campaign optimization, user behavior analytics (based on server logs analysis, page tagging, external data); Perform A/B Testing, platform feature usage analysis Business Area: Retail. The world's largest digital coupon marketplace. The company owns the largest coupon sites in the US, UK, Germany, Netherlands, France Case Study #2: Coupon Marketplace
- 32. Coupon Marketplace: Project Details 32 Project Facts: • 500 million visits a year • 25TB+ HP Vertica Data Warehouse • 50TB+ Hadoop Cluster • Near-Real time data visualization Technology Stack: • Hadoop Cluster (Amazon EMR) /Hive/Hue/MapReduce/Flume/Spark • HP Vertica, MySQL • Python • Tableau Major Activities: • Near-Real time data integration processes design and implementation • Hadoop cluster optimization • Data Warehouse re-design and optimization • Data Science algorithms design
- 33. Coupon Web Analytics Platform 33 Coupon Web-Site JS Libs Web Logs Operational databases Coupon Web-Site JS Libs Web Logs Operational databases 3rd Party API MPP Data Warehouse Cluster Raw Data Hadoop Cluster ETL Additional Data Stores Data Scientists BI/Marketing Team REST/SOAP
- 34. 34 Business Goals: Insights and optimization of all web, mobile, and social channels Optimization of recommendations for each visitor High return on online marketing investments Business Area: Web Analytics Platform by Fortune 100 company is a data storage and analytics on visitors' digital journeys Case Study #3: Online Analytics Platform
- 35. Online Analytics Platform Details 35 Key Facts: • Big Data > 1PB • 10+ GB per customer/day • 10+ Hadoop Clusters • 15+ Aster Data Clusters Technologies: • Hadoop/HBase/HiveQL • Aster Data • Oracle • Java/Flex
- 36. Solution Architecture 36 Customer Marketing Team Customer Web Server Environment Web Analytics Platform Web Analytics Data Offerings Business Rules Schedule Recommendation Rule Engine
Editor's Notes
- #26: Client Our client is a leading travel site in a world. Engagement Partnering with SoftServe, the combined teams developed an and implementation of Hadoop Cluster which collects log data from ~270-300 Web Servers including HTTPD Access and Error logs, as well as Application Server Servlet and OS Service Logs for further operational and retrospective analysis. Result The client has decreased their time to react on a issues which happens with web-servers as well as increased insight into ROI analysis for marketing campaigns which enabled company to increase number of visitors.
- #32: Clickstream Data: Google Analytics Site Catalyst, SaaS App from Adobe (prev. Omniture) Apache Web Logs Beacon JavaScript Library Financial Data: Data, provided by Affiliate Networks though API, FTP etc Marketing Data: Kenshoo: used as a platform to analyze the effectiveness of pay per click Google Ad campaigns. The Kenshoo Conversion Feed provides sales and commission data to measure ROI on campaigns
- #36: Tools & Technologies Extended List: SaaS, Hadoop/HDFS, Hadoop/Hbase, Aster Data, Java/Flex, J2EE, Java Script, Scape SSH/SFTP library, Velocity, Linux, Bash RDL, SQL, XSL Java, XML, Oracle database, JMS, Java Servlet, JDBC, JBoss, Flash RDL, Macromedia Flash.
- #37: Hadoop/HiveQL: Raw data about website users behavior Aggregation information for historical analytics Customized scheduled reports HBase: Online query for immediate data access: User geographical and demographics information Recent user purchase, search, unsubscribe activities