Achieve big data analytic platform with lambda architecture on cloud
Download as PPTX, PDF7 likes1,063 views
This document discusses achieving a big data analytic platform using the Lambda architecture on cloud infrastructure. It begins by explaining why moving to the cloud provides benefits like elastic scaling, reduced operational overhead, and increased focus on innovation. Common cloud services at Trend Micro like an analytic engine and cloud storage are then described. The document introduces the Lambda architecture and proposes a serving layer as a service. Key lessons learned from building big data solutions on AWS include the pros of unlimited scalability and easy disaster recovery compared to on-premises infrastructure.
![1.User invokes submitJob
2.Auth service check userās credential
3.AE knows user name and group
4.AE matches the job and
deliver it to target cluster
5.AE pull data from CS
6.Job run on target cluster
7.AE output result to CS
8. AE sends msg to SNS
Topic if user specified
Usecase#3 ā User submits job to target cluster to
run (1/4)
16
AE SaaSusers
submitJob
EMR
Cloud Storage
1.
2.
4.
3.
clusterCriteria:
[[āsched:adhocā,
āenv:prodā],
[āenv:prodā]]
group:SPN,
tag:
āsched:routineā,
āenv:prodā
validUser
is SPN
group
group:SPN,
tag:
āsched:adhocā,
āenv:prodā
5.
7.
6.
8.
Auth Service](https://image.slidesharecdn.com/scottmiao-achievebigdataanalyticplatformwithlambdaarchitectureoncloud-v01-160912055220/85/Achieve-big-data-analytic-platform-with-lambda-architecture-on-cloud-16-320.jpg)
![Usecase#3 ā User submits job to target cluster to
run (2/4)
ā¢ Sample payload of submitJob API
2
{
"clusterCriterias": [
{
"tags": [
"sechd:adhoc",
"env:prod"
]
},
{
"tags": [
"env:prod"
]
}
],
"commandArgs": "$inputPaths $outputPaths",
// see below](https://image.slidesharecdn.com/scottmiao-achievebigdataanalyticplatformwithlambdaarchitectureoncloud-v01-160912055220/85/Achieve-big-data-analytic-platform-with-lambda-architecture-on-cloud-17-320.jpg)
![Usecase#3 ā User submits job to target cluster to
run (3/4)
2
// see previous
"fileDependencies": "s3://path/to/my/main.sh,s3://path/to/my/test.pig",
"inputPaths": [
"cs://path/to/my/input/dataā
// or you can use metadata search for input data
// ācsq://first_entry_date:['2016-05-30T09:00:000Z','2016-05-30T09:01:000Z'}ā
],
"name": "SubmitJob_pig_cs_to_cs_csq",
"outputPaths": [
"cs://path/to/my/output/result"
],
"tags": [
"env:my-test"
],
"notifyTo" : "arn:aws:sns:us-east-1:123456789123:my-sns"
}](https://image.slidesharecdn.com/scottmiao-achievebigdataanalyticplatformwithlambdaarchitectureoncloud-v01-160912055220/85/Achieve-big-data-analytic-platform-with-lambda-architecture-on-cloud-18-320.jpg)
Achieve big data analytic platform with lambda architecture on cloud
- 1. Achieve Big Data Analytic Platform with Lambda Architecture on Cloud SPN Infra. , Trend Micro Scott Miao & SPN infra. 9/10/2016 1
- 2. Who am I ā¢ Scott Miao ā¢ RD, SPN, Trend Micro ā¢ Hadoop ecosystem about 6 years ā¢ AWS for BigData about 3 years ā¢ Expertise in HDFS/MR/HBase/AWS EMR ā¢ @takeshimiao ā¢ @slideshare
- 3. Agenda ā¢ Why go on Cloud ā¢ Common Cloud Services in Trend ā¢ Lambda Architecture on Cloud ā¢ Servicing Layer as-a Service ā¢ What we learned
- 4. Why go on Cloud
- 5. Data volume increases 1.5 ~ 2x every year Growth becomes 2x
- 6. Return of Investment ā¢ On traditional infra., we put a lot of efforts on services operation ā¢ On the Cloud, we can leverage its elasticities to automate our services ā¢ More focus on innovation !! Time Money Revenue Cost
- 7. Why AWS ?
- 8. AWS is a leader of IaaS platform https://www.gartner.com/doc/reprints?id=1-2G2O5FC&ct=150519&st=sbSource: Gartner (May 2015)
- 9. AWS Evaluation Cost acceptable Functionalities satisfied Performance satisfied
- 10. Common Cloud Services in Trend ANALYTIC ENGINE + CLOUD STORAGE
- 11. Common Services on the Cloud Cloud CI/CD Common Auth Analytic Engine Cloud Storage
- 12. AE + CS Analytic Engine ā¢Computation service for Trenders ā¢Based on AWS EMR ā¢Simple RESTful API calls ā¢Computing on demand ā¢Short live ā¢Long running ā¢No operation effort ā¢Pay by computing resources Cloud Storage ā¢Storage service for Trenders ā¢Based on AWS S3 ā¢Simple RESTful API calls ā¢Share data to all in one place ā¢Metadata search for files ā¢No operation effort ā¢Pay by storage size used
- 13. Analytic Engine is aā¦ A common Big Data computation service on Cloud (AWS) 2
- 14. Major Features in nutshell 14 AE CS submitJob EMR createCluster Input from ā¢ cs path ā¢ cs metadata search ā¢ Pig UDFs support Output to CS with meta data UIs Cost visibility (AWS Cost explor.) Client logs (SumoLogic) Cluster info. (Proxy Gateway) Visibility ā¢ Fully HA ā¢ Fully automated ā¢ Auto recovery
- 15. Support usecases 1. User creates a cluster 2. User can create multiple clusters as he/she need 3. User submits job to target cluster to run 4. AE delivers job to secondary cluster if target cluster down 5. Diff. group of users are not allowed to submit cluster(s) 6. Diff. group of users are not allowed to delete cluster 7. Only same group of users are allowed to delete cluster 8. User wants to know what their current cost is 9. User wants to troubleshoot his/her submitted job 10. User wants to observe his/her cluster status 2
- 16. 1.User invokes submitJob 2.Auth service check userās credential 3.AE knows user name and group 4.AE matches the job and deliver it to target cluster 5.AE pull data from CS 6.Job run on target cluster 7.AE output result to CS 8. AE sends msg to SNS Topic if user specified Usecase#3 ā User submits job to target cluster to run (1/4) 16 AE SaaSusers submitJob EMR Cloud Storage 1. 2. 4. 3. clusterCriteria: [[āsched:adhocā, āenv:prodā], [āenv:prodā]] group:SPN, tag: āsched:routineā, āenv:prodā validUser is SPN group group:SPN, tag: āsched:adhocā, āenv:prodā 5. 7. 6. 8. Auth Service
- 17. Usecase#3 ā User submits job to target cluster to run (2/4) ā¢ Sample payload of submitJob API 2 { "clusterCriterias": [ { "tags": [ "sechd:adhoc", "env:prod" ] }, { "tags": [ "env:prod" ] } ], "commandArgs": "$inputPaths $outputPaths", // see below
- 18. Usecase#3 ā User submits job to target cluster to run (3/4) 2 // see previous "fileDependencies": "s3://path/to/my/main.sh,s3://path/to/my/test.pig", "inputPaths": [ "cs://path/to/my/input/dataā // or you can use metadata search for input data // ācsq://first_entry_date:['2016-05-30T09:00:000Z','2016-05-30T09:01:000Z'}ā ], "name": "SubmitJob_pig_cs_to_cs_csq", "outputPaths": [ "cs://path/to/my/output/result" ], "tags": [ "env:my-test" ], "notifyTo" : "arn:aws:sns:us-east-1:123456789123:my-sns" }
- 19. Usecase#3 ā User submits job to target cluster to run (4/4) ā¢ All existing job types used in on-premise are supported ā¢ Pure MR ā¢ Pig and UDFs ā¢ Hadoop streaming ā Python, Ruby, etc 2
- 20. Usecase#8 ā User wants to know what their current cost is (1/2) 20 ā¢ Billing & Cost management -> Cost Explorer -> Launch Cost Explorer ā¢ Filtered by ā¢ tags: āsys = aeā and ācomp = emrā and āother = <your-cluster-name>ā ā¢ Group by Service
- 21. Usecase#8 ā User wants to know what their current cost is (2/2) - Billing and Cost Analysis ā¢ Attach tags to your AWS resources 21 Tag Key Tag Value (sample) Description name aesaas-s-11-api *optional* for AWS cost explorer stack aesaas-s-11 *optional* for AWS cost explorer service aesaas *optional* for AWS cost explorer owner spn *required* the bill is under whose budget env prod|stg|dev *required* environment type sys ae *required* the system name comp api-server|emr *required* the subcomponent name other spn-stg *optional* an optional tag that free for other usage.
- 22. Why we use AE instead of EMR directly ? ā¢ Abstraction ā¢ Avoid locked-in ā¢ Hide details impl. behind the scene ā¢ AWS EMR was not design for long running jobs ā¢ >= AMI-3.1.1 ā 256 ACTIVE or PENDING jobs (STEPs) ā¢ < AMI-3.1.1 ā 256 jobs in total ā¢ Better integrated with other common services ā¢ Keep our hands off from AWS native codes ā¢ Centralized Authentication & Authorization ā¢ Leverage our internal LDAP server ā¢ No AWS tokens for user http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/AddingStepstoaJobFlow.html
- 23. Lambda Architecture on Cloud
- 24. Next Phase Cloud Infra. AE-v1.0 AE + CS (v1.1~) Lambda arch. 24
- 25. What is Lambda (Ī») Architecture 2
- 26. Data Ingestion Batch Layer Master Dataset Speed Layer Streaming Processing Batch Processing Batch View Merged View Real-Time View Serving Layer Data Access API Batch Layer as-a Service Serving Layer as-a Service A data-processing architecture designed to handle massive quantities of data by taking advantage of both batch- and stream- processing methods https://en.wikipedia.org/wiki/Lambda_architecture
- 27. Servicing Layer as-a Service METADATA STORE
- 28. Goals Help everyone to easily access metadata shared by several teams ā¢ Access data in one place ā¢ Avoid storage duplication ā¢ Share immediately to all ā¢ Provide unified intelligence Common metadata storage for several services ā¢ Abstract to hide infra & ops ā¢ Customize for different needs 28 (on aws)
- 29. Usecase ā¢ Store all threat entities into one place from new born ā Every team can leverage contributions from other teams at very early stage 2
- 30. Features 30 Metadata Store Service Random Writes Bulk Writes Sync Query Async Query Automatic Provision Customizable Schema Unified Intelligence Threat Monitor
- 31. Borrow idea from Star Schema ā¢ A schema design widely used in data warehousing 31 Historical data ā measurements or metrics for a specific event Descriptive attributes ā characteristics to describe and select the fact data
- 32. Basic Idea ā¢ Refer to Star Schema design ā Fact table ā¢ Put all records into this table (Single Source of Truth) ā¢ Affordable for random and bulk load of writes ā¢ Fast random reads by rowkey ā Dimension table ā¢ Fast and flexible info. discovery ā¢ Get rowkey of records stored in Fact table ā¢ Then retrieve records by rowkey
- 33. Reference Implementation ā Part 1 ā¢ This Star Schema concept can be fulfill by different impl. ā¢ A famous one is HBase + Indexer + Solr http://www.hadoopsphere.com/2013/11/the-evolving-hbase-ecosystem.html https://community.hortonworks.com/articles/1181/hbase-indexing-to-solr-with-hdp-search-in-hdp-23.html
- 34. Reference Implementation ā Part 2 2 http://www.slideshare.net/AmazonWebServices/bdt310-big-data-architectural-patterns-and- best-practices-on-aws #p57
- 35. Dimension Tables Schema Dimension Tables Engine: Elastic Search Dimension Tables Engine: MySQL (RDS) Dimension Tables Engine: Dynamo DB Propagate data to dimension storage 35 Fact Tables (Dynamo DB) Propagato r Dynamo DB Streams Propagation Rules Random Writes Bulk Writes (Eventually Consistent)
- 39. What we learned FROM BIG DATA ON CLOUD
- 40. Pros & Cons Aspects IDC AWS Data Capacity Limited by physical rack space No limitation in seasonable amount Computation Capacity Limited by physical rack space No limitation in seasonable amount DevOps Hard, due to on physical machine/ VM farm Easy, due to code is everything (CI/CD) Scalability Hard, due to on physical machine/ VM farm Easy, relied on ELB, Autoscaling group from AWS
- 41. Pros & Cons Aspects IDC AWS Disaster Recovery Hard, due to on physical machine/ VM farm Easy, due to code is everything Data Location Limited due to IDC location Various and easy due to multiple regions of AWS Cost Implied in Total Cost of Ownership Acceptable cost with Cost Conscious Design Something more detailsā¦
- 43. We Are Hiring !
- 44. Backup
- 45. AE SaaS Architecture Design
- 46. IDC High Level Architecture Design 46 AZb AE API servers RDS Private ELB AZa AZb AZc AE API servers RDS services services services peering HTTPS EMR EMR Cross-account S3 buckets Time based Auto Scaling group worker s worker sMulti-AZs Auto Scaling group Time based Auto Scaling group Eureka Eureka VPN HTTPS/HTTP Basic Cloud StorageInternet HTTPS/HTTP Basic Amazon SNS Oregon (us-west-2) SJC1 SPN VPC CI slave Splunk forwarde r peering VPN Splunk peering
- 47. What is Netflix Genie ā¢ A practice from Netflix ā¢ A hadoop client to submit jobs to EMR ā¢ Flexible data model design to adopt diff kind of cluster ā¢ Flexible Job/cluster matching design (based on tags) ā¢ Cloud characteristics built-in design ā e.g. auto-scaling, load-balance, etc ā¢ Itās goal is plain & simple ā¢ We use it as an internal component 47https://github.com/Netflix/genie/wiki
- 48. What is Netflix Eureka ā¢ Is a RESTful service ā¢ Built by Netflix ā¢ A critical component for Genie to do Load Balance and failover 48 Genie API API API
- 49. 9/12/2016 Confidential | Copyright 2016 TrendMicro Inc. 49 AWS EMR (Elastic MapReduce)
- 52. 2
- 53. 9/12/2016 Confidential | Copyright 2016 TrendMicro Inc. 53 Lessons Learned on AWS details
- 54. Different types of Auto-scaling group 54 Service Auto Scaling Group Type Features Provision Deploy/Conf ig Method OpsWorks 24/7 ā¢manual creation/deletion ā¢configure one instance for one AZ ā¢ CloudFormation ā¢ AWS::OpsWorks::In stance. AutoScalingType chef recipe time-based ā¢can specify time slot(s) based on hour unit, on everyday or any day in week ā¢configure one instance for one AZ load-based ā¢can specify CPU/MEM/workload avg. based on an OPS layer ā¢UP: when to increase instances ā¢Down: when to decrease instances ā¢No max./min. # of instances setting ā¢configure one instance for one AZ EC2 ā¢can set max./min. for # of instance ā¢Multi-AZs support ā¢ CloudFormation ā¢ AWS::AutoScaling:: AutoScalingGroup ā¢ AWS::AutoScaling:: LaunchConfigurati on user-data
- 55. ELB + Auto-Scaling Group ā¢ ELB ā Health Check ā¢ Determining the route for coming requests ā¢ Auto-Scaling Groups ā Monitoring EC2 instance by CloudWatch ā If EC2 abnormal, then terminate and start a new one ā¢ ELB + Auto-Scaling Group ā Auto attach/detach EC2 instance(s) to ELB if Auto-Scaling Group launch/terminate EC2 http://docs.aws.amazon.com/autoscaling/latest/userguide/autoscaling-load-balancer.html
- 56. Auto Recovery based on Monit ā¢ OpsWorks already use Monit for Auto Recovery ā Leverage the Monit on EC2 ā Have practices in on-premise 2 AZ1 AZ2 API server API server https://mmonit.com/monit/ Auto Scaling group ā¢ Instance check by CloudWatch ā¢ Process check by Monit ā¢ No process ā restart process ā¢ Process health check failed ā terminate EC2 ā¢ Terminate EC2 !Auto Scaling group launch new EC2
Editor's Notes
- #5: Whatās our goal
- #11: Whatās our goal
- #32: https://en.wikipedia.org/wiki/Star_schema
- #35: Do not use CloudSearch Aurora is good !!
- #36: MongoDB is excluded based on Chienās suggestion.
- #38: TAO: The Associations and Objects, a distributed Graph data store Unicorn: Graph-aware search system Graph API: interface to users