Big Data Analytics with Amazon Web Services
0 likes472 views
This document provides an overview of a seminar on Big Data Analytics with Amazon Web Services. It discusses how AWS enables cost-effective data generation, collection, storage, analytics and sharing. It describes AWS services like EC2, S3, DynamoDB, EMR and the AWS Marketplace which provide the infrastructure and tools needed for distributed data analytics. It also presents a success story of Brightcove using AWS to power its online video platform.
Big Data Analytics with Amazon Web Services
- 1. Big Data Analytics with Amazon Web Services Dr. Matt Wood An Online Seminar. Tuesday 16th October.
- 2. Hello, and thank you.
- 3. Big Data Analytics An introduction
- 4. Big Data Analytics An introduction The story of analytics on AWS
- 5. Big Data Analytics An introduction The story of analytics on AWS AWS Marketplace
- 6. Big Data Analytics An introduction The story of analytics on AWS AWS Marketplace Success story: Brightcove
- 8. Data for competitive advantage.
- 9. Using data Customer segmentation, financial modeling, system analysis, line-of-sight, business intelligence.
- 10. Generation Collection & storage Analytics & computation Collaboration & sharing
- 11. Cost of data generation is falling.
- 12. lower cost, increased throughput Generation Collection & storage Analytics & computation Collaboration & sharing
- 13. Generation HIGHLY CONSTRAINED Collection & storage Analytics & computation Collaboration & sharing
- 14. Very high barrier to turning data into information.
- 15. Move from a data generation challenge to analytics challenge.
- 16. Enter the Cloud.
- 19. Move to a distributed data approach.
- 20. Maturation of two things.
- 21. Software for distributed storage and analysis Maturation of two things.
- 22. Software for distributed storage and analysis Maturation of two things. Infrastructure for distributed storage and analysis
- 23. Software Frameworks for data-intensive workloads. Distributed by design.
- 24. Infrastructure Platform for data-intensive workloads. Distributed by design.
- 26. Generation HIGHLY CONSTRAINED Collection & storage Analytics & computation Collaboration & sharing
- 27. Generation Collection & storage Analytics & computation Collaboration & sharing
- 28. Lower the barrier to entry.
- 29. Accelerate time to market and increase agility.
- 30. Enable new business opportunities.
- 31. Washington Post Pinterest NASA
- 32. “AWS enables Pfizer to explore difficult or deep scientific questions in a timely, scalable manner and helps us make better decisions more quickly” Michael Miller, Pfizer
- 33. 2 THE STORY OF ANALYTICS
- 34. EC2 Utility computing. 6 years young.
- 35. Scale out systems Embarrassingly parallel problems. Queue based distribution. Small, medium and high scale.
- 39. Cost optimization. EC2 Utility computing. 6 years young.
- 40. Achieving economies of scale 100% Time
- 41. Achieving economies of scale 100% Reserved capacity Time
- 42. Achieving economies of scale 100% On-demand Reserved capacity Time
- 43. Achieving economies of scale UNUSED CAPACITY 100% On-demand Reserved capacity Time
- 44. Spot Instances Bid on unused EC2 capacity. Very large discount. Perfect for batch runs. Balance cost and scale.
- 45. <$1000 per hour
- 46. Map/reduce Pattern for distributed computing. Software frameworks such as Hadoop. Write two functions. Scale up.
- 47. Map/reduce Pattern for distributed computing. Software frameworks such as Hadoop. Write two functions. Scale up. Complex cluster configuration and management.
- 48. Amazon Elastic MapReduce Managed Hadoop clusters. Easy to provision and monitor. Write two functions. Scale up. Optimized for S3 access.
- 49. S3 Input data
- 50. S3 Input data Code Elastic MapReduce
- 51. S3 Input data Code Elastic Name MapReduce node
- 52. S3 Input data Code Elastic Name MapReduce node Elastic cluster
- 53. S3 Input data Code Elastic Name MapReduce node HDFS Elastic cluster
- 54. S3 Input data Code Elastic Name MapReduce node Queries HDFS + BI Via JDBC, Pig, Hive Elastic cluster
- 55. S3 Input data Code Elastic Name Output MapReduce node S3 + SimpleDB Queries HDFS + BI Via JDBC, Pig, Hive Elastic cluster
- 56. S3 Input data Output S3 + SimpleDB
- 67. Performance
- 69. Cluster Compute Intel Xeon E5-2670 10 gig E non-blocking network 60.5 Gb Placement groupings
- 70. Cluster Compute Intel Xeon E5-2670 10 gig E non-blocking network 60.5 Gb Placement groupings + GPU enabled instances
- 72. IO performance Performance Compute performance
- 74. DynamoDB Predictable, consistent performance Unlimited storage Single digit millisecond latencies No schema for unstructured data Backed on solid state drives
- 75. ...and SSDs for all. New Hi1 storage instances.
- 76. hi1.4xlarge 2 x 1Tb SSDs 10 GigE network HVM: 90k IOPS read, 9k to 75k write PV: 120k IOPS read, 10k to 85k write
- 77. “The hi1.4xlarge configuration is about half the system cost for the same throughput.” Netflix http://techblog.netflix.com/2012/07/benchmarking-high-performance-io-with.html
- 78. Generation Collection & storage Analytics & computation Collaboration & sharing
- 79. Performance + ease of use
- 81. Extend platform with partners
- 82. Innovate on behalf of customers
- 83. Remove undifferentiated heavy lifting
- 87. Generation Collection & storage Analytics & computation Collaboration & sharing
- 88. Generation Collection & storage Analytics & computation Collaboration & sharing
- 89. Collection & storage Acunu Reflex Apache Cassandra NoSQL database MongoDB With and without EBS RAID storage Couchbase Community and Enterprise editions ScaleArc MySQL load balancing
- 95. Generation Collection & storage Analytics & computation Collaboration & sharing
- 96. Generation Collection & storage Analytics & computation Collaboration & sharing
- 97. Analytics & computation KarmaSphere Analytics for Amazon Elastic MapReduce MapR M5 Hadoop Distribution Metamarkets Event based data processing
- 98. Analytics & computation StackIQ Rocks+ HPC clusters with MPI, Grid Engine Univa Grid Engine One click cluster deployment Quantivo Data association analytics
- 99. Generation Collection & storage Analytics & computation Collaboration & sharing
- 100. Generation Collection & storage Analytics & computation Collaboration & sharing
- 101. Collaboration & sharing Aspera Faspex 20 Mbps data transfer
- 102. 4 SUCCESS STORY