Immersion Day - Como construir seu Data Lake em dias na AWS
0 likes•178 views
The document discusses AWS Lake Formation, a service that allows customers to easily build and manage data lakes. It allows users to load data from various sources into storage like S3, cleanse and catalog the data, and enforce security policies across services. This simplifies and accelerates building data lakes from what typically takes months to just days. Lake Formation also provides tools for data preparation, analytics, machine learning and real-time processing on the data lake.
Immersion Day - Como construir seu Data Lake em dias na AWS
- 1. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
- 2. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
- 3. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Data every 5 years There is more data than people think 15 years live for Data platforms need to 1,000x scale >10x grows
- 4. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. There are more people accessing data And more requirements for making data available Data Scientists Analysts Business Users Applications Secure Real time Flexible Scalable
- 5. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. A data lake is a centralized repository that allows you to store all your structured and unstructured data at any scale
- 6. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Why data lakes? Data Lakes provide: Relational and non-relational data Scale-out to EBs Diverse set of analytics and machine learning tools Work on data without any data movement Designed for low cost storage and analytics OLTP ERP CRM LOB Data Warehouse Business Intelligence Data Lake 100110000100101011100 101010111001010100001 011111011010 0011110010110010110 0100011000010 Devices Web Sensors Social Catalog Machine Learning DW Queries Big data processing Interactive Real-time
- 7. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon S3 | AWS Glue Any analytic workload, any scale, at the lowest possible cost AWS Direct Connect AWS Snowball AWS Snowmobile AWS Database Migration Service AWS IoT Core Amazon Kinesis Data Firehose Amazon Kinesis Data Streams Amazon Kinesis Video Streams On-premises Data Movement Amazon SageMaker AWS Deep Learning AMIs Amazon Rekognition Amazon Lex AWS DeepLens Amazon Comprehend Amazon Translate Amazon Transcribe Amazon Polly Amazon Athena Amazon EMR Amazon Redshift Amazon Elasticsearch Service Amazon Kinesis Amazon QuickSight Analytics Machine Learning Real-time Data Movement Data Lake
- 8. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. More data lakes & analytics on AWS than anywhere else
- 9. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Typical steps of building a data lake Setup storage1 Move data2 Cleanse, prep, and catalog data 3 Configure and enforce security and compliance policies 4 Make data available for analytics 5
- 10. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Building data lakes can still take months
- 11. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Data preparation accounts for ~80% of the work Building training sets Cleaning and organizing data Collecting data sets Mining data for patterns Refining algorithms Other
- 12. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Sample of steps required Find sources Create Amazon Simple Storage Service (Amazon S3) locations Configure access policies Map tables to Amazon S3 locations ETL jobs to load and clean data Create metadata access policies Configure access from analytics services Rinse and repeat for other: data sets, users, and end-services And more: manage and monitor ETL jobs update metadata catalog as data changes update policies across services as users and permissions change manually maintain cleansing scripts create audit processes for compliance … Manual | Error-prone | Time consuming
- 13. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Enforce security policies across multiple services Gain and manage new insights Identify, ingest, clean, and transform data Build a secure data lake in days AWS Lake Formation
- 14. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. How it works
- 15. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Register existing data or import new Amazon S3 forms the storage layer for Lake Formation Register existing S3 buckets that contain your data Ask Lake Formation to create required S3 buckets and import data into them Data is stored in your account. You have direct access to it. No lock-in. Data Lake Storage Data Catalog Access Control Data import Lake Formation Crawlers ML-based data prep
- 16. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Easily load data to your data lake logs DBs Blueprints Data Lake Storage Data Catalog Access Control Data import Lake Formation Crawlers ML-based data prep one-shot incremental
- 17. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. With blueprints You 1. Point us to the source 2. Tell us the location to load to in your data lake 3. Specify how often you want to load the data Blueprints 1. Discover the source table(s) schema 2. Automatically convert to the target data format 3. Automatically partition the data based on the partitioning schema 4. Keep track of data that was already processed 5. You can customize any of the above
- 18. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Blueprints build on AWS Glue
- 19. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Easily de-duplicate your data with ML transformations
- 20. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Secure once, access in multiple ways Data Lake Storage Data Catalog Access Control Lake Formation Admin
- 21. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Security permissions in Lake Formation Control data access with simple grant and revoke permissions Specify permissions on tables and columns rather than on buckets and objects Easily view policies granted to a particular user Audit all data access at one place
- 22. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Security permissions in Lake Formation Search and view permissions granted to a user, role, or group in one place Verify permissions granted to a user Easily revoke policies for a user
- 23. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Grant table and column-level permissions User 1 User 2
- 24. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Security – deep dive User IAM users, roles Active Directory Amazon S3
- 25. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Search and collaborate across multiple users Text-based, faceted search across all metadata Add attributes like Data owners, stewards, and other as table properties Add data sensitivity level, column definitions, and others as column properties Text-based search and filtering Query data in Amazon Athena
- 26. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Audit and monitor in real time See detailed alerts in the console Download audit logs for further analytics Data ingest and catalog notifications also published to Amazon CloudWatch events
- 27. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Example: a data lake in 3 easy steps
- 28. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Step 1: Blueprints to ingest data
- 29. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Monitor the import 1
- 30. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Imported data as table in the data lake
- 31. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Step 2: Grant permissions to securely share data
- 32. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Step 3: Run query in Amazon Athena
- 33. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Lake Formation Pricing No additional charges – Only pay for the underlying services used.
- 34. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Customer interest “We are very excited about the launch of AWS Lake Formation, which provides a central point of control to easily load, clean, secure, and catalog data from thousands of clients to our AWS-based data lake, dramatically reducing our operational load. … Additionally, AWS Lake Formation will be HIPAA compliant from day one …” Aaron Symanski, CTO, Change Healthcare “I can’t wait for my team to get our hands on AWS Lake Formation. With an enterprise-ready option like Lake Formation, we will be able to spend more time deriving value from our data rather than doing the heavy lifting involved in manually setting up and managing our data lake.” Joshua Couch, VP Engineering, Fender Digital
- 35. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. https://aws.amazon.com/es/lake-formation/ https://aws.amazon.com/es/lake-formation/features/ https://aws.amazon.com/es/lake-formation/faqs/
- 36. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Ricardo Serafim Analytics Specialist Solutions Architect [email protected]