Deep Dive on Amazon S3 (May 2016)
1 like680 views
This document summarizes a presentation about using Amazon S3 for video ingestion. It discusses how a company called Xstream uses S3 for ingesting large video files. Key benefits of S3 for this use case include unlimited scalability, built-in security and encryption, and automatic integrity checks. The presentation also covers various S3 features like lifecycle policies, versioning, and transfer acceleration that help optimize storage and performance. Real-world metrics are provided on Xstream's extensive use of S3 for video storage and delivery.
![Standard Storage -> Standard-IA
"Rules": [
{
"Status": "Enabled",
"Prefix": ”old_files",
"Transitions": [
{
"Days": 30,
"StorageClass": "STANDARD_IA"
},
{
"Days": 365,
"StorageClass": "GLACIER"
}
],
"ID": ”lifecycle_rule",
}
]
Standard à Standard-IA](https://image.slidesharecdn.com/20160503s3deepdive-160505080126/85/Deep-Dive-on-Amazon-S3-May-2016-29-320.jpg)
!["Rules": [
{
"Status": "Enabled",
"Prefix": ”old_files",
"Transitions": [
{
"Days": 30,
"StorageClass": "STANDARD_IA"
},
{
"Days": 365,
"StorageClass": "GLACIER"
}
],
"ID": ”lifecycle_rule",
}
]
Standard-IA -> Amazon Glacier
Standard-IA à Amazon Glacier
Standard Storage -> Standard-IA](https://image.slidesharecdn.com/20160503s3deepdive-160505080126/85/Deep-Dive-on-Amazon-S3-May-2016-30-320.jpg)
!["Rules": [
{
…
"Expiration": {
"Days": 60
},
"NoncurrentVersionExpiration": {
"NoncurrentDays": 30
}
]
}
Lifecycle policy to expire versioned objects
Current version will expire after 60
days.
Older versions will be permanently
deleted after 30 days.](https://image.slidesharecdn.com/20160503s3deepdive-160505080126/85/Deep-Dive-on-Amazon-S3-May-2016-33-320.jpg)
!["Rules": [
{
…
"Expiration": {
"Days": 60,
"ExpiredObjectDeleteMarker" : true
},
"NoncurrentVersionExpiration": {
"NoncurrentDays": 30
}
]
}
Lifecycle policy to expire delete markers
Current version will expire after 60
days. A delete marker will be placed
and expire after 60 days.
Older versions will be permanently
deleted after 30 days.](https://image.slidesharecdn.com/20160503s3deepdive-160505080126/85/Deep-Dive-on-Amazon-S3-May-2016-35-320.jpg)
![Incomplete multipart uploads will
expire seven days after initiation
"Rules": [
{
…
"AbortIncompleteMultipartUpload": {
"DaysAfterInitiation": 7
}
]
}
Lifecycle policy to expire multipart uploads](https://image.slidesharecdn.com/20160503s3deepdive-160505080126/85/Deep-Dive-on-Amazon-S3-May-2016-44-320.jpg)
Deep Dive on Amazon S3 (May 2016)
- 1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Deep Dive on Amazon S3 Julien Simon, Principal Technical Evangelist, AWS [email protected] - @julsimon Loke Dupont, Head of Services, Xstream A/S [email protected]
- 2. Agenda • Introduction • Case study: Xstream A/S • Amazon S3 Standard-Infrequent Access • Amazon S3 Lifecycle Policies • Amazon S3 Versioning • Amazon S3 Performance & Transfer Acceleration
- 4. S3: our customer promise Durable 99.999999999% Available Designed for 99.99% Scalable Gigabytes à Exabytes
- 5. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Using Amazon S3 for video ingestion Loke Dupont, Head of Services, Xstream A/S [email protected]
- 6. What does Xstream do? Xstream is an online video platform provider. We sell OVP’s to broadcasters, ISP’s, cable companies etc. What we are trying to provide is a ”white-label Netflix” that our customers can use to provide video services to end users. As part of that delivery, we ingest large amounts of video.
- 7. Challenges of ingesting video
- 8. Challenges of premium video ingestion • Very large files (upwards of several hundred GB) • Content security is extremely important • Content integrity is very important (no video corruption) • Content often arrive in batches of 1000+ videos • Content needs to be available to all ingest processes
- 9. Ingest workflow Decrypt Transcode Packaging DRM Upload
- 10. Ingest architecture Ingest DatabaseWorkflow ManagerIngest API Queue Workers WorkersWorkers Workers Workers • Amazon RDS MySQL instance for data • Running 100% on Amazon EC2 instances • Planning to replace EC2 with AWS Lambda and Amazon SQS
- 11. How does Amazon S3 help?
- 12. Amazon S3 real world usage In April, in just one region, we had: • 300 TB/month of short term storage in S3 • 62 million PUT/COPY/POST/LIST requests • 55 million GET requests In the same region we had 848 TB of Amazon Glacier long term archive storage
- 13. Previous workflow vs. Amazon S3 Previous workflow • Large files moved between machines • Managing access had to be done pr. machine • Disk space had to be managed carefully • Encryption at rest was tricky • Constant file integrity checks Amazon S3 • Files always accessible on Amazon S3 • Managing access for bucket using policy and Amazon IAM • Running our of space, practically impossible • Encryption is easy • S3 checks integrity for us, using checksums
- 14. What else do we get for free? Versioning, which allows us to retrieve deleted and modified objects. Easy Amazon Glacier integration for long term content archiving of “mezzanine” assets. Alternatively Amazon S3-IA could be used. Event notifications using Amazon SNS, Amazon SQS and AWS Lambda
- 15. Demo Amazon S3 events & AWS Lambda Sample code: http://cloudvideo.link/lambda.zip
- 16. Lesser known Amazon S3 features – Bucket tagging Bucket tagging is a great feature for cost allocation. Assign custom tags to your bucket and they can be used to separate cost pr. customer or pr. project.
- 17. Getting cost with tags Setup Cost allocation tags in preferences. Using the AWS Cost Explorer to create a new report. Select filter by “tags” and select the tag you want to filter by.
- 18. Lesser known Amazon S3 features – Lifecycle Use the lifecycle feature to automatically transition to the Amazon S3 Infrequent Access storage class, or even to Amazon Glacier. Be careful about retrieval costs especially when using Amazon Glacier backed storage.
- 19. Lessons learned
- 20. Lessons learned from Amazon Glacier Verify archive creation before deleting data. Retrieval is priced by “peak rate” – spread it out Retrieval has several hours latency
- 21. AWS Storage cost comparison
- 22. Things we wish we new earlier • Don’t use Amazon S3 filesystem wrappers • Use Amazon IAM roles whenever possible • If there is an AWS service for it, use that! • Auto scaling, auto scaling, auto scaling
- 23. Amazon S3 Standard-IA Expired object delete marker Incomplete multipart upload expiration Lifecycle policy & Versioning Object naming Multipart operations Transfer Acceleration Continuous Innovation for Amazon S3 Performance 16/3 16/3 19/4 September 2015
- 25. Choice of storage classes on Amazon S3 Standard Active data Archive dataInfrequently accessed data Standard - Infrequent Access Amazon Glacier
- 26. 11 9s of durability Standard-Infrequent Access storage Designed for 99.9% availability Durable Available Same throughput as Amazon S3 Standard storage High performance • Server-side encryption • Use your encryption keys • KMS-managed encryption keys Secure • Lifecycle management • Versioning • Event notifications • Metrics Integrated • No impact on user experience • Simple REST API • Single bucket Easy to use
- 28. Lifecycle policies • Automatic tiering and cost controls • Includes two possible actions: • Transition: archives to Standard-IA or Amazon Glacier after specified time • Expiration: deletes objects after specified time • Allows for actions to be combined • Set policies at the prefix level aws s3api put-bucket-lifecycle-configuration --bucket BUCKET_NAME --lifecycle-configuration file://LIFECYCLE_JSON_FILE
- 29. Standard Storage -> Standard-IA "Rules": [ { "Status": "Enabled", "Prefix": ”old_files", "Transitions": [ { "Days": 30, "StorageClass": "STANDARD_IA" }, { "Days": 365, "StorageClass": "GLACIER" } ], "ID": ”lifecycle_rule", } ] Standard à Standard-IA
- 30. "Rules": [ { "Status": "Enabled", "Prefix": ”old_files", "Transitions": [ { "Days": 30, "StorageClass": "STANDARD_IA" }, { "Days": 365, "StorageClass": "GLACIER" } ], "ID": ”lifecycle_rule", } ] Standard-IA -> Amazon Glacier Standard-IA à Amazon Glacier Standard Storage -> Standard-IA
- 31. Versioning S3 buckets • Protects from accidental overwrites and deletes • New version with every upload • Easy retrieval and rollback of deleted objects • Three states of an Amazon S3 bucket • No versioning (default) • Versioning enabled • Versioning suspended { "Status": "Enabled", "MFADelete": "Disabled" } aws s3api put-bucket-versioning --bucket BUCKET_NAME --versioning-configuration file://VERSIONING_JSON_FILE
- 32. Restricting deletes • For additional security, enable MFA (multi-factor authentication) in order to require additional authentication to: • Change the versioning state of your bucket • Permanently delete an object version • MFA delete requires both your security credentials and a code from an approved authentication device
- 33. "Rules": [ { … "Expiration": { "Days": 60 }, "NoncurrentVersionExpiration": { "NoncurrentDays": 30 } ] } Lifecycle policy to expire versioned objects Current version will expire after 60 days. Older versions will be permanently deleted after 30 days.
- 34. Delete markers • Deleting a versioned object puts a delete marker on the current version of the object • No storage charge for delete marker • No need to keep delete markers when all versions have expired (they slow down LIST operations) • Use a lifecycle policy to automatically remove the delete marker when previous versions of the object no longer exist
- 35. "Rules": [ { … "Expiration": { "Days": 60, "ExpiredObjectDeleteMarker" : true }, "NoncurrentVersionExpiration": { "NoncurrentDays": 30 } ] } Lifecycle policy to expire delete markers Current version will expire after 60 days. A delete marker will be placed and expire after 60 days. Older versions will be permanently deleted after 30 days.
- 37. <my_bucket>/2013_11_13-164533125.jpg <my_bucket>/2013_11_13-164533126.jpg <my_bucket>/2013_11_13-164533127.jpg <my_bucket>/2013_11_13-164533128.jpg <my_bucket>/2013_11_12-164533129.jpg <my_bucket>/2013_11_12-164533130.jpg <my_bucket>/2013_11_12-164533131.jpg <my_bucket>/2013_11_12-164533132.jpg <my_bucket>/2013_11_11-164533133.jpg <my_bucket>/2013_11_11-164533134.jpg <my_bucket>/2013_11_11-164533135.jpg <my_bucket>/2013_11_11-164533136.jpg Use a key-naming scheme with randomness at the beginning for high TPS • Most important if you regularly exceed 100 TPS on a bucket • Avoid starting with a date • Avoid starting with sequential numbers Don’t do this… Distributing key names
- 38. Distributing key names …because this is going to happen 1 2 N 1 2 N Partition Partition Partition Partition
- 39. Distributing key names Add randomness to the beginning of the key name… <my_bucket>/521335461-2013_11_13.jpg <my_bucket>/465330151-2013_11_13.jpg <my_bucket>/987331160-2013_11_13.jpg <my_bucket>/465765461-2013_11_13.jpg <my_bucket>/125631151-2013_11_13.jpg <my_bucket>/934563160-2013_11_13.jpg <my_bucket>/532132341-2013_11_13.jpg <my_bucket>/565437681-2013_11_13.jpg <my_bucket>/234567460-2013_11_13.jpg <my_bucket>/456767561-2013_11_13.jpg <my_bucket>/345565651-2013_11_13.jpg <my_bucket>/431345660-2013_11_13.jpg Other ideas • Store objects as a hash of their name and add the original name as metadata “deadbeef_mix.mp3” à 0aa316fb000eae52921aab1b4697424958a53ad9 • Reverse key name to break sequences
- 40. Distributing key names …so your transactions can be distributed across the partitions 1 2 N 1 2 N Partition Partition Partition Partition
- 41. Parallelizing PUTs with multipart uploads • Increase aggregate throughput by parallelizing PUTs on high-bandwidth networks • Move the bottleneck to the network where it belongs • Increase resiliency to network errors; fewer large restarts on error-prone networks https://aws.amazon.com/fr/premiumsupport/knowledge-center/s3-multipart-upload-cli/
- 42. Choose the right part size • Maximum number of parts: 10,000 • Part size: from 5MB to 5GB • Strike a balance between part size and number of parts • Too many small parts à connection overhead (TCP handshake & slow start) • Too few large parts à not enough benefits of multipart
- 43. Incomplete multipart upload expiration policy • Multipart upload feature improves PUT performance • Partial upload does not appear in bucket list • Partial upload does incur storage charges • Set a lifecycle policy to automatically expire incomplete multipart uploads after a predefined number of days Incomplete multipart upload expiration
- 44. Incomplete multipart uploads will expire seven days after initiation "Rules": [ { … "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 } ] } Lifecycle policy to expire multipart uploads
- 45. Parallelize your GETs • Use Amazon CloudFront to offload Amazon S3 and benefit from range-based GETs • Use range-based GETs to get multithreaded performance when downloading objects • Compensates for unreliable networks • Benefits of multithreaded parallelism • Align your ranges with your parts! CloudFront EC2 S3
- 46. Parallelizing LIST • Parallelize LIST when you need a sequential list of your keys • Secondary index to get a faster alternative to LIST • Sorting by metadata • Search ability • Objects by timestamp “Building and Maintaining an Amazon S3 Metadata Index without Servers” AWS blog post by Mike Deck on using Amazon DynamoDB and AWS Lambda
- 48. Amazon S3 Transfer Acceleration • Designed for long distance transfers • Send data to Amazon S3 using the 54 AWS Edge Locations • Up to 6 times faster thanks to the internal AWS network • No change required (software, firewalls, etc.) • Must be explicitely set by customers, on a per-bucket basis • Pay according to volume : from $0.04 / GB • You’re only charged if transfer is faster than using Amazon S3 endpoints
- 49. Amazon S3 Transfer Acceleration { "Status": "Enabled" } aws s3api put-bucket-accelerate-configuration --bucket BUCKET_NAME --accelerate-configuration file://ACCELERATE_JSON_FILE
- 50. AWS Snowball • New version: 80 Terabytes (+60%) • Available in Europe (eu-west-1) • All regions available by the end of 2016 • $250 per operation • 25 Snowballs à 2 Petabytes in a week for $6250
- 51. Recap • Case study: Xstream A/S • Amazon S3 Standard-Infrequent Access • Amazon S3 Lifecycle Policies • Amazon S3 Versioning • Amazon S3 Performance & Transfer Acceleration
- 52. Thank You!