2017 AWS DB Day | AWS 데이터베이스 개요 - 나의 업무에 적합한 데이터베이스는?
1 like•3,193 views
The document discusses Amazon Web Services' database solutions, highlighting services like Amazon RDS, DynamoDB, Aurora, and Redshift. It emphasizes the advantages of managed database services such as lower operational overhead, scalability, and high availability, along with features like rapid provisioning and automated backups. The document also includes customer use cases demonstrating improved performance and cost-effectiveness in various applications.
2017 AWS DB Day | AWS 데이터베이스 개요 - 나의 업무에 적합한 데이터베이스는?
- 1. Databases onAWS Blair Layton Amazon Web Services, Business Development Manager Database Services, Asia Pacific Time : 10:10 – 11:00
- 2. Amazon RDS Amazon DynamoDB Amazon Redshift Amazon ElastiCache Compute Storage AWS Global Infrastructure Database Application Services Deployment & Administration Networking AWS Database Services
- 4. If you host your databases on-premises Power, HVAC, net Rack & stack Server maintenance OS patches DB s/w patches Database backups Scaling High availability DB s/w installs OS installation you App optimization
- 5. If you host your databases in EC2 Power, HVAC, net Rack & stack Server maintenance OS patches DB s/w patches Database backups Scaling High availability DB s/w installs OS installation you App optimization
- 6. If you choose a managed DB service like RDS Power, HVAC, net Rack & stack Server maintenance OS patches DB s/w patches Database backups App optimization High availability DB s/w installs OS installation you Scaling
- 7. differentiated effort increases the uniqueness of an application
- 8. Relational Databases Managed Service Simple and fast to scale Fast, predictable performance Low cost, pay for what you use Amazon RDS Amazon Aurora
- 9. Key Features Provision a database in 6 minutes Provision a MAZ database with a few mouse clicks Scale a database up/down with 60-90 seconds downtime Apply patches with 60-90 seconds downtime Add read replicas with a few mouse clicks Protect your backups and logs with 11 9’s of durability Recover to any point in time from nightly backups + logs Detailed metrics, down to 1 second intervals Secure your data with single click encryption at rest Amazon RDS
- 11. Demo
- 12. Amazon Aurora
- 13. MySQL-compatible and PostgreSQL-compatible relational database platform Performance and availability of commercial databases Simplicity and cost-effectiveness of open source databases Delivered as a managed service What is Amazon Aurora?
- 14. A service-oriented architecture applied to the database Moved the logging and storage layer into a multitenant, scale-out database-optimized storage service Integrated with other AWS services like Amazon EC2, Amazon VPC, Amazon DynamoDB, Amazon SWF, and Amazon Route 53 for control plane operations Integrated with Amazon S3 for continuous backup with 99.999999999% durability Control PlaneData Plane Amazon DynamoDB Amazon SWF Amazon Route 53 Logging + Storage SQL Transactions Caching Amazon S3 1 2 3
- 15. I/O traffic patterns: MySQL vs. Aurora Binlog Data Double-write bufferLog records FRM files, metadata T Y P E O F W R I T E S EBS mirrorEBS mirror AZ 1 AZ 2 Amazon S3 MYSQL WITH STANDBY SEQUENTIAL WRITE SEQUENTIAL WRITE EBS Amazon Elastic Block Store (EBS) Primary Instance Standby Instance AZ 1 AZ 3 Primary Instance Amazon S3 AZ 2 Replica Instance AMAZON AURORA ASYNC 4/6 QUORUM DISTRIBUTED WRITES
- 16. Key Features Highly available storage, with 6 copies across 3 AZs No more MySQL replica lag! Auto-scaling storage for IOPS and capacity Parallel, distributed & asynchronous recovery Survivable caches: quick restart & no brown outs Faster, more predictable failover Up to 5x faster than MySQL and more than 2x faster than PostgreSQL on the same hardware Cheaper at scale!
- 17. Recent Announcements Performance enhancements: Fast DDL, fast index build, spatial indexing, hot row contention Availability features: Zero-downtime patching, database cloning (Q2), database backtrack (Q2) Eco-system integration: Load from S3, IAM integration (Q2), select into S3 (Q2), log upload to CloudWatch Logs & S3 (Q2) Cost reduction: t2.small – cuts cost of entry by half – you can run Aurora for $1 / day Growing footprint: London, Montreal, Ohio, and San Francisco – now available in all 3AZ regions
- 18. 2/3 of top 100 AWS customers 8 of top 10 gaming customers Fastest growing service in AWS history
- 19. Feature Aurora MySQL MariaDB PostgreSQL Oracle SQL Server VPC High availability Instance Scaling Encryption Read replicas Oracle Golden Gate / DMS DMSCross region replicas Max Storage 64 TB 6 TB 6 TB 6 TB 6 TB 4 TB Scale Storage Auto scaling Provisioned IOPS NA 30,000 30,000 30,000 30,000 20,000 Largest Instance R3.8XL R3.8XL M4.10XL R3.8XL M4.10XL R3.8XL M4.10XL R3.8XL M4.10XL R3.8XL M4.10XL RDS Engine Capability Matrix
- 21. Automates sampling of data Exposes data via API Provides UI to show Database Load Database Load: Performance Insights at a glance
- 23. In-Memory Cache Elastic and reliable Memcached or Redis Managed Service Amazon ElastiCache
- 24. Why In-Memory Caching? Everything is connected - phones, tablets, cars, air conditioners, toasters Demand for real-time performance – online games, ad tech, eCommerce, social apps, etc. Load is spikey and unpredictable Database performance is often the bottleneck
- 25. Memcached vs. Redis • In-memory • Multithreaded / Multicore • No persistence • String values • Easy horizontal clustering • In-memory • Single-threaded • Read replicas • Clustering • Persistence • Atomic operations • Advanced data types - http://redis.io/topics/data-types • Pub/sub messaging
- 26. RDS and ElastiCache are Behind Grab’s Taxi-Booking App The latency of a cab call must be low, and remain low even in times of peak traffic of hundreds of thousands of cab requests per minute. We use ElastiCache for Redis in front of RDS MySQL to keep our systems’ real time performance at any scale. Ryan Ooi Sr. Devops Engineer, Grab ” “ • Grab is a popular taxi hailing app in southeast Asia. • Average response time of the API layer is <40ms, mandating an in-memory layer to achieve such performance. • A small devops team that tried running Redis on EC2 before, but that was too much work. Using both RDS and ElastiCache in Multi-AZ allowed them to outsource all the management to AWS.
- 27. NoSQL Database Durable low latency Managed Service Massive and seamless scalability Amazon DynamoDB
- 28. = • Consistent, single-digit millisecond latency at any scale • Highly durable and available—3x replication • Accessible via simple and powerful APIs • Supports both document and key- value data models • No table size or throughout limits
- 29. Consistent low latency whether scaling up/down or operating at your provisioned limits Durable Low Latency – At Scale
- 32. Relational data warehouse Massively parallel; petabyte scale Managed Service HDD and SSD platforms $1,000/TB/year; starts at $0.25/hour Amazon Redshift a lot faster a lot cheaper a whole lot simpler
- 33. Key Capabilities Scaleable: Petabyte scale data warehouse: from 160 GB to 2PB Fast: parallel execution with compressed, sorted data on optimized hardware Inexpensive: Start from $0.25/hour or $1,000/TB/year Managed Service: Easy to provision, backup, restore, patch and scale Secure: Load and store encrypted data, SSL, Audit logging Innovative: 100 new features since launch! Large Ecosystem: Major data integration and visualization ISVs support Redshift with a large consulting partner base
- 34. Amazon.com – Weblog analysis Web log analysis for Amazon.com 1PB+ workload, 2TB/day, growing 67% YoY Largest table: 400 TB Want to understand customer behavior Solution Legacy DW—query across 1 week/hr. Hadoop—query across 1 month/hr.
- 35. Query 15 months of data (1PB) in 14 minutes Load 5B rows in 10 minutes 21B rows joined with 10B rows – 3 days (Hive) to 2 hours Load pipeline: 90 hours (Oracle) to 8 hours 64 clusters 800 total nodes 13PB provisioned storage 2 DBAs Data warehouses can be fast and simple
- 37. Amazon RDS Amazon DynamoDB Amazon Redshift Amazon ElastiCache Compute Storage AWS Global Infrastructure Database Application Services Deployment & Administration Networking AWS Database Services
- 38. Databases on EC2 Any database that runs on Windows or Linux! Many AMIs available from technology partners Oracle Database, MS SQL Server, MongoDB, Vertica, … White papers available on best practices Oracle Database, MS SQL Server, MongoDB, Cassandra, … Why? No managed service Full control Exceed limits of managed service, e.g. > 6TB of storage on RDS
- 39. New X1 Instance - Tons of Memory • Designed for large-scale, in-memory applications in the cloud • Ideal for in-memory databases like SAP HANA and big data processing apps like Spark and Presto • Powered by Intel® Xeon® E7 8880 v3 Haswell processors • Features up to 2TB of memory and up to 128 vCPUs per instance. • Expanding to 4TB in 2017. With 8TB and 16TB in the works too!
- 40. AWS Database Migration Service (AWS DMS) DMS migrates databases to AWS easily and securely with minimal downtime. It can migrate your data to and from most widely used commercial and open-source databases. Amazon Aurora Amazon DynamoDB
- 41. Less than $10 per TB with CDC!
- 42. AWS Schema Conversion Tool (AWS SCT) SCT helps automate many database schema and code conversion tasks when migrating between database engines or data warehouse engines Amazon Aurora
- 43. SCT helps with converting tables, views, and code Sequences User Defined Types Synonyms Packages Stored Procedures Functions Triggers Schemas Tables Indexes Views
- 44. SCT can tell you how hard the migration will be 1. Connect SCT to Source and Target databases. 2. Run Assessment Report. 3. Read Executive Summary. 4. Follow detailed instructions.
- 45. Which Service Should You Use? Situation Solution Existing application Use your existing engine on RDS • MySQL Amazon Aurora, RDS for MySQL • PostgreSQL RDS for PostgreSQL • Oracle, SQL Server RDS for Oracle, RDS for SQL Server New application • If you can avoid relational features DynamoDB • If you need relational features Amazon Aurora Data Warehouse & BI • Amazon Redshift and Amazon QuickSight Ad hoc analysis of data in S3 • Amazon Athena and Amazon QuickSight Spark, Hadoop, Hive, HBase • Amazon EMR Log analytics, operational monitoring and search • Amazon Elasticsearch Service
- 47. 반도체 쪽으로 유학을 가려던 평범한 전기공학도 어쩌다한 창업 지금은 레코벨에서 빅데이터를 경험중 AWS 사용 6년째 Me?
- 48. • 추천서비스 • 개인화 마케팅 서비스 • 검색광고 솔루션 • Retargeting 광고 • 대용량 푸시 서버 ++ 데이터를 이용한 돈벌이 데이터가 돈이 된다는 것을 증명하는 사람들
- 49. 추천서비스?
- 51. 0 50 100 150 200 250 300 350 2014. 12 2015. 3 2015. 9 2016. 4 2016. 10 2017. 5
- 52. 추천서비스 사용 사이트 숫자 0 50 100 150 200 250 300 350 2014. 12 2015. 3 2015. 9 2016. 4 2016. 10 2017. 5
- 53. 추천서비스 고군분투기 on AWS 이 세션에서 다룰 내용 IDC 에서 클라우드로 최초 10개 미만이던 사이트 숫자를 300개 이상으로 빠르게 성장한 이야기
- 54. 추천서비스 고군분투기 on AWS 데이터 분석은 크게 3 단계 Logging Analysis Operation Visualize
- 55. 추천서비스 on IDC
- 56. 추천서비스 고군분투기 on AWS Legacy on IDC Server on IDC Logger API DB (+Engine) Client Site User Behavior Result (HTML) JS SDK • 대기업위주의 SI • Engineer 보다는 Data Scientists 위주 • 간간히 알바/외주를 써서 몇 개의 고객사에 대해서만 IDC를 빌려서 구현
- 57. 추천서비스 고군분투기 on AWS Legacy on IDC Server on IDC Logger API DB (+Engine) Client Site User Behavior Result (HTML) JS SDK • 고객사 영업되면, 세팅 한 세월 • 장애 대응 한 세월 • 테스트 한 세월 • 내가 SE 도 아니고..
- 58. 추천서비스 고군분투기 on AWS AWS 로 이사가기 Server on IDC Logger API DB (+Engine) • AWS 로 이사가기로 결정 • Logger / API 는 묶어서 사용 • DB 는 RDS 를 이용
- 59. • 관리용이성 • 뛰어난 확장성 • 가용성 및 내구성 • 빠른속도(SSD) • 보안 • 저렴한 비용 Amazon RDS
- 60. 추천서비스 고군분투기 on AWS AWS 위 첫 번째 집 • 큰 규모의 업체는 SI • 작은 규모의 업체는 클라우드 MySQLs Logger API
- 61. 하지만 가정은 보통 깨어지고.. MAU 천만 업체 등장
- 62. 추천서비스 고군분투기 on AWS AWS 위 두 번째 집 • 서비스 쪽과 엔진 쪽 분리 • LogDB + ResultDB • Rec. Engine MySQLs Logger API Partial Log Recommendation Result Recommendation Engine
- 63. • 관리용이성 • 뛰어난 확장성 • 가용성 및 내구성 • 빠른속도(SSD) • 보안 • 저렴한 비용 Amazon RDS
- 64. • 빠른속도(SSD) • 저렴한 비용 => SSD $0.138/GB.Month • + 긴 RDS 생성시간 Amazon RDS
- 65. 대대적인 구조 개선 시작
- 66. 추천서비스 고군분투기 on AWS 구조개선 목표 • Logger : 안정성 및 TPS • Engine : 속도 및 안정성 • API : Business 로직 최소화
- 67. • 간편성 • 내구성 • 확장가능 • 보안 • 가용성 • 저렴한비용 • 간편한 데이터 전송 • 통합 • 손쉬운관리 S3 (Simple Storage Service)
- 68. 추천서비스 고군분투기 on AWS AWS 위 세 번째 집 • S3 에 데이터 저장 Infra on AWS Logger API Client Site User Behavior Result (JSON) EngineLog Bucket Result Bucket
- 69. 추천서비스 고군분투기 on AWS Logger Kinesis PUT Producer Consumer S3 Log Bucket
- 70. Real-time Streaming Data Buffer • 실시간성 • 사용편의성 • 병렬처리 • 탄력성 • 저렴한비용 • 안정성 Amazon Kinesis Stream
- 71. 추천서비스 고군분투기 on AWS Engine • EC2 : c4.8xlarge – 36 CPU / 132 ECU / 60G Mem. – Provisioned IOPS EBS (3000) • PostgreSQL + Spring Boot PUT S3 Result Bucket GET S3 Log Bucket …/{siteId}/{recType}/{item or userId}.json…/{siteId}/{logType}/{date}/{time}.json
- 72. 추천서비스 고군분투기 on AWS API GET S3 Result Bucket …/{siteId}/{recType}/{item or userId}.json API "recType": "a002", "itemId": "354427372", "results”: [ { "itemId": "362054013", "categoryId": "5502167”, "rank": 1, "score": 1, http://{apiUrl}/{recType}?itemId={itemId}Req Res
- 73. 성공적으로 20개 사이트 라이브 MAU 천만이 넘어도 문제 없음 이때만든로거는지금까지도문제없음
- 74. 그런데
- 75. 사이트 200개 추가
- 76. 추천서비스 고군분투기 on AWS 상황 1 • 가장 쉽게 이 상황에 대처하는 방법? – Logger / API 는 Scale-out – Engine 220개를 띄운다. => 먼저 EC2 instance 의 Limit 을 풀어야한다. • c4.8xlarge instance : $2.3/hour – AWS 과금형식이 시간당 과금 – 200 개 사이트 중에 수 분 내에 엔진이 끝나는 사이트도.. – + Provisioned IOPS 도 비싸
- 77. 추천서비스 고군분투기 on AWS 상황 2 • 추천 고객사가 다양해 지면서 다양한 e-Commerce 고객사 등장 • e-Commerce 가 아닌 고객사도 등장 • Data Schema 가 다들 다름 • 이 때 레코벨의 개발자 2명 • Java + SpringBoot + PostgreSQL 수정이 너무 어려워
- 78. 그런데
- 79. 추천서비스 고군분투기 on AWS 두마리 토끼 • 더 가격이 싼 구조 – 비싼 c4.8xlarge 대체 – Provisioned IOPS 도 대체 • 개발자 없이, 빠르게 고객사의 요구사항을 들어줄 수 있는 구조 – Data Scientist 들이 SQL 은 빠르게 수정할 수 있으니, 최대한 SQL 로 모든걸 해결할 수 있게하자.
- 80. Data Warehouse 서비스 • 신속함 • 저렴함 • 간편성 • 탄력성 • 보안 • 호환성 (PostgreSQL) Amazon Redshift
- 81. 추천서비스 고군분투기 on AWS AWS 위 네 번째 집 • 1/6 가격 • 쉬운 수정 • 개발자 2명 • DS 4명 • 사이트 220개EngineLogger Log bucket Result Bucket API 추천데이터 로그
- 82. • 간편성 • 내구성 • 확장가능 • 보안 • 가용성 • 저렴한비용 • 간편한 데이터 전송 • 통합 • 손쉬운관리 S3 (Simple Storage Service) • 100만개의 상품을 가진 사이트면, 한번 엔진이 돌때마다 비용이.. – $0.0047 / 1,000 * 1,000,000 = $4.7 • 하루에 세번, 한달이면.. – $4.7 * 3 * 30 = $423
- 83. 고성능, 고-확장성 DB • 고성능 • 뛰어난 보안 • MySQL/PostgreSQL 호환 • 뛰어난 확장성 • 높은 가용성 및 내구성 • 완전 관리형 RDS : Amazon Aurora
- 84. 추천서비스 고군분투기 on AWS AWS 위 다섯 번째 집 EngineLogger Log bucket Result On Aurora API 추천데이터 로그
- 85. 추천서비스 고군분투기 on AWS Remind & Tips • RDS / EC2 의 IOPS 에 항상 주의 • 웬만한 데이터는 S3, 하지만 PUT 가격에 주의 • 데이터 버퍼로써 Kinesis 활용 • Aurora 는 요금체계가 기존 RDS 와 다르다. (IOPS 과금) • Redshift 는 쉽고 빠르다.
- 87. Thank you!
- 88. Back to the Demo!