Adam Dagnall: Advanced S3 compatible storage integration in CloudStack
Download as PPTX, PDF1 like1,593 views
The document discusses Cloudian HyperStore, an advanced S3-compatible object storage solution designed for multi-cloud and on-premises environments, emphasizing its scalability, data management capabilities, and integration with CloudStack. It outlines the architecture, features, and benefits of HyperStore, including policy-based data protection, versioning, and encryption, making it suitable for handling the growing complexities of unstructured data. The document also highlights the importance of S3 compatibility for investment protection and the ease of managing multiple storage environments.
Adam Dagnall: Advanced S3 compatible storage integration in CloudStack
- 1. Cloudian HyperStore Advanced S3 Compatible Storage for CloudStack Adam Dagnall, SE Director, Northern EMEA [email protected]
- 2. 2 Agenda • High Level Intro & The Importance of S3 • HyperStore Architecture • HyperStore Features • CloudStack Integration
- 4. Data fuels the transformation to digital enterprise More connected people, apps, and things generating more data in many forms Human data Sensor data Business data faster growth than traditional business data 10x 90% of the world’s data was created in the last 2 years.
- 5. Challenge: Capacity and Complexity Fundamental challenges are growing 2010 2020 Sensors & Devices Social Media VoIP Enterprise Data Volume in Exabytes 2018 Los Angeles Denver New York CLOUD: Move towards multi-cloud ON-PREM: More capacity & silos Increasing Storage ComplexityAccelerating Capacity Growth © 2018, Cloudian, Inc. Confidential. Do Not Distribute
- 6. S3: de-facto Object Storage API Standard Amazon’s cloud is 10 times larger than the next 14 competitors combined. © 2017, Cloudian, Inc.8 Launched in 2006, AWS dominates the public cloud market
- 7. © 2017, Cloudian, Inc.9 On-Premises Storage is Still Essential Why: • Performance • Security • Compliance • Cost of access APPLICATIONS ON PREM STORAGE DATA CENTER Challenge: Multiple Storage Environments to Manage
- 8. © 2017, Cloudian, Inc.10 Meet Cloudian • On-premises S3-compatible object storage • Ideal solution for hybrid cloud • Start small, scale to 100s of petabytes
- 9. © 2017, Cloudian, Inc. 11 Cloudian HyperStore: S3 Object Storage • Highly scalable • Scale by adding nodes • None of the limitations associated with Block and File • Custom metadata • Stored in Cassandra • Can be exported / Indexed • REST based (HTTP / HTTPS) Data Access • S3 API • Management API • NON RAID • Configurable Replicas • Configurable Erasure Coding
- 10. S3 Functionality S3 is a modern storage protocol combining • Connectivity • Access control • Data management • Reporting & Notifications • Advanced Functionality • Tiering • Encryption • Billing • Monitoring • Notifications • Replication • Time limited access …. that can be used across the internet. Standard Storage Protocols S3 API 12
- 11. © 2017, Cloudian, Inc.13 S3 Compatibility
- 12. 14 Why S3 Compatibility Matters 1. Cloud storage standard established at Amazon; world’s largest object storage environment 2. MSPs worldwide adopting S3 for cloud storage services 3. Hundreds of applications now support S3, many more in development S3 Compatibility = Investment Protection
- 13. Unstructured Data Consolidation – additional use cases Backup File NAS Offload Media Asset Mgmt Sync and Share AI / Machine Learning S3 API
- 14. ©16 Industry’s Most Complete Feature Set 100% Native S3 API Multi Tenancy, QoS, Billing Policy-based data protection One System for Files & Objects Policy-based Tiering to Cloud Flexible Deployment Options Up to 14 nines data durability _ 2 1 Cloud-like cost GB/ mo.
- 16. 18 HyperStore Node Hardware CPU HDD Drives Network SSD Drives • CPUs – CPU Clock speed is preferred over number of cores, but of course the more cores the better. • Memory – More memory is required as a node has more drives/capacity • OS/Meta-data disks - More capacity is required for meta-data storage as a node has more drives/capacity • Network Interface – For better performance it is highly recommended to deploy nodes with 2 x 10GbE interfaces as a minimum, more ports can be utilized by bonding ports together for resilience and greater bandwidth capability. RAM
- 17. ©19 250TB 500TB 750TB 1PB Limitlessly Scalable Grows with your needs • Infinite capacity • Never migrate data again • Performance increases as nodes are added Looks like one device
- 18. HyperStore High Level Structure Region Data Center Nodes Data Center Nodes
- 19. Support for multiple Topologies and Protection Schemes © 2016, Cloudian, Inc. Confidential. Do Not Distribute21 Replication Factor Clients Replicated EC EC Single Site Distributed EC DC1 S3 Bucket – RF=3 DC1 S3 Bucket – EC:4+2 Clients S3 Bucket – RF=2 over EC:4+2 Clients DC1 – EC:4+2 DC2 – EC:4+2 Replication Storage Overhead Raw TBs / RF Storage Overhead (k/k+m) x Raw TBs Storage Overhead (k/k+m) x Raw TBs / DC RF DC1 DC3DC2 S3 Bucket – Distributed EC:7+5 ClientsStorage Overhead (k/k+m) x Raw TBs Low Latency Network Required
- 20. Peer-to-peer system = no SPOF Distributed Everything = Data , Metadata, Configuration User Defined Location Affinity DC1 DC2 Add Node <-> Auto Rebalance Server <-> vNodes <-> Disks Distributed & Elastic Geo Cluster
- 21. 23 Support for Multiple Configurations Storage MSP On-Premises Apps Multi-Site Hybrid Apps Stand alone, on-premise storage Replicate or migrate to the cloud for capacity, data protection, content distribution Fully S3-compatible storage servicesReplicate or distribute data across sites SITE A SITE B SITE C
- 22. This eliminates the traditional bottlenecks associated with other architectures. • S3 Service • HyperStore Service • Cassandra DB service • Redis DB service • Admin service • Cloudian Management Console (CMC) • Supporting Services Key Services Run on ALL nodes HyperStore Services: © 2017, Cloudian, Inc.24
- 23. © 2017, Cloudian, Inc. Confidential. Do Not Distribute25 Start Small, Grow Without Disruption The Solution for Scalable Storage Begin with just 3 nodes • Full data protection Grow to 100s of Petabytes • Zero down-time scaling • Unlock more efficient & redundant policies as system scales Single Management Environment • Introduce new Regions & DC’s after initial deployment • Unlock additional protection polices as system scales
- 25. © 2017, Cloudian, Inc.27 Policy Based Data Protection & Consistency Settings Tunable Consistency Levels Sync or Async IO Acknowledgements Single and Multi Site Support • Granular Control, set at the bucket level • Unlock more efficient protection schemes by adding nodes
- 26. ©28 Hybrid Cloud Tiering Metadata kept on local platform APPLICATIONS S3 ON PREM DATA CENTER TBs to PBs Policy-based • Multicloud storage • Bimodal access
- 27. 29 Versioning and WORM support • Accidental deletion • Ransomware • Rouge admins • Malicious attacks Enabled at the Bucket Level, versioning can provide a safeguard against: When using WORM objects in the bucket cannot be deleted, overwritten or modified from any source or by any command until the retention period has expired. Once locked, the bucket lock policy cannot be deleted from the bucket. The bucket can only be deleted when the retention period for all objects in the bucket has expired and the objects have been deleted
- 28. Hybrid Cloud: Cross Region Replication Target Use Cases Secure Vault – Log all changes to separate Cluster Hyperstore Backup – Backup data & Restore to any point time DR – Protection from Primary Cluster Loss (eg Disasters) v2 RESTORE Objects from Past To New Bucket (eg Jan 1) Photo. jpg ALL new PUTs are replicated to Destination Cluster v1 v2 v3 v1 v2 v3 Source Cluster Destination Cluster Photo.jpg Dec1 Jan1 Feb1 DELETEs has NO Affect on Destination Cluster v1 v2 v3 v1 v2 v3 Delete Marker Jan9 Supported Deployments Hyperstore 6 -> Hyperstore6 Hyperstore 6 -> © 2017, Cloudian, Inc.30
- 29. Built In S3 Client: Bucket & Objects Browser Object Publishing Object Expiration Object Lifecycle Management © 2017, Cloudian, Inc.31
- 30. 32 Encryption Object S3 bucket SSL Object Master key Data keys Server side Encryption key (regular SSE) • Server manages master key and generates per-object key that’s stored in object metadata. • Can be integrated with an external key management system. Object Per-object key Encrypt S3 bucket SSL Customer-provided encryption key (SSE-C) • Encryption key is never stored. • Customer must use same key on PUT and GET. Object Encrypted Object & key S3 bucket Master key AWS SDK Envelope keys Client-side Encryption • Client provides and manages master keys. • With AWS SDK, dynamic “envelope” key generated and used to encrypt object and key. • Encrypted envelope key is sent and stored as object metadata, and checked on retrieval. KMS Support for SSE • Supported with Gemalto KeySecure
- 31. Compression 3 Key Advantages of Utilizing Cloudian HyperStore Compression Faster Speed Higher Compression Rate Choose between 3 different compression algorithms to match speed and compression rate requirements • Enabled at bucket level 1. Reduced Storage Utilization 2. Reduced Network Utilization 3. Improved Replication Speed © 2017, Cloudian, Inc.33
- 32. Multi-Tenancy © 2017, Cloudian, Inc.34
- 33. QoS Requests per Min Storage Bytes Storage Objects Inbound Bytes/Min Outbound Bytes/Min HyperStore Software Defined Storage PROGRAMMABLE APIs USER GROUP If Limit is reached, requests are rejected until next windowDefine Limits – Warning and Max Levels © 2017, Cloudian, Inc.35
- 34. Smart Support HyperStore Telemetrics Global Analytics Proactive Resolution Real Time Monitoring Cloudian Smart Support S3 Operations Resource Usage © 2017, Cloudian, Inc.36
- 36. Unified Storage – Multiple Use Cases © 2017, Cloudian, Inc.38 Backup Snapshots Templates NFSWeb Apps
- 37. NFS Support • NFS Gateway support is offered via HyperFile • Allows access to same object via S3 or NFS • Suitable use cases : • VM templates/snapshots storage • Backup workloads (reading/writing large files) • Media Repository Sharing S3 NFS 39
- 38. CloudStack Integration – SSO •Integrated Management GUI •Integrated provisioning • Store OS images & snapshots • Single Sign-on 40
- 39. •Open standards approach – full S3 compatibility •Store templates, ISO images and snapshots on a multi-tenant hybrid object storage platform •Share data between user projects and virtual machine instances •Built in advanced data protection features including: •Versioning •Erasure Coding •Replication •DR with multi DC support, multiple regions and tunable consistency settings 41 Key Benefits
- 40. Questions?