The Enteprise File Fabric and IBM COS | Solution Guide
0 likes•53 views
This document outlines the solution architecture and performance evaluation of Storage Made Easy's Enterprise File Fabric integrated with IBM Cloud Object Storage, detailing configuration steps, functionality, and testing results. The performance tests indicate that the configuration can support organizations of up to 20,000 users with high efficiency, particularly for managing large datasets and ensuring encryption for data security. The document emphasizes the benefits of using IBM COS, including enhanced durability, availability, and effective data governance for enterprises.
The Enteprise File Fabric and IBM COS | Solution Guide
- 1. SOLUTION GUIDE | © Copyright Storage Made Easy® 2019 1 INTRODUCTION This document outlines the solution architecture, configuration steps, functionality and performance characteristics of Storage Made Easy™ Enterprise File Fabric™ software with IBM Cloud Object Storage (COS). A separate guide provides information on standalone desktop applications S3 Drive and S3 Explorer. PRODUCTS TESTED Storage Made Easy software: • Enterprise File Fabric 1705.01 IBM software: • IBM Cloud Object Storage – On Premise (version 3.10.2.72) • IBM Cloud Object Storage – US South (version 3.11.0.89) • IBM Cloud Object Storage – US Region (version 3.11.0.89) CERTIFICATION The certification exercise included installing products and executing functional and performance tests with IBM Cloud Object Storage - on premise and public cloud. The File Fabric product is shipped as a virtual appliance and was provided as an ova image. SOLUTION GUIDE IBM COS - STORAGE MADE EASY ENTERPRISE FILE FABRIC
- 2. SOLUTION GUIDE | © Copyright Storage Made Easy® 2019 2 SOLUTION ARCHITECTURE HARDWARE Client Load Generators 2 Ubuntu Linux Servers • Ubuntu Server 16.04 • VM Specification • RAM: 16GB • CPUs: 8 • HDD: 500GB (SSD) • Test data ranging from 64KB to 10GB Load Balancer HaProxy on CentOS 7.2 • Standard Configuration • VM Specification
- 3. SOLUTION GUIDE | © Copyright Storage Made Easy® 2019 3 Enterprise File Fabric Services • 2 - CentOS 7.2 Linux Virtual Machines • SME standard OVA images for VMware • Specifications • RAM: 12GB • CPUs: 8 • HDD: 2x 36GB(SSD) • OS & Scratch Drive Enterprise File Fabric Metadata Server • CentOS 7.2 Linux Virtual Machine • MariaDB 5.5 • SME standard OVA image for VMware • Specifications • RAM: 16GB • CPUs: 8 • HDD: 2x 36GB(SSD) • OS & Database Partition IBM CLOUD OBJECT STORAGE Fabric was tested against the following COS storage backends: •IBM Cloud Object Storage – On Premise (version 3.10.2.72) •IBM Cloud Object Storage – US South (version 3.11.0.89) •IBM Cloud Object Storage – US Region (version 3.11.0.89) ENTERPRISE FILE FABRIC CONFIGURATION The Enterprise File Fabric platform is a multi-tenant system allowing for thousands of users across multiple organizations. Testing was of a single organization configured exclusively with IBM COS for user data.
- 4. SOLUTION GUIDE | © Copyright Storage Made Easy® 2019 4 ORGANIZATION ACCOUNT CREATION Log in as the Enterprise File Fabric Administrator, select from Menu, “Add a User” Organizations are Users with an “Organization Package” specified FILE FABRIC CONFIGURATION
- 5. SOLUTION GUIDE | © Copyright Storage Made Easy® 2019 5 ADDING IBM COS ON PREMISE OR CLOUD OFFERING AS STORAGE PROVIDER The first time an organization logs into an account, a wizard guides the addition of storage providers approved in the Organization’s Cloud Package. The Enterprise File Fabric supports over 60 backend providers. IBM COS endpoint, Access & Secret Keys are entered: FILE FABRIC CONFIGURATION
- 6. SOLUTION GUIDE | © Copyright Storage Made Easy® 2019 6 File Fabric will then index existing objects and metadata: Storage is then available for users: FILE FABRIC CONFIGURATION
- 7. SOLUTION GUIDE | © Copyright Storage Made Easy® 2019 7 CREATING ACCOUNTS FOR LOAD TESTING Storage Made Easy Enterprise File Fabric supports LDAP, Active Directory, SAML, and local user accounts. Local accounts can be created through API, however for testing Active Directory was utilized. Authorization Systems can be added under Organization -> “Auth Systems”. Active Directory can be configured against a local AD system or remotely via the SME ADproxy. Once added, users can be imported under Organization -> “Users”. FILE FABRIC CONFIGURATION
- 8. SOLUTION GUIDE | © Copyright Storage Made Easy® 2019 8 Importing of users: Users and Groups/Roles auto-registered or filtered for import based on name or group membership. FILE FABRIC CONFIGURATION
- 9. SOLUTION GUIDE | © Copyright Storage Made Easy® 2019 9 PERFORMANCE TEST 1 - 10,000 USER OFFICE ORGANIZATION This test simulated a large typical SME organization. The organization has 10,000 registered users with typical file size falling between 1MB-4MB and 2-4 opens/saves per hour. In order to meet stated requirements, the Enterprise File Fabric will need to respond to between 20,000 and 40,000 requests in an hour. OFFICE RESULTS SET Tested configuration was able to exceed the requirements of a 10,000 user organization. The configuration as tested was suitable for an organization of over 20,000 users. Peak uploads & download of files in the 1-4MB range exceeded 60,000 operations in one hour. Based on the result data, the number of concurrent requests on the File Fabric nodes would be in the range of 8-16 requests for a 10,000 user organization. Bandwidth usage for 4MB files peaked around 75MB/s, allowing for use of 1Gb networking between instances, clients and storage. However, 10Gb networking would be preferred for larger organizations or organizations with a larger average file size. All testing was conducted with encryption both In-Flight and At-Rest. In-Flight encryption was enabled via TLS1.2 using an AES-256 Cipher, and was utilized at all data hops: • Client to Load Balancer • Load Balancer to File Fabric • File Fabric to IBM COS Storage Encryption did add some latency to testing and increased CPU demands. CPU utilization was below 90% on all download tests and mixed workload testing; however, 100% write tests with 32 concurrency did exceed 90% CPU utilization. Since the Enterprise File Fabric has a scale-out web component, accommodating larger organizations is a simple matter of adding more instances. TESTING PARAMETERS ✦ Cycle between 1, 2, 4, 8, 16 and 32 open connections ✦ File sizes between 256KB & 64MB • 256KB 62,000 Reads/Writes - 35 % of Operations •1MB 62,000 Reads/ Writes - 35 % of Operations 4MB 31,000 Reads/ Writes - 18 % of Operations •16MB 13,000 Reads/ W r i t e s - 7 % o f Operations •64MB 10,000 Reads/ W r i t e s - 5 % o f Operations
- 10. SOLUTION GUIDE | © Copyright Storage Made Easy® 2019 10 PERFORMANCE TEST 1 - 10,000 USER OFFICE ORGANIZATION
- 11. SOLUTION GUIDE | © Copyright Storage Made Easy® 2019 11 PERFORMANCE TEST 1 - 10,000 USER OFFICE ORGANIZATION
- 12. SOLUTION GUIDE | © Copyright Storage Made Easy® 2019 12 PERFORMANCE TEST 2 - MEDIA COMPANIES Some organizations have larger datasets than standard office workers. These organizations may have very large video files, healthcare images, or other atypical workloads. These larger files take advantage of Multipart Uploads when used in conjunction with IBM COS. MPU is an important feature of the S3 API as it allows larger files to avoid being chunked in a proprietary format that could render them unusable to other data accessors. While the Enterprise File Fabric can chunk data if MPU is not enabled, it is preferred use the native functions of IBM COS where available such as: • Multipart Uploads (MPU) • SecureSlice • Server Side Encryption (SSE-C) M-Stream™ File Transfer Acceleration, a capability of the Enterprise File Fabric™, platform, splits files into parts that are streamed in parallel from one storage server to another. It can dramatically speed up uploads, downloads and server-to- server copying of objects. TESTING PARAMETERS ✦ Cycle between 1, 2, 4, 8, 16 and 32 open connections ✦ File sizes over 16MB ✦ S3 Native MPU Chunking at 100MB increments for files over 105MB MEDIA FILES RESULTS SET During testing of larger media files, the client systems became the bottleneck in sending or receiving data. Each workload driver was able to push close to 250MB/s before the workload generator utilized all CPU time. Further testing with the TLS Encryption disabled showed increased network performance with large files. All testing was conducted with encryption both In-Flight and At-Rest. In-Flight encryption was enabled via TLS 1.2 using an AES-256 Cipher, and was utilized at all data hops: • Client to Load Balancer • Load Balancer to File Fabric • Enterprise File Fabric to IBM COS Storage To bypass the client bottleneck, a transfer of a single 4 GB file was also made from IBM COS cloud (US South) to IBM COS on-premises. With M-Stream™ active 4 threads saturated the 1 GB connection to the Internet - an optimum transfer. Encryption did add some latency to testing and increased CPU demands. CPU utilization was below 90% on all large media tests with the bottleneck coming from the client nodes and not the storage or File Fabric. Testing files above 2GB did not show any significant change in bandwidth vs 2GB. As such, larger files will take more time to transfer and will move at the rates indicated for 2GB.
- 13. SOLUTION GUIDE | © Copyright Storage Made Easy® 2019 13 PERFORMANCE TEST 2 - LARGE MEDIA COMPANIES
- 14. SOLUTION GUIDE | © Copyright Storage Made Easy® 2019 14 CONCLUSION The performance characteristics of the Storage Made Easy Enterprise File Fabric are largely dependent on the backend storage provider. Lackluster performance arising from lack of compatibility with the Amazon S3 API, to inability to fulfill concurrent requests often arise when the backend storage is not a mature offering. IBM Cloud Object Storage excelled when configured as a backend provider for Storage Made Easy Enterprise File Fabric. During testing all S3 API based features of the Enterprise File Fabric were enabled and tested satisfactorily against On-Premises and Cloud based COS endpoints. Advanced S3 API features such as multipart upload (MPU) provide native storage of files as objects without the need for proprietary chunking. Ranged read requests allow for enhanced performance during file downloads when used with large objects. IBM COS additionally supports offloading of Data-at- Rest encryption from the File Fabric via S3 API SSE-C or SecureSlice enablements on vaults. Enterprises choose to use IBM Cloud Object Storage because of its enhanced durability, availability and cost per terabyte vs traditional storage. As such, IBM COS is an excellent pairing choice for enterprises with as little as 25 users to over 20,000 users with the Storage Made Easy Enterprise File Fabric. The File Fabric removes all complexity from using object based storage and the associated APIs. Users enjoy simple access to storage on any device in the office or when working remotely. Administrator’s have peace of mind from the enhanced governance ABOUT STORAGE MADE EASY® The Storage Made Easy Enterprise File Fabric™ enables IT to regain control of "cloud data sprawl" to unify on-premises and on-cloud files into a single easily managed converged infrastructure with policy-based controls for governance, audit and security. The File Fabric solution provides a “blanket” enterprises wrap around all their data: on premises, within public and private clouds, as well as on third-party software vendor clouds (SharePoint Online and Salesforce for example). Customers use the SME File Fabric product to unify corporate data to enable workgroup collaboration with universal controls for security, encryption, audit, and data access. Companies will have access to a greater understanding of where and how sensitive data is stored, transferred, and accessible with particular focus on how companies are able to police and audit data to ensure enforcement of the General Data Protection Regulations (GDPR) in addition to other regional data legislation. USA +1 415 477 1053 UK +44 20 8212 7316 EU +41 435 080 078 [email protected] Free hosted and enterprise trials available from www.storagemadeeasy.com @SMEStorage /StorageMadeEasy /company/storage-made-easy