Object Storage in a Cloud-Native Container Envirnoment
Download as PPTX, PDF•2 likes•1,013 views
The document introduces MinIO, an object storage server compatible with Amazon S3, emphasizing its features such as being written in Golang and designed for cloud computing environments. It explains the distinction between stateless and stateful containers, advocating for a hybrid application stack that utilizes various technologies and dynamic compute resource management. The presentation also details infrastructure setup and deployment of MinIO instances to manage both immutable and mutable data efficiently.
Object Storage in a Cloud-Native Container Envirnoment
- 2. ● Intro about myself / Minio ● Modern Application Stack ● Stateful Containers ● Object Storage CONTENTS 2
- 3. ● Frank Wessels ○ Software dev in medical imaging ○ CT & MR ○ 2D & 3D / GPU ● Last couple of years in Cloud Computing ● Now with Minio (Palo Alto) 3
- 4. ● Object Storage Server ● Amazon S3 Compatible ● Written in Golang / Apache License 2.0 ● Founder: Anand Babu Periasamy ○ Formerly GlusterFS ■ Distributed File System ■ Now part of Red Hat 4
- 5. MINIO SERVER MINIO CLIENT MINIO SDK 5
- 6. 6
- 7. LEGACY STACK (LAMP) Linux Apache MySQL PHP MODERN STACK Docker / Kubernetes Dynamic compute using Lambda Various storage technologies (Redis / ElasticSearch / Object Storage / etc) Stateless vs Stateful (next slide) 7
- 8. STATELESS CONTAINERS Application logic Immutable (no state) Think “functional” programming or SSA Scalable instances: single → thousands → millions STATEFUL CONTAINERS Use mix of technologies depending on needs 8 PARADIGM: MANY STATELESS + FEW STATEFUL
- 9. Mix of languages best suited / Hybrid Environment, eg: Javascript and Golang Node.js and Java Orchestration software (Docker Swarm / Kubernetes / Mesosphere) Everything automated Automatic resource management Isolation Allocated per tenant 9
- 10. MINIO Provision object storage as containers Infrastructure Containers Orchestration Docker Swarm 10
- 11. 11
- 12. etc.. DIFFERENT KINDS OF STATE / DATA 12 SQL AMQP Kafka MESSAGES TEXTJSONLARGE BLOBS KV DATA Consul etcd CONFIG DATA
- 13. REDIS AS YOUR KEY-VALUE STORE Typically run on large memory system (256 GB or more) Persisted to disk Clustered version available FUTURE Imagine persistent memory (no more saving!) Question: who stores persistent data on Redis now? 13
- 14. STATELESS STATEFUL API Server (golang) Webserver (node.js/angular ) (users) (index)(blobs) (search) (text search) 14
- 15. 15
- 16. USE OBJECT STORAGE FOR UNSTRUCTURED DATA/BLOBS EXAMPLES: 16
- 17. IMMUTABLE DATA (Logs, Video, Archive, Documents, VMs / Containers) MUTABLE DATA (Index, Metadata) 17
- 18. Source: Human-Computer Interaction & Knowledge Discovery
- 19. 1 BINARY / 3 FLAVORS FS backend $ minio server /dir XL backend with Erasure code & bitrot protection $ minio server /disk1 /disk2 /disk3 /disk4 ... /disk16 Distributed up to 16 servers $ minio server host1:/disk host2:/disk host3:/disk ... host16:/disk 19
- 20. OBJECT ERASURE-CODED OVER 16 DRIVES 20
- 21. Minio 2 Minio 3 Minio 16 JBOD Erasure Code (2x2, 4x4, 6x6, 8x8) - Maximum 16 Disks / Setup JBODJBODJBOD Minio 1 21
- 22. Minio constrained by design Don’t worry about failed drives, designed to let things fail 16 disk/server: can stand up to 8 failed Migrate data to new servers as servers age (think 5 years) Deploy many Minio instances Use KV store to keep track of Minio instance22
- 23. 23
- 24. github.com/minio/minio github.com/minio/mc github.com/minio/minio-go / minio-java / minio-js /minio-hs / more… Auxiliary repos github.com/minio/blake2b-simd github.com/minio/sha256-simd github.com/minio/asm2plan9s github.com/minio/dsync 24
- 25. THANK YOU // QUESTIONS? 25 minio.io slack.minio.io @minio
- 26. APPENDIX 26
- 27. SMC 5018A-AR12L (Intel Atom) SMC 1U SoC Atom C2750 platform with 12 x 3.5” drive bays Cisco UCS C240 M4 Rack Server Cisco 2U DP E5-2600v3 platform with 12 x 3.5” drive bays Quanta Grid D51B-2U (OCP Compliant) Quanta 2U DP E5-2600v3 platform with 12 x 3.5” drive bays Intel® Server System R2312WTTYSR Intel 2U DP E5-2600v3 platform with 12 x 3.5” drive bays Cisco UCS C240 27
- 28. SAS / SATANVMe / SSD RAID HGST Open Ethernet Drive JBOD 28
- 29. 29
- 30. You may have just saved us about 48 hours of configuration hell. Thanks to you and the whole Minio team! - @jacobgadikian @minio looks very promising! I have a term for it: Apple of object storage. Not only it just works but it’s beautiful. - @dharmeshkakadia @minio it's YOU who deserve the thanks for all your support of the Go community in the open and behind the scenes. - @bketelsen Look at @minio. They look for people with passion and who are excited about the tech they are building. Tech can be learned, passion can't. - @goinggodotnet I don't cry... Be positive. It's a time to upgrade Deis v2. Bye Ceph. Hello Minio. - @monamour555 COMMUNITY 30
Editor's Notes
- #30: New title: Minio Usage