London Ceph Day Keynote: Building Tomorrow's Ceph
3 likes•1,305 views
This document provides an overview of the history and development of Ceph, an open-source distributed storage system. It discusses Sage Weil's initial research that led to Ceph's creation, the incubation of Ceph at DreamHost, the launch of Inktank to support Ceph's development and adoption, and the current state of Ceph including its growing community and usage in production deployments. It also outlines Weil's vision for Ceph's future, including improving governance, adding new technologies like tiering and erasure coding, and expanding its role in areas like big data and the enterprise storage market.
London Ceph Day Keynote: Building Tomorrow's Ceph
- 1. Building Tomorrow's Ceph Sage Weil
- 11. UCSC research grant ● “Petascale object storage” ● DOE: LANL, LLNL, Sandia ● Scalability ● Reliability ● Performance ● ● Raw IO bandwidth, metadata ops/sec HPC file system workloads ● Thousands of clients writing to same file, directory
- 12. Distributed metadata management â—Ź Innovative design â—Ź Subtree-based partitioning for locality, efficiency â—Ź Dynamically adapt to current workload â—Ź Embedded inodes â—Ź Prototype simulator in Java (2004) â—Ź First line of Ceph code â—Ź Summer internship at LLNL â—Ź High security national lab environment â—Ź Could write anything, as long as it was OSS
- 13. The rest of Ceph ● RADOS – distributed object storage cluster (2005) ● EBOFS – local object storage (2004/2006) ● CRUSH – hashing for the real world (2005) ● Paxos monitors – cluster consensus (2006) → emphasis on consistent, reliable storage → scale by pushing intelligence to the edges → a different but compelling architecture
- 15. Industry black hole â—Ź Many large storage vendors â—Ź â—Ź Proprietary solutions that don't scale well Few open source alternatives (2006) â—Ź â—Ź Limited community and architecture (Lustre) â—Ź â—Ź Very limited scale, or No enterprise feature sets (snapshots, quotas) PhD grads all built interesting systems... â—Ź â—Ź ...and then went to work for Netapp, DDN, EMC, Veritas. They want you, not your project
- 16. A different path â—Ź Change the world with open source â—Ź â—Ź â—Ź Do what Linux did to Solaris, Irix, Ultrix, etc. What could go wrong? License â—Ź â—Ź â—Ź GPL, BSD... LGPL: share changes, okay to link to proprietary code Avoid unsavory practices â—Ź Dual licensing â—Ź Copyright assignment
- 17. Incubation 17
- 19. DreamHost! â—Ź Move back to LA, continue hacking â—Ź Hired a few developers â—Ź Pure development â—Ź No deliverables
- 20. Ambitious feature set â—Ź Native Linux kernel client (2007-) â—Ź Per-directory snapshots (2008) â—Ź Recursive accounting (2008) â—Ź Object classes (2009) â—Ź librados (2009) â—Ź radosgw (2009) â—Ź strong authentication (2009) â—Ź RBD: rados block device (2010)
- 21. The kernel client â—Ź ceph-fuse was limited, not very fast â—Ź Build native Linux kernel implementation â—Ź Began attending Linux file system developer events (LSF) â—Ź â—Ź â—Ź Early words of encouragement from ex-Lustre devs Engage Linux fs developer community as peer Initial attempts merge rejected by Linus â—Ź â—Ź â—Ź Not sufficient evidence of user demand A few fans and would-be users chimed in... Eventually merged for v2.6.34 (early 2010)
- 22. Part of a larger ecosystem â—Ź Ceph need not solve all problems as monolithic stack â—Ź Replaced ebofs object file system with btrfs â—Ź â—Ź Avoid reinventing the wheel â—Ź Robust, well-supported, well optimized â—Ź Kernel-level cache management â—Ź â—Ź Same design goals Copy-on-write, checksumming, other goodness Contributed some early functionality â—Ź Cloning files â—Ź Async snapshots
- 23. Budding community â—Ź #ceph on irc.oftc.net, [email protected] â—Ź Many interested users â—Ź A few developers â—Ź Many fans â—Ź Too unstable for any real deployments â—Ź Still mostly focused on right architecture and technical solutions
- 24. Road to product â—Ź â—Ź DreamHost decides to build an S3-compatible object storage service with Ceph Stability â—Ź â—Ź Focus on core RADOS, RBD, radosgw Paying back some technical debt â—Ź â—Ź â—Ź Build testing automation Code review! Expand engineering team
- 25. The reality â—Ź Growing incoming commercial interest â—Ź Early attempts from organizations large and small â—Ź Difficult to engage with a web hosting company â—Ź No means to support commercial deployments â—Ź Project needed a company to back it â—Ź â—Ź Build and test a product â—Ź â—Ź Fund the engineering effort Support users Bryan built a framework to spin out of DreamHost
- 26. Launch 26
- 28. Do it right ● How do we build a strong open source company? ● How do we build a strong open source community? ● Models? ● ● RedHat, Cloudera, MySQL, Canonical, … Initial funding from DreamHost, Mark Shuttleworth
- 29. Goals â—Ź A stable Ceph release for production deployment â—Ź â—Ź DreamObjects Lay foundation for widespread adoption â—Ź Platform support (Ubuntu, Redhat, SuSE) â—Ź Documentation â—Ź Build and test infrastructure â—Ź Build a sales and support organization â—Ź Expand engineering organization
- 30. Branding ● Early decision to engage professional agency ● ● MetaDesign Terms like ● ● ● “Brand core” “Design system” Project vs Company ● ● ● Shared / Separate / Shared core Inktank != Ceph Aspirational messaging: The Future of Storage
- 31. Slick graphics â—Ź broken powerpoint template 31
- 34. Traction â—Ź Too many production deployments to count â—Ź We don't know about most of them! â—Ź Too many customers (for me) to count â—Ź Growing partner list â—Ź â—Ź Lots of inbound Lots of press and buzz
- 35. Quality â—Ź Increased adoption means increased demands on robust testing â—Ź Across multiple platforms â—Ź Include platforms we don't like â—Ź Upgrades â—Ź â—Ź â—Ź Rolling upgrades Inter-version compatibility Expanding user community + less noise about bugs = a good sign
- 36. Developer community â—Ź Significant external contributors â—Ź First-class feature contributions from contributors â—Ź Non-Inktank participants in daily Inktank stand-ups â—Ź External access to build/test lab infrastructure â—Ź Common toolset â—Ź â—Ź Email (kernel.org) â—Ź â—Ź Github IRC (oftc.net) Linux distros
- 37. CDS: Ceph Developer Summit â—Ź Community process for building project roadmap â—Ź 100% online â—Ź Google hangouts â—Ź Wikis â—Ź Etherpad â—Ź First was this Spring, second is next week â—Ź Great feedback, growing participation â—Ź Indoctrinating our own developers to an open development model
- 38. The Future 38
- 39. Governance How do we strengthen the project community? â—Ź 2014 is the year â—Ź Might formally acknowledge my role as BDL â—Ź Recognized project leads â—Ź RBD, RGW, RADOS, CephFS) â—Ź Formalize processes around CDS, community roadmap â—Ź External foundation?
- 40. Technical roadmap â—Ź How do we reach new use-cases and users â—Ź How do we better satisfy existing users â—Ź How do we ensure Ceph can succeed in enough markets for Inktank to thrive â—Ź Enough breadth to expand and grow the community â—Ź Enough focus to do well
- 41. Tiering ● ● Client side caches are great, but only buy so much. Can we separate hot and cold data onto different storage devices? ● ● ● ● Cache pools: promote hot objects from an existing pool into a fast (e.g., FusionIO) pool Cold pools: demote cold data to a slow, archival pool (e.g., erasure coding) How do you identify what is hot and cold? Common in enterprise solutions; not found in open source scale-out systems → key topic at CDS next week
- 42. Erasure coding â—Ź Replication for redundancy is flexible and fast â—Ź For larger clusters, it can be expensive Storage overhead 3x replication Repair traffic MTTDL (days) 1x 2.3 E10 RS (10, 4) 1.4x 10x 3.3 E13 LRC (10, 6, 5) â—Ź 3x 1.6x 5x 1.2 E15 Erasure coded data is hard to modify, but ideal for cold or read-only objects â—Ź Cold storage tiering â—Ź Will be used directly by radosgw
- 43. Multi-datacenter, geo-replication â—Ź Ceph was originally designed for single DC clusters â—Ź â—Ź â—Ź Synchronous replication Strong consistency Growing demand â—Ź â—Ź â—Ź Enterprise: disaster recovery ISPs: replication data across sites for locality Two strategies: â—Ź use-case specific: radosgw, RBD â—Ź low-level capability in RADOS
- 44. RGW: Multi-site and async replication â—Ź Multi-site, multi-cluster â—Ź â—Ź Zones: radosgw sub-cluster(s) within a region â—Ź â—Ź Regions: east coast, west coast, etc. Can federate across same or multiple Ceph clusters Sync user and bucket metadata across regions â—Ź â—Ź Global bucket/user namespace, like S3 Synchronize objects across zones â—Ź Within the same region â—Ź Across regions â—Ź Admin control over which zones are master/slave
- 45. RBD: simple DR via snapshots â—Ź Simple backup capability â—Ź â—Ź Based on block device snapshots Efficiently mirror changes between consecutive snapshots across clusters â—Ź Now supported/orchestrated by OpenStack â—Ź Good for coarse synchronization (e.g., hours) â—Ź Not real-time
- 46. Async replication in RADOS ● One implementation to capture multiple use-cases ● ● RBD, CephFS, RGW, … RADOS A harder problem ● ● ● Scalable: 1000s OSDs → 1000s of OSDs Point-in-time consistency Three challenges ● Infer a partial ordering of events in the cluster ● Maintain a stable timeline to stream from – ● either checkpoints or event stream Coordinated roll-forward at destination – do not apply any update until we know we have everything that happened before it
- 47. CephFS → This is where it all started – let's get there ● Today ● ● ● QA coverage and bug squashing continues NFS and CIFS now large complete and robust Need ● ● Directory fragmentation ● Snapshots ● ● Multi-MDS QA investment Amazing community effort
- 49. Big data When will be stop talking about MapReduce? Why is “big data” built on such a lame storage model? ● Move computation to the data ● Evangelize RADOS classes ● librados case studies and proof points ● Build a general purpose compute and storage platform
- 50. The enterprise How do we pay for all our toys? ● Support legacy and transitional interfaces ● ● ● iSCSI, NFS, pNFS, CIFS Vmware, Hyper-v Identify the beachhead use-cases ● Only takes one use-case to get in the door ● Earn others later ● Single platform – shared storage resource ● Bottom-up: earn respect of engineers and admins ● Top-down: strong brand and compelling product
- 51. Why we can beat the old guard â—Ź It is hard to compete with free and open source software â—Ź Unbeatable value proposition â—Ź Ultimately a more efficient development model â—Ź It is hard to manufacture community â—Ź Strong foundational architecture â—Ź Native protocols, Linux kernel support â—Ź â—Ź â—Ź Unencumbered by legacy protocols like NFS Move beyond traditional client/server model Ongoing paradigm shift â—Ź Software defined infrastructure, data center
- 52. Thank you, and Welcome!