Scylla Summit 2018: Rebuilding the Ceph Distributed Storage Solution with Seastar
Download as PPTX, PDF•2 likes•2,105 views
This document summarizes plans to rebuild the Ceph distributed storage system using Seastar, a framework for high-performance event-driven applications. Ceph is an open-source distributed storage platform that provides object, block, and file storage at scale. It uses a thread pool model that has limitations around lock contention and context switching. Seastar uses an asynchronous message passing model without locks that could improve Ceph's performance. The plan is to backfill Ceph components starting with critical I/O paths to prioritize basic functionality, then add supporting features later to fully rebuild Ceph on Seastar.
![A message from the alien
▪ std::future versus seastar::future
// in rocksdb
Status SequentialFile::read(size_t n,
char* buf)
{
return alien::submit_to(io_shard,
[n,buf,this]
{
return do_dma_read(n, buf);
}).wait();
}
▪ Dedicated cores for the alien territory](https://image.slidesharecdn.com/rb24gzbps7ilifrpaymk-signature-c70946598b27be0bc87f8b87a668754a6c4a7b459e55f9ea75947283672e7d39-poli-181107215613/85/Scylla-Summit-2018-Rebuilding-the-Ceph-Distributed-Storage-Solution-with-Seastar-19-320.jpg)
Scylla Summit 2018: Rebuilding the Ceph Distributed Storage Solution with Seastar
- 1. Rebuild the Ceph Distributed Storage with Seastar Kefu Chai Software Engineer, Red Hat
- 2. Kefu Chai He is a software engineer who has been focusing on storage systems for over a decade. Currently, he is working on Ceph, an open source, distributed storage solution.
- 3. What is Ceph anyway?
- 4. About Ceph ▪ A platform • Object, block and filesystem ▪ Scalable • from 10 to 10,000 nodes (OSDs) ▪ No single point failure ▪ Free and Open Source (LGPL)
- 6. CRUSH — the “partitioner” of objects
- 8. good not not good enough
- 9. The latency analysis for write path courtesy of Myonngwon Oh, et al., ALCA: Locality-Aware Lock Contention Avoidance for NVMe-Based Scale-out Storage System
- 10. Blank Slide for full size diagram Storage Backend Network Messenger Thread Pool serving requests
- 11. A closer look at the thread pool
- 12. Random R/W performance The performance of random I/O is throttled by CPU
- 13. Blank Slide for full size diagram
- 14. Latencies in I/O path ▪ TCP/IP stack in kernel space ▪ Coarse grained locks ▪ Internal fragmentation ▪ Memory copy for direct-I/O
- 15. Seastar to the Rescue
- 16. Non-seastar ▪ Thread pool ▪ Lock ▪ Share state ▪ TCP/IP in kernel ▪ Callbacks ▪ Multiple-threaded ▪ Fiber ▪ Lockless ▪ Message passing ▪ TCP/IP in userspace (0-copy) ▪ future/promise ▪ single-threaded Seastar
- 18. A message from the alien
- 19. A message from the alien ▪ std::future versus seastar::future // in rocksdb Status SequentialFile::read(size_t n, char* buf) { return alien::submit_to(io_shard, [n,buf,this] { return do_dma_read(n, buf); }).wait(); } ▪ Dedicated cores for the alien territory
- 20. Backfill with priority ▪ Critical path first • Basic I/O paths ▪ Supporting features later • recovery • Metric reporting • …
- 21. Thank You Any Questions ? Please stay in touch [email protected] @tchaik0v