Ceph Object Store
0 gefällt mir7,266 aufrufe
Das Dokument erklärt die Grundlagen und Architektur des Ceph Object Store, einem skalierbaren, verteilten Speichersystem, das für hohe Leistung, Zuverlässigkeit und Skalierbarkeit ausgelegt ist. Es behandelt Komponenten wie OSD-Daemons, den CRUSH-Algorithmus zur intelligenten Objektverteilung sowie Hardware- und Netzwerkvirtualisierung. Zudem werden Benchmarks, spezifische Netzwerk-Konfigurationen und Empfehlungen zur Nutzung von Ceph, insbesondere in Bezug auf CephFS, diskutiert.
![IEEE 802.3ad != IEEE 802.3ad
[node01] > iperf -s -B node01.ceph-cluster
[node02] > iperf -c node01.ceph-cluster -P 2
[node03] > iperf -c node01.ceph-cluster -P 2
------------------------------------------------------------
Server listening on TCP port 5001
Binding to local address node01.ceph-cluster
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 10.102.5.11 port 5001 connected with 10.102.5.12 port 49412
[ 5] local 10.102.5.11 port 5001 connected with 10.102.5.12 port 49413
[ 6] local 10.102.5.11 port 5001 connected with 10.102.5.13 port 59947
[ 7] local 10.102.5.11 port 5001 connected with 10.102.5.13 port 59946
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 342 MBytes 286 Mbits/sec
[ 5] 0.0-10.0 sec 271 MBytes 227 Mbits/sec
[SUM] 0.0-10.0 sec 613 MBytes 513 Mbits/sec
[ 6] 0.0-10.0 sec 293 MBytes 246 Mbits/sec
[ 7] 0.0-10.0 sec 338 MBytes 283 Mbits/sec
[SUM] 0.0-10.0 sec 631 MBytes 529 Mbits/sec](https://image.slidesharecdn.com/7bc9hl1xqfok1u5exbrj-signature-54bf449e9c3ce5ffd4b9e1d7f3e697c04a265e076e357e65ba3b3ef0a2e93254-poli-141126174132-conversion-gate02/85/Ceph-Object-Store-34-320.jpg)
![IEEE 802.3ad != IEEE 802.3ad
[node01] > iperf -s -B node01.ceph-cluster
[node02] > iperf -c node01.ceph-cluster -P 2
[node03] > iperf -c node01.ceph-cluster -P 2
------------------------------------------------------------
Server listening on TCP port 5001
Binding to local address node01.ceph-cluster
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 10.102.5.11 port 5001 connected with 10.102.5.12 port 49412
[ 5] local 10.102.5.11 port 5001 connected with 10.102.5.12 port 49413
[ 6] local 10.102.5.11 port 5001 connected with 10.102.5.13 port 59947
[ 7] local 10.102.5.11 port 5001 connected with 10.102.5.13 port 59946
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 342 MBytes 286 Mbits/sec
[ 5] 0.0-10.0 sec 271 MBytes 227 Mbits/sec
[SUM] 0.0-10.0 sec 613 MBytes 513 Mbits/sec
[ 6] 0.0-10.0 sec 293 MBytes 246 Mbits/sec
[ 7] 0.0-10.0 sec 338 MBytes 283 Mbits/sec
[SUM] 0.0-10.0 sec 631 MBytes 529 Mbits/sec ???](https://image.slidesharecdn.com/7bc9hl1xqfok1u5exbrj-signature-54bf449e9c3ce5ffd4b9e1d7f3e697c04a265e076e357e65ba3b3ef0a2e93254-poli-141126174132-conversion-gate02/85/Ceph-Object-Store-35-320.jpg)
Ceph Object Store
- 1. Ceph Object Store oder: Wie speichert man terabyteweise Dokumente Daniel Schneller [email protected] @dschneller
- 6. Wo kamen wir her?
- 7. Warum wollten wir da weg?
- 9. Und der Weg?
- 10. Ceph Grundlagen
- 11. “Unified, distributed storage system designed for excellent performance, reliability and scalability”
- 12. Stark skalierbar Commodity Hardware Kein Single Point of Failure
- 13. Ceph Komponenten
- 14. OSD Daemons Object Storage Device Daemons
- 15. CRUSH Algorithmus Intelligente Objektverteilung ohne zentrale Metadaten
- 16. RADOS Reliable Autonomous Distributed Object Store
- 17. Objekte
- 18. Data Pools Sammelbecken für Objekte mit gleichen Anforderungen
- 20. Placement Groups
- 23. Monitors Erste Anlaufstelle für Clients
- 24. Hardware Setup
- 25. Storage Virtualization Bare Metal Hardware Compute Virtualization Network Virtualization Virtual Infrastructure Application
- 26. Storage Virtualization Bare Metal Hardware Compute Virtualization Network Virtualization Virtual Infrastructure Application
- 29. Storage Disk I/O pro Node
- 32. Netzwerk
- 33. IEEE 802.3ad != IEEE 802.3ad > cat /etc/network/interfaces ... auto bond2 iface bond2 inet manual bond-slaves p2p3 p2p4 # interfaces to bond bond-mode 802.3ad # activate LACP bond-miimon 100 # monitor link health bond-xmit_hash_policy layer3+4 # use Layer 3+4 for link selection pre-up ip link set dev bond2 mtu 9000 # set Jumbo Frames auto vlan-ceph-clust iface vlan-ceph-clust inet static pre-up ip link add link bond2 name vlan-ceph-clust type vlan id 105 pre-up ip link set dev vlan-ceph-clust mtu 9000 # Jumbo Frames post-down ip link delete vlan-ceph-clust address ... netmask ... network ... broadcast ... ...
- 34. IEEE 802.3ad != IEEE 802.3ad [node01] > iperf -s -B node01.ceph-cluster [node02] > iperf -c node01.ceph-cluster -P 2 [node03] > iperf -c node01.ceph-cluster -P 2 ------------------------------------------------------------ Server listening on TCP port 5001 Binding to local address node01.ceph-cluster TCP window size: 85.3 KByte (default) ------------------------------------------------------------ [ 4] local 10.102.5.11 port 5001 connected with 10.102.5.12 port 49412 [ 5] local 10.102.5.11 port 5001 connected with 10.102.5.12 port 49413 [ 6] local 10.102.5.11 port 5001 connected with 10.102.5.13 port 59947 [ 7] local 10.102.5.11 port 5001 connected with 10.102.5.13 port 59946 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 342 MBytes 286 Mbits/sec [ 5] 0.0-10.0 sec 271 MBytes 227 Mbits/sec [SUM] 0.0-10.0 sec 613 MBytes 513 Mbits/sec [ 6] 0.0-10.0 sec 293 MBytes 246 Mbits/sec [ 7] 0.0-10.0 sec 338 MBytes 283 Mbits/sec [SUM] 0.0-10.0 sec 631 MBytes 529 Mbits/sec
- 35. IEEE 802.3ad != IEEE 802.3ad [node01] > iperf -s -B node01.ceph-cluster [node02] > iperf -c node01.ceph-cluster -P 2 [node03] > iperf -c node01.ceph-cluster -P 2 ------------------------------------------------------------ Server listening on TCP port 5001 Binding to local address node01.ceph-cluster TCP window size: 85.3 KByte (default) ------------------------------------------------------------ [ 4] local 10.102.5.11 port 5001 connected with 10.102.5.12 port 49412 [ 5] local 10.102.5.11 port 5001 connected with 10.102.5.12 port 49413 [ 6] local 10.102.5.11 port 5001 connected with 10.102.5.13 port 59947 [ 7] local 10.102.5.11 port 5001 connected with 10.102.5.13 port 59946 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 342 MBytes 286 Mbits/sec [ 5] 0.0-10.0 sec 271 MBytes 227 Mbits/sec [SUM] 0.0-10.0 sec 613 MBytes 513 Mbits/sec [ 6] 0.0-10.0 sec 293 MBytes 246 Mbits/sec [ 7] 0.0-10.0 sec 338 MBytes 283 Mbits/sec [SUM] 0.0-10.0 sec 631 MBytes 529 Mbits/sec ???
- 36. Messen! …und die Ergebnisse verstehen
- 37. CenterDevice
- 38. Gesamtarchitektur Node 1 OSD 1 … … Node 2 … … … Node 3 … … … Node 4 … OSD 48 … Bare Metal Ceph
- 39. Gesamtarchitektur Node 1 OSD 1 … … Rados GW Node 2 … … … Rados GW Node 3 … … … Rados GW Node 4 … OSD 48 … Rados GW Bare Metal Ceph
- 40. Gesamtarchitektur Node 1 OSD 1 … … Rados GW Node 2 … … … Rados GW Node 3 … … … Rados GW Node 4 … OSD 48 … Rados GW VM 1 HAProxy VM 1 HAProxy VM 1 HAProxy VM … HAProxy VMs Bare Metal Ceph
- 41. Gesamtarchitektur Node 1 OSD 1 … … Rados GW Node 2 … … … Rados GW Node 3 … … … Rados GW Node 4 … OSD 48 … Rados GW VMs Bare Metal VM 1 Ceph HAProxy CenterDevice Swift VM 1 HAProxy CenterDevice Swift VM 1 HAProxy CenterDevice Swift VM … HAProxy CenterDevice Swift
- 42. Vorteile
- 43. Nachteile
- 44. Caveats
- 45. CephFS Not recommended for production data.
- 46. Scrubbing Integrität hat ihren Preis. Aber man kann handeln!
- 47. Zukunft
- 48. Rados Gateway Ceph Caching Tier SSD based Journaling 10GBit/s Networking
- 49. Zum Schluss