SlideShare a Scribd company logo

Gluster for sysadmins

•
Gluster.org, profile picture
Gluster.org

GlusterFS is a POSIX-compliant distributed file system that aggregates various storage bricks across commodity servers into a single global namespace. It has no single point of failure or performance bottleneck. Red Hat Storage is an enterprise implementation of GlusterFS. It uses a elastic hashing algorithm to distribute files across bricks without a centralized metadata server. Various translators and volumes types provide features like replication, distribution, striping, and geo-replication. Administration involves adding peers, creating and managing distributed volumes, and manipulating bricks.

Technology
1 of 61
Downloaded 77 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
Demystifying Gluster GlusterFS and RHS for the SysAdmin Dustin L. Black, RHCA Sr. Technical Account Manager, Red Hat 2012-11-08
DUSTIN L. BLACK, RHCA2 #whoami ● Systems and Infrastructure Geek ● Decade+ of Linux, UNIX, networking ● <notacoder/> ● Believe in Open Source Everything ● Sr. Technical Account Manager, Red Hat GSS ● dustin@redhat.com
DUSTIN L. BLACK, RHCA3 #whatis TAM ● Premium named-resource support ● Proactive and early access ● Regular calls and on-site engagements ● Customer advocate within Red Hat and upstream ● Multi-vendor support coordinator ● High-touch access to engineering ● Influence for software enhancements ● NOT Hands-on or consulting
DUSTIN L. BLACK, RHCA4 Agenda ● Technology Overview ● Scaling Up and Out ● A Peek at GlusterFS Logic ● Redundancy and Fault Tolerance ● Data Access ● General Administration ● Use Cases ● Common Pitfalls
Demystifying Gluster GlusterFS and RHS for the SysAdmin Technology Overview
DUSTIN L. BLACK, RHCA6 What is GlusterFS? ● POSIX-Like Distributed File System ● No Metadata Server ● Network Attached Storage (NAS) ● Heterogeneous Commodity Hardware ● Aggregated Storage and Memory ● Standards-Based – Clients, Applications, Networks ● Flexible and Agile Scaling ● Capacity – Petabytes and beyond ● Performance – Thousands of Clients ● Single Global Namespace
DUSTIN L. BLACK, RHCA7 What is Red Hat Storage? ● Enterprise Implementation of GlusterFS ● Software Appliance ● Bare Metal Installation ● Built on RHEL + XFS ● Subscription Model ● Storage Software Appliance ● Datacenter and Private Cloud Deployments ● Virtual Storage Appliance ● Amazon Web Services Public Cloud Deployments
DUSTIN L. BLACK, RHCA8 RHS vs. Traditional Solutions ● A basic NAS has limited scalability and redundancy ● Other distributed filesystems limited by metadata ● SAN is costly & complicated but high performance & scalable ● RHS ● Linear Scaling ● Minimal Overhead ● High Redundancy ● Simple and Inexpensive Deployment
Demystifying Gluster GlusterFS and RHS for the SysAdmin Technology Stack
DUSTIN L. BLACK, RHCA10 Terminology ● Brick ● A filesystem mountpoint ● A unit of storage used as a GlusterFS building block ● Translator ● Logic between the bits and the Global Namespace ● Layered to provide GlusterFS functionality ● Volume ● Bricks combined and passed through translators ● Node / Peer ● Server running the gluster daemon and sharing volumes
DUSTIN L. BLACK, RHCA11 Foundation Components ● Private Cloud (Datacenter) ● Common Commodity x86_64 Servers ● RHS: Hardware Compatibility List (HCL) ● Public Cloud ● Amazon Web Services (AWS) ● EC2 + EBS
DUSTIN L. BLACK, RHCA12 Disk, LVM, and Filesystems ● Direct-Attached Storage (DAS) -or- ● Just a Bunch Of Disks (JBOD) ● Hardware RAID ● RHS: RAID 6 required ● Logical Volume Management (LVM) ● XFS, EXT3/4, BTRFS ● Extended attributes support required ● RHS: XFS required
DUSTIN L. BLACK, RHCA13 Gluster Components ● glusterd ● Elastic volume management daemon ● Runs on all export servers ● Interfaced through gluster CLI ● glusterfsd ● GlusterFS brick daemon ● One process for each brick ● Managed by glusterd
DUSTIN L. BLACK, RHCA14 Gluster Components ● glusterfs ● NFS server daemon ● FUSE client daemon ● mount.glusterfs ● FUSE native mount tool ● gluster ● Gluster Console Manager (CLI)
DUSTIN L. BLACK, RHCA15 Data Access Overview ● GlusterFS Native Client ● Filesystem in Userspace (FUSE) ● NFS ● Built-in Service ● SMB/CIFS ● Samba server required ● Unified File and Object (UFO) ● Simultaneous object-based access
DUSTIN L. BLACK, RHCA16 Putting it All Together
Demystifying Gluster GlusterFS and RHS for the SysAdmin Scaling
DUSTIN L. BLACK, RHCA18 Scaling Up ● Add disks and filesystems to a node ● Expand a GlusterFS volume by adding bricks XFS
DUSTIN L. BLACK, RHCA19 Scaling Out ● Add GlusterFS nodes to trusted pool ● Add filesystems as new bricks
Demystifying Gluster GlusterFS and RHS for the SysAdmin Under the Hood
DUSTIN L. BLACK, RHCA21 Elastic Hash Algorithm ● No central metadata ● No Performance Bottleneck ● Eliminates risk scenarios ● Location hashed intelligently on path and filename ● Unique identifiers, similar to md5sum ● The “Elastic” Part ● Files assigned to virtual volumes ● Virtual volumes assigned to multiple bricks ● Volumes easily reassigned on the fly
DUSTIN L. BLACK, RHCA22 Translators
Demystifying Gluster GlusterFS and RHS for the SysAdmin Distribution and Replication
DUSTIN L. BLACK, RHCA24 Distributed Volume ● Files “evenly” spread across bricks ● File-level RAID 0 ● Server/Disk failure could be catastrophic
DUSTIN L. BLACK, RHCA25 Replicated Volume ● Copies files to multiple bricks ● File-level RAID 1
DUSTIN L. BLACK, RHCA26 Distributed Replicated Volume ● Distributes files across replicated bricks ● RAID 1 plus improved read performance
DUSTIN L. BLACK, RHCA27 Geo Replication ● Asynchronous across LAN, WAN, or Internet ● Master-Slave model -- Cascading possible ● Continuous and incremental ● Data is passed between defined master and slave only
DUSTIN L. BLACK, RHCA28 Replicated Volumes vs Geo-replication Replicated Volumes Geo-replication Mirrors data across clusters Mirrors data across geographically distributed clusters Provides high-availability Ensures backing up of data for disaster recovery Synchronous replication (each and every file operation is sent across all the bricks) Asynchronous replication (checks for the changes in files periodically and syncs them on detecting differences)
Demystifying Gluster GlusterFS and RHS for the SysAdmin Layered Functionality
DUSTIN L. BLACK, RHCA30 Striped Volumes ● Individual files split among bricks ● Similar to RAID 0 ● Limited Use Cases – HPC Pre/Post Processing
DUSTIN L. BLACK, RHCA31 Distributed Striped Volume ● Files striped across two or more nodes ● Striping plus scalability
DUSTIN L. BLACK, RHCA32 Striped Replicated Volume ● RHS 2.0 / GlusterFS 3.3+ ● Similar to RAID 10 (1+0)
DUSTIN L. BLACK, RHCA33 Distributed Striped Replicated Volume ● RHS 2.0 / GlusterFS 3.3+ ● Limited Use Cases – Map Reduce
Demystifying Gluster GlusterFS and RHS for the SysAdmin Data Access
DUSTIN L. BLACK, RHCA35 GlusterFS Native Client (FUSE) ● FUSE kernel module allows the filesystem to be built and operated entirely in userspace ● Specify mount to any GlusterFS node ● Native Client fetches volfile from mount server, then communicates directly with all nodes to access data ● Recommended for high concurrency and high write performance ● Load is inherently balanced across distributed volumes
DUSTIN L. BLACK, RHCA36 NFS ● Standard NFS v3 clients ● Note: Mount with vers=3 option ● Standard automounter is supported ● Mount to any node, or use a load balancer ● GlusterFS NFS server includes Network Lock Manager (NLM) to synchronize locks across clients ● Better performance for reading many small files from a single client ● Load balancing must be managed externally
DUSTIN L. BLACK, RHCA37 SMB/CIFS ● GlusterFS volume is first mounted with the Native Client ● Redundantly on the GlusterFS peer -or- ● On an external server ● Native mount point is then shared via Samba ● Must be setup on each node you wish to connect to via CIFS ● Load balancing must be managed externally
Demystifying Gluster GlusterFS and RHS for the SysAdmin General Administration
DUSTIN L. BLACK, RHCA39 Preparing a Brick # lvcreate -L 100G -n lv_brick1 vg_server1 # mkfs -t xfs -i size=512 /dev/vg_server1/lv_brick1 # mkdir /brick1 # mount /dev/vg_server1/lv_brick1 /brick1 # echo '/dev/vg_server1/lv_brick1 /brick1 xfs defaults 1 2' >> /etc/fstab
DUSTIN L. BLACK, RHCA40 Adding Nodes (peers) and Volumes gluster> peer probe server3 gluster> peer status Number of Peers: 2 Hostname: server2 Uuid: 5e987bda-16dd-43c2-835b-08b7d55e94e5 State: Peer in Cluster (Connected) Hostname: server3 Uuid: 1e0ca3aa-9ef7-4f66-8f15-cbc348f29ff7 State: Peer in Cluster (Connected) gluster> volume create my-dist-vol server2:/brick2 server3:/brick3 gluster> volume info my-dist-vol Volume Name: my-dist-vol Type: Distribute Status: Created Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: server2:/brick2 Brick2: server3:/brick3 gluster> volume start my-dist-vol Distributed Volume Peer Probe
DUSTIN L. BLACK, RHCA41 Distributed Striped Replicated Volume gluster> volume create test-volume replica 2 stripe 2 transport tcp server1:/exp1 server1:/exp2 server2:/exp3 server2:/exp4 server3:/exp5 server3:/exp6 server4:/exp7 server4:/exp8 Multiple bricks of a replicate volume are present on the same server. This setup is not optimal. Do you still want to continue creating the volume? (y/n) y Creation of volume test-volume has been successful. Please start the volume to access data. <- test-volume<- test-volume <- distributed files -><- distributed files -> stripe 2stripe 2 replica 2replica 2
DUSTIN L. BLACK, RHCA42 Distributed Striped Replicated Volume gluster> volume info test-volume Volume Name: test-volume Type: Distributed-Striped-Replicate Volume ID: 8f8b8b59-d1a1-42fe-ae05-abe2537d0e2d Status: Created Number of Bricks: 2 x 2 x 2 = 8 Transport-type: tcp Bricks: Brick1: server1:/exp1 Brick2: server2:/exp3 Brick3: server1:/exp2 Brick4: server2:/exp4 Brick5: server3:/exp5 Brick6: server4:/exp7 Brick7: server3:/exp6 Brick8: server4:/exp8 gluster> volume create test-volume stripe 2 replica 2 transport tcp server1:/exp1 server2:/exp3 server1:/exp2 server2:/exp4 server3:/exp5 server4:/exp7 server3:/exp6 server4:/exp8 Creation of volume test-volume has been successful. Please start the volume to access data.
DUSTIN L. BLACK, RHCA43 Manipulating Bricks in a Volume gluster> volume add-brick my-dist-vol server4:/brick4 gluster> volume remove-brick my-dist-vol server2:/brick2 start gluster> volume remove-brick my-dist-vol server2:/brick2 status Node Rebalanced-files size scanned failures status --------- ----------- ----------- ----------- ----------- ------------ localhost 16 16777216 52 0 in progress 192.168.1.1 13 16723211 47 0 in progress gluster> volume remove-brick my-dist-vol server2:/brick2 commit gluster> volume rebalance my-dist-vol fix-layout start gluster> volume rebalance my-dist-vol start gluster> volume rebalance my-dist-vol status Node Rebalanced-files size scanned failures status --------- ----------- ----------- ----------- ----------- ------------ localhost 112 15674 170 0 completed 10.16.156.72 140 3423 321 2 completed
DUSTIN L. BLACK, RHCA44 Migrating Data / Replacing Bricks gluster> volume replace-brick my-dist-vol server3:/brick3 server5:/brick5 start gluster> volume replace-brick my-dist-vol server3:/brick3 server5:/brick5 status Current File = /usr/src/linux-headers-2.6.31-14/block/Makefile Number of files migrated = 10567 Migration complete gluster> volume replace-brick my-dist-vol server3:/brick3 server5:/brick5 commit
DUSTIN L. BLACK, RHCA45 Volume Options gluster> volume set my-dist-vol auth.allow 192.168.1.* gluster> volume set my-dist-vol auth.reject 10.* gluster> volume set my-dist-vol nfs.volume-access read-only gluster> volume set my-dist-vol nfs.disable on gluster> volume set my-dist-vol features.read-only on gluster> volume set my-dist-vol performance.cache-size 67108864 gluster> volume set my-dist-vol auth.allow 192.168.1.* gluster> volume set my-dist-vol auth.reject 10.* NFS Auth Other advanced options
DUSTIN L. BLACK, RHCA46 Volume Top Command gluster> volume set my-dist-vol auth.allow 192.168.1.* gluster> volume set my-dist-vol auth.reject 10.* gluster> volume top my-dist-vol read brick server3:/brick3 list-cnt 3 Brick: server:/export/dir1 ==========Read file stats======== read filename call count 116 /clients/client0/~dmtmp/SEED/LARGE.FIL 64 /clients/client0/~dmtmp/SEED/MEDIUM.FIL 54 /clients/client2/~dmtmp/SEED/LARGE.FIL ● Many top commands are available for analysis of files, directories, and bricks ● Read and write performance test commands available ● Perform active dd tests and measure throughput
DUSTIN L. BLACK, RHCA47 Volume Profiling gluster> volume set my-dist-vol auth.allow 192.168.1.* gluster> volume set my-dist-vol auth.reject 10.* gluster> volume profile my-dist-vol start gluster> volume profile my-dist-vol info Brick: Test:/export/2 Cumulative Stats: Block 1b+ 32b+ 64b+ Size: Read: 0 0 0 Write: 908 28 8 ... %-latency Avg- Min- Max- calls Fop latency Latency Latency ___________________________________________________________ 4.82 1132.28 21.00 800970.00 4575 WRITE 5.70 156.47 9.00 665085.00 39163 READDIRP 11.35 315.02 9.00 1433947.00 38698 LOOKUP 11.88 1729.34 21.00 2569638.00 7382 FXATTROP 47.35 104235.02 2485.00 7789367.00 488 FSYNC ------------------ Duration : 335 BytesRead : 94505058 BytesWritten : 195571980
DUSTIN L. BLACK, RHCA48 Geo-Replication # ssh-keygen -f /var/lib/glusterd/geo-replication/secret.pem # ssh-copy-id -i /var/lib/glusterd/geo-replication/secret.pem repluser@slavehost1 gluster> volume geo-replication my-dist-vol repluser@slavehost1::my-dist-repl start Starting geo-replication session between my-dist-vol & slavehost1:my-dist-repl has been successful gluster> volume geo-replication my-dist-vol status MASTER SLAVE STATUS -------------------------------------------------------------------------------- my-dist-vol ssh://repluser@slavehost1::my-dist-repl OK Setup SSH Keys Replicate Via SSH to Remote GlusterFS Volume gluster> volume info my-dist-vol ... Options Reconfigured: geo-replication.indexing: on Output of volume info Now Reflects Replication
Demystifying Gluster GlusterFS and RHS for the SysAdmin Use Cases
DUSTIN L. BLACK, RHCA50 Common Solutions ● Media / Content Distribution Network (CDN) ● Backup / Archive / Disaster Recovery (DR) ● Large Scale File Server ● Home directories ● High Performance Computing (HPC) ● Infrastructure as a Service (IaaS) storage layer
DUSTIN L. BLACK, RHCA51 Hadoop – Map Reduce ● Access data within and outside of Hadoop ● No HDFS name node single point of failure / bottleneck ● Seamless replacement for HDFS ● Scales with the massive growth of big data
DUSTIN L. BLACK, RHCA52 CIC Electronic Signature Solutions ● Challenge ● Must leverage economics of the cloud ● Storage performance in the cloud too slow ● Need to meet demanding client SLA’s ● Solution ● Red Hat Storage Software Appliance ● Amazon EC2 and Elastic Block Storage (EBS) ● Benefits ● Faster development and delivery of new products ● SLA’s met with headroom to spare ● Accelerated cloud migration ● Scale-out for rapid and simple expansion ● Data is highly available for 24/7 client access Hybrid Cloud: Electronic Signature Solutions ● Reduced time-to- market for new products ● Meeting all client SLAs ● Accelerating move to the cloud
DUSTIN L. BLACK, RHCA53 Pandora Internet Radio ● Challenge ● Explosive user & title growth ● As many as 12 file formats for each song ● ‘Hot’ content and long tail ● Solution ● Three data centers, each with a six-node GlusterFS cluster ● Replication for high availability ● 250+ TB total capacity ● Benefits ● Easily scale capacity ● Centralized management; one administrator to manage day-to-day operations ● No changes to application ● Higher reliability Private Cloud: Media Serving ● 1.2 PB of audio served per week ● 13 million files ● Over 50 GB/sec peak traffic
DUSTIN L. BLACK, RHCA54 Brightcove • Over 1 PB currently in Gluster • Separate 4 PB project in the works Private Cloud: Media Serving ● Challenge ● Explosive customer & title growth ● Massive video in multiple locations ● Costs rising, esp. with HD formats ● Solution ● Complete scale-out based on commodity DAS/JBOD and GlusterFS ● Replication for high availability ● 1PB total capacity ● Benefits ● Easily scale capacity ● Centralized management; one administrator to manage day-to-day operations ● Higher reliability ● Path to multi-site
DUSTIN L. BLACK, RHCA55 Pattern Energy • Rapid and advance weather predictions • Maximizing energy assets • Cost savings and avoidance High Performance Computing for Weather Prediction ● Challenge ● Need to deliver rapid advance weather predictions ● Identify wind and solar abundance in advance ● More effectively perform preventative maintenance and repair ● Solution ● 32 HP compute nodes ● Red Hat SSA for high throughput and availability ● 20TB+ total capacity ● Benefits ● Predicts solar and wind patterns 3 to 5 days in advance ● Maximize energy production and repair times ● Avoid costs of outsourcing weather predictions ● Solution has paid for itself many times over
Demystifying Gluster GlusterFS and RHS for the SysAdmin Common Pitfalls
DUSTIN L. BLACK, RHCA57 Split-Brain Syndrome ● Communication lost between replicated peers ● Clients write separately to multiple copies of a file ● No automatic fix ● May be subjective which copy is right – ALL may be! ● Admin determines the “bad” copy and removes it ● Self-heal will correct the volume ● Trigger a recursive stat to initiate ● Proactive self-healing in RHS 2.0 / GlusterFS 3.3
DUSTIN L. BLACK, RHCA58 Quorum Enforcement ● Disallows writes (EROFS) on non-quorum peers ● Significantly reduces files affected by split-brain ● Preferred when data integrity is the priority ● Not preferred when application integrity is the priority
Demystifying Gluster GlusterFS and RHS for the SysAdmin Do it!
60 Do it! ● Build a test environment in VMs in just minutes! ● Get the bits: ● Fedora 17 has GlusterFS packages natively (3.2) ● RHS appliance eval. ISO available on RHN (3.3) ● Go upstream: www.gluster.org (3.3)
Thank You! ● dustin@redhat.com ● storage-sales@redhat.com ● RHS: www.redhat.com/storage/ ● GlusterFS: www.gluster.org ● TAM: access.redhat.com/support/offerings/tam/ @Glusterorg @RedHatStorage Gluster Red Hat Storage Demystifying Gluster GlusterFS and RHS for the SysAdmin Slides Available at: http://people.redhat.com/dblack/lceu2012

More Related Content

PDF
Git best practices workshop
Otto Kekäläinen
 
PDF
Git basics
GHARSALLAH Mohamed
 
PPTX
Understanding GIT and Version Control
Sourabh Sahu
 
PDF
Gitはじめの一歩
Ayana Yokota
 
PDF
Yahoo! JAPANのプライベートRDBクラウドとマルチライター型 MySQL #dbts2017 #dbtsOSS
Yahoo!デベロッパーネットワーク
 
PDF
Github - Git Training Slides: Foundations
Lee Hanxue
 
PPTX
Git and git workflow best practice
Majid Hosseini
 
PDF
はじめてのGit　forデザイナー＆コーダー
Saeko Yamamoto
 
Git best practices workshop
Otto Kekäläinen
 
Git basics
GHARSALLAH Mohamed
 
Understanding GIT and Version Control
Sourabh Sahu
 
Gitはじめの一歩
Ayana Yokota
 
Yahoo! JAPANのプライベートRDBクラウドとマルチライター型 MySQL #dbts2017 #dbtsOSS
Yahoo!デベロッパーネットワーク
 
Github - Git Training Slides: Foundations
Lee Hanxue
 
Git and git workflow best practice
Majid Hosseini
 
はじめてのGit　forデザイナー＆コーダー
Saeko Yamamoto
 

What's hot (20)

PDF
Git slides
Nanyak S
 
PDF
git and github
Darren Oakley
 
PDF
30分でわかるマイクロサービスアーキテクチャ 第2版
Naoki (Neo) SATO
 
PDF
Git 기본개념과 사용법 그리고 어플리케이션
Dabi Ahn
 
PDF
デザイナのためのGit入門
dsuke Takaoka
 
PPTX
Derin Öğrenme (Deep Learning) Nedir?
eburhan
 
PDF
ノンプログラマでも今日から使える「Git」でバージョン管理
H2O Space. Co., Ltd.
 
PDF
GitOps Toolkit (Cloud Native Nordics Tech Talk)
Weaveworks
 
PDF
[Python] Quick book for dell switch_os10
Jo Hoon
 
PDF
Open dronemapハンズオン
Mizutani Takayuki
 
PPTX
Github
JaneAlamAdnan
 
PDF
実例で学ぶ、明日から使えるSpring Boot Tips #jsug
Toshiaki Maki
 
PPTX
Github
MeetPatel710
 
PDF
GIT | Distributed Version Control System
Mohammad Imam Hossain
 
PPTX
Git and Github Session
GoogleDevelopersStud1
 
PDF
いつやるの？Git入門
Masakazu Matsushita
 
PPTX
GitHub Basics - Derek Bable
"FENG "GEORGE"" YU
 
PPTX
Git Tutorial For Beginners | What is Git and GitHub? | DevOps Tools | DevOps ...
Simplilearn
 
PPTX
Using redmine as a sla ticketing system, helpdesk or service desk software
Aleksandar Pavic
 
PPTX
「チーム開発実践入門」勉強会
Yu Ishikawa
 
Git slides
Nanyak S
 
git and github
Darren Oakley
 
30分でわかるマイクロサービスアーキテクチャ 第2版
Naoki (Neo) SATO
 
Git 기본개념과 사용법 그리고 어플리케이션
Dabi Ahn
 
デザイナのためのGit入門
dsuke Takaoka
 
Derin Öğrenme (Deep Learning) Nedir?
eburhan
 
ノンプログラマでも今日から使える「Git」でバージョン管理
H2O Space. Co., Ltd.
 
GitOps Toolkit (Cloud Native Nordics Tech Talk)
Weaveworks
 
[Python] Quick book for dell switch_os10
Jo Hoon
 
Open dronemapハンズオン
Mizutani Takayuki
 
Github
JaneAlamAdnan
 
実例で学ぶ、明日から使えるSpring Boot Tips #jsug
Toshiaki Maki
 
Github
MeetPatel710
 
GIT | Distributed Version Control System
Mohammad Imam Hossain
 
Git and Github Session
GoogleDevelopersStud1
 
いつやるの？Git入門
Masakazu Matsushita
 
GitHub Basics - Derek Bable
"FENG "GEORGE"" YU
 
Git Tutorial For Beginners | What is Git and GitHub? | DevOps Tools | DevOps ...
Simplilearn
 
Using redmine as a sla ticketing system, helpdesk or service desk software
Aleksandar Pavic
 
「チーム開発実践入門」勉強会
Yu Ishikawa
 
Ad

Viewers also liked (20)

PDF
State of the_gluster_-_lceu
Gluster.org
 
PDF
Qemu gluster fs
Gluster.org
 
ODP
Lcna tutorial-2012
Gluster.org
 
PDF
Gluster wireshark niels_de_vos
Gluster.org
 
ODP
Responsibilities of gluster_maintainers
Gluster.org
 
ODP
Accessing gluster ufo_-_eco_willson
Gluster.org
 
PDF
On demand file-caching_-_gustavo_brand
Gluster.org
 
ODP
Sdc challenges-2012
Gluster.org
 
ODP
GlusterFS Containers
Mohamed Ashiq
 
ODP
Gsummit apis-2013
Gluster.org
 
PDF
Disperse xlator ramon_datalab
Gluster.org
 
ODP
Gluster technical overview
Gluster.org
 
ODP
Software defined storage
Gluster.org
 
PDF
Hands On Gluster with Jeff Darcy
Gluster.org
 
PDF
Gsummit apis-2012
Gluster.org
 
PDF
Gluster Webinar: Introduction to GlusterFS v3.3
GlusterFS
 
ODP
Lisa 2015-gluster fs-hands-on
Gluster.org
 
PDF
Gluster as Block Store in Containers
Gluster.org
 
PDF
Guardian Open Platform Launch Event
Matt McAlister
 
ODP
Lcna example-2012
Gluster.org
 
State of the_gluster_-_lceu
Gluster.org
 
Qemu gluster fs
Gluster.org
 
Lcna tutorial-2012
Gluster.org
 
Gluster wireshark niels_de_vos
Gluster.org
 
Responsibilities of gluster_maintainers
Gluster.org
 
Accessing gluster ufo_-_eco_willson
Gluster.org
 
On demand file-caching_-_gustavo_brand
Gluster.org
 
Sdc challenges-2012
Gluster.org
 
GlusterFS Containers
Mohamed Ashiq
 
Gsummit apis-2013
Gluster.org
 
Disperse xlator ramon_datalab
Gluster.org
 
Gluster technical overview
Gluster.org
 
Software defined storage
Gluster.org
 
Hands On Gluster with Jeff Darcy
Gluster.org
 
Gsummit apis-2012
Gluster.org
 
Gluster Webinar: Introduction to GlusterFS v3.3
GlusterFS
 
Lisa 2015-gluster fs-hands-on
Gluster.org
 
Gluster as Block Store in Containers
Gluster.org
 
Guardian Open Platform Launch Event
Matt McAlister
 
Lcna example-2012
Gluster.org
 
Ad

Similar to Gluster for sysadmins (20)

PDF
Gluster fs tutorial part 2 gluster and big data- gluster for devs and sys ...
Tommy Lee
 
ODP
Glusterfs for sysadmins-justin_clift
Gluster.org
 
ODP
Dustin Black - Red Hat Storage Server Administration Deep Dive
Gluster.org
 
PDF
Introducing gluster filesystem by aditya
Aditya Chhikara
 
ODP
Red Hat Gluster Storage : GlusterFS
bipin kunal
 
PDF
Gluster fs architecture_&_roadmap_atin_punemeetup_2015
Atin Mukherjee
 
PDF
Gluster fs architecture_&amp;_roadmap-vijay_bellur-linuxcon_eu_2013
Gluster.org
 
PDF
GlusterFS : un file system open source per i big data di oggi e domani - Robe...
Codemotion
 
PDF
GlusterFs: a scalable file system for today's and tomorrow's big data
Roberto Franchini
 
PDF
Glusterfs and openstack
openstackindia
 
PDF
GlusterFS Update and OpenStack Integration
Etsuji Nakai
 
PPTX
Software Defined storage
Kirillos Akram
 
PDF
GlusterFS Talk for CentOS Dojo Bangalore
Raghavendra Talur
 
ODP
The Future of GlusterFS and Gluster.org
John Mark Walker
 
ODP
GlusterFs Architecture & Roadmap - LinuxCon EU 2013
Gluster.org
 
ODP
Gluster fs architecture_future_directions_tlv
Sahina Bose
 
ODP
Lisa 2015-gluster fs-introduction
Gluster.org
 
PDF
Red Hat Gluster Storage Performance
Red_Hat_Storage
 
PDF
Gluster intro-tdose
Gluster.org
 
ODP
Gluster intro-tdose
Gluster.org
 
Gluster fs tutorial part 2 gluster and big data- gluster for devs and sys ...
Tommy Lee
 
Glusterfs for sysadmins-justin_clift
Gluster.org
 
Dustin Black - Red Hat Storage Server Administration Deep Dive
Gluster.org
 
Introducing gluster filesystem by aditya
Aditya Chhikara
 
Red Hat Gluster Storage : GlusterFS
bipin kunal
 
Gluster fs architecture_&_roadmap_atin_punemeetup_2015
Atin Mukherjee
 
Gluster fs architecture_&amp;_roadmap-vijay_bellur-linuxcon_eu_2013
Gluster.org
 
GlusterFS : un file system open source per i big data di oggi e domani - Robe...
Codemotion
 
GlusterFs: a scalable file system for today's and tomorrow's big data
Roberto Franchini
 
Glusterfs and openstack
openstackindia
 
GlusterFS Update and OpenStack Integration
Etsuji Nakai
 
Software Defined storage
Kirillos Akram
 
GlusterFS Talk for CentOS Dojo Bangalore
Raghavendra Talur
 
The Future of GlusterFS and Gluster.org
John Mark Walker
 
GlusterFs Architecture & Roadmap - LinuxCon EU 2013
Gluster.org
 
Gluster fs architecture_future_directions_tlv
Sahina Bose
 
Lisa 2015-gluster fs-introduction
Gluster.org
 
Red Hat Gluster Storage Performance
Red_Hat_Storage
 
Gluster intro-tdose
Gluster.org
 
Gluster intro-tdose
Gluster.org
 

More from Gluster.org (20)

PDF
Automating Gluster @ Facebook - Shreyas Siravara
Gluster.org
 
PDF
nfusr: a new userspace NFS client based on libnfs - Shreyas Siravara
Gluster.org
 
PDF
Facebook’s upstream approach to GlusterFS - David Hasson
Gluster.org
 
PDF
Throttling Traffic at Facebook Scale
Gluster.org
 
PDF
GlusterFS w/ Tiered XFS
Gluster.org
 
PDF
Gluster Metrics: why they are crucial for running stable deployments of all s...
Gluster.org
 
PDF
Up and Running with Glusto & Glusto-Tests in 5 Minutes (or less)
Gluster.org
 
PDF
Data Reduction for Gluster with VDO
Gluster.org
 
PDF
Releases: What are contributors responsible for
Gluster.org
 
PDF
RIO Distribution: Reconstructing the onion - Shyamsundar Ranganathan
Gluster.org
 
PDF
Gluster and Kubernetes
Gluster.org
 
PDF
Native Clients, more the merrier with GFProxy!
Gluster.org
 
PDF
Gluster: a SWOT Analysis
Gluster.org
 
PDF
GlusterD-2.0: What's Happening? - Kaushal Madappa
Gluster.org
 
PDF
Scalability and Performance of CNS 3.6
Gluster.org
 
PDF
What Makes Us Fail
Gluster.org
 
PDF
Gluster as Native Storage for Containers - past, present and future
Gluster.org
 
PDF
Heketi Functionality into Glusterd2
Gluster.org
 
PDF
Architecture of the High Availability Solution for Ganesha and Samba with Kal...
Gluster.org
 
PDF
Challenges with Gluster and Persistent Memory with Dan Lambright
Gluster.org
 
Automating Gluster @ Facebook - Shreyas Siravara
Gluster.org
 
nfusr: a new userspace NFS client based on libnfs - Shreyas Siravara
Gluster.org
 
Facebook’s upstream approach to GlusterFS - David Hasson
Gluster.org
 
Throttling Traffic at Facebook Scale
Gluster.org
 
GlusterFS w/ Tiered XFS
Gluster.org
 
Gluster Metrics: why they are crucial for running stable deployments of all s...
Gluster.org
 
Up and Running with Glusto & Glusto-Tests in 5 Minutes (or less)
Gluster.org
 
Data Reduction for Gluster with VDO
Gluster.org
 
Releases: What are contributors responsible for
Gluster.org
 
RIO Distribution: Reconstructing the onion - Shyamsundar Ranganathan
Gluster.org
 
Gluster and Kubernetes
Gluster.org
 
Native Clients, more the merrier with GFProxy!
Gluster.org
 
Gluster: a SWOT Analysis
Gluster.org
 
GlusterD-2.0: What's Happening? - Kaushal Madappa
Gluster.org
 
Scalability and Performance of CNS 3.6
Gluster.org
 
What Makes Us Fail
Gluster.org
 
Gluster as Native Storage for Containers - past, present and future
Gluster.org
 
Heketi Functionality into Glusterd2
Gluster.org
 
Architecture of the High Availability Solution for Ganesha and Samba with Kal...
Gluster.org
 
Challenges with Gluster and Persistent Memory with Dan Lambright
Gluster.org
 

Recently uploaded (20)

PPTX
Simple and concise overview about Quantum computing..pptx
mughal641
 
PDF
Oracle AI Vector Search- Getting Started and what's new in 2025- AIOUG Yatra ...
Sandesh Rao
 
PPTX
AI in Daily Life: How Artificial Intelligence Helps Us Every Day
vanshrpatil7
 
PDF
AI Unleashed - Shaping the Future -Starting Today - AIOUG Yatra 2025 - For Co...
Sandesh Rao
 
PPTX
Introduction to Flutter by Ayush Desai.pptx
ayushdesai204
 
PDF
How Open Source Changed My Career by abdelrahman ismail
a0m0rajab1
 
PDF
The Future of Mobile Is Context-Aware—Are You Ready?
iProgrammer Solutions Private Limited
 
PDF
Accelerating Oracle Database 23ai Troubleshooting with Oracle AHF Fleet Insig...
Sandesh Rao
 
PDF
SparkLabs Primer on Artificial Intelligence 2025
SparkLabs Group
 
PDF
CIFDAQ's Market Wrap : Bears Back in Control?
CIFDAQ
 
PPTX
Agile Chennai 18-19 July 2025 Ideathon | AI Powered Microfinance Literacy Gui...
AgileNetwork
 
PDF
Security features in Dell, HP, and Lenovo PC systems: A research-based compar...
Principled Technologies
 
PDF
Data_Analytics_vs_Data_Science_vs_BI_by_CA_Suvidha_Chaplot.pdf
CA Suvidha Chaplot
 
PDF
Structs to JSON: How Go Powers REST APIs
Emily Achieng
 
PDF
The Future of Artificial Intelligence (AI)
Mukul
 
PDF
Peak of Data & AI Encore - Real-Time Insights & Scalable Editing with ArcGIS
Safe Software
 
PPTX
OA presentation.pptx OA presentation.pptx
pateldhruv002338
 
PDF
Economic Impact of Data Centres to the Malaysian Economy
flintglobalapac
 
PPTX
Dev Dives: Automate, test, and deploy in one place—with Unified Developer Exp...
AndreeaTom
 
PDF
Using Anchore and DefectDojo to Stand Up Your DevSecOps Function
Anchore
 
Simple and concise overview about Quantum computing..pptx
mughal641
 
Oracle AI Vector Search- Getting Started and what's new in 2025- AIOUG Yatra ...
Sandesh Rao
 
AI in Daily Life: How Artificial Intelligence Helps Us Every Day
vanshrpatil7
 
AI Unleashed - Shaping the Future -Starting Today - AIOUG Yatra 2025 - For Co...
Sandesh Rao
 
Introduction to Flutter by Ayush Desai.pptx
ayushdesai204
 
How Open Source Changed My Career by abdelrahman ismail
a0m0rajab1
 
The Future of Mobile Is Context-Aware—Are You Ready?
iProgrammer Solutions Private Limited
 
Accelerating Oracle Database 23ai Troubleshooting with Oracle AHF Fleet Insig...
Sandesh Rao
 
SparkLabs Primer on Artificial Intelligence 2025
SparkLabs Group
 
CIFDAQ's Market Wrap : Bears Back in Control?
CIFDAQ
 
Agile Chennai 18-19 July 2025 Ideathon | AI Powered Microfinance Literacy Gui...
AgileNetwork
 
Security features in Dell, HP, and Lenovo PC systems: A research-based compar...
Principled Technologies
 
Data_Analytics_vs_Data_Science_vs_BI_by_CA_Suvidha_Chaplot.pdf
CA Suvidha Chaplot
 
Structs to JSON: How Go Powers REST APIs
Emily Achieng
 
The Future of Artificial Intelligence (AI)
Mukul
 
Peak of Data & AI Encore - Real-Time Insights & Scalable Editing with ArcGIS
Safe Software
 
OA presentation.pptx OA presentation.pptx
pateldhruv002338
 
Economic Impact of Data Centres to the Malaysian Economy
flintglobalapac
 
Dev Dives: Automate, test, and deploy in one place—with Unified Developer Exp...
AndreeaTom
 
Using Anchore and DefectDojo to Stand Up Your DevSecOps Function
Anchore
 

Gluster for sysadmins

  • 1. Demystifying Gluster GlusterFS and RHS for the SysAdmin Dustin L. Black, RHCA Sr. Technical Account Manager, Red Hat 2012-11-08
  • 2. DUSTIN L. BLACK, RHCA2 #whoami ● Systems and Infrastructure Geek ● Decade+ of Linux, UNIX, networking ● <notacoder/> ● Believe in Open Source Everything ● Sr. Technical Account Manager, Red Hat GSS ● [email protected]
  • 3. DUSTIN L. BLACK, RHCA3 #whatis TAM ● Premium named-resource support ● Proactive and early access ● Regular calls and on-site engagements ● Customer advocate within Red Hat and upstream ● Multi-vendor support coordinator ● High-touch access to engineering ● Influence for software enhancements ● NOT Hands-on or consulting
  • 4. DUSTIN L. BLACK, RHCA4 Agenda ● Technology Overview ● Scaling Up and Out ● A Peek at GlusterFS Logic ● Redundancy and Fault Tolerance ● Data Access ● General Administration ● Use Cases ● Common Pitfalls
  • 5. Demystifying Gluster GlusterFS and RHS for the SysAdmin Technology Overview
  • 6. DUSTIN L. BLACK, RHCA6 What is GlusterFS? ● POSIX-Like Distributed File System ● No Metadata Server ● Network Attached Storage (NAS) ● Heterogeneous Commodity Hardware ● Aggregated Storage and Memory ● Standards-Based – Clients, Applications, Networks ● Flexible and Agile Scaling ● Capacity – Petabytes and beyond ● Performance – Thousands of Clients ● Single Global Namespace
  • 7. DUSTIN L. BLACK, RHCA7 What is Red Hat Storage? ● Enterprise Implementation of GlusterFS ● Software Appliance ● Bare Metal Installation ● Built on RHEL + XFS ● Subscription Model ● Storage Software Appliance ● Datacenter and Private Cloud Deployments ● Virtual Storage Appliance ● Amazon Web Services Public Cloud Deployments
  • 8. DUSTIN L. BLACK, RHCA8 RHS vs. Traditional Solutions ● A basic NAS has limited scalability and redundancy ● Other distributed filesystems limited by metadata ● SAN is costly & complicated but high performance & scalable ● RHS ● Linear Scaling ● Minimal Overhead ● High Redundancy ● Simple and Inexpensive Deployment
  • 9. Demystifying Gluster GlusterFS and RHS for the SysAdmin Technology Stack
  • 10. DUSTIN L. BLACK, RHCA10 Terminology ● Brick ● A filesystem mountpoint ● A unit of storage used as a GlusterFS building block ● Translator ● Logic between the bits and the Global Namespace ● Layered to provide GlusterFS functionality ● Volume ● Bricks combined and passed through translators ● Node / Peer ● Server running the gluster daemon and sharing volumes
  • 11. DUSTIN L. BLACK, RHCA11 Foundation Components ● Private Cloud (Datacenter) ● Common Commodity x86_64 Servers ● RHS: Hardware Compatibility List (HCL) ● Public Cloud ● Amazon Web Services (AWS) ● EC2 + EBS
  • 12. DUSTIN L. BLACK, RHCA12 Disk, LVM, and Filesystems ● Direct-Attached Storage (DAS) -or- ● Just a Bunch Of Disks (JBOD) ● Hardware RAID ● RHS: RAID 6 required ● Logical Volume Management (LVM) ● XFS, EXT3/4, BTRFS ● Extended attributes support required ● RHS: XFS required
  • 13. DUSTIN L. BLACK, RHCA13 Gluster Components ● glusterd ● Elastic volume management daemon ● Runs on all export servers ● Interfaced through gluster CLI ● glusterfsd ● GlusterFS brick daemon ● One process for each brick ● Managed by glusterd
  • 14. DUSTIN L. BLACK, RHCA14 Gluster Components ● glusterfs ● NFS server daemon ● FUSE client daemon ● mount.glusterfs ● FUSE native mount tool ● gluster ● Gluster Console Manager (CLI)
  • 15. DUSTIN L. BLACK, RHCA15 Data Access Overview ● GlusterFS Native Client ● Filesystem in Userspace (FUSE) ● NFS ● Built-in Service ● SMB/CIFS ● Samba server required ● Unified File and Object (UFO) ● Simultaneous object-based access
  • 16. DUSTIN L. BLACK, RHCA16 Putting it All Together
  • 17. Demystifying Gluster GlusterFS and RHS for the SysAdmin Scaling
  • 18. DUSTIN L. BLACK, RHCA18 Scaling Up ● Add disks and filesystems to a node ● Expand a GlusterFS volume by adding bricks XFS
  • 19. DUSTIN L. BLACK, RHCA19 Scaling Out ● Add GlusterFS nodes to trusted pool ● Add filesystems as new bricks
  • 20. Demystifying Gluster GlusterFS and RHS for the SysAdmin Under the Hood
  • 21. DUSTIN L. BLACK, RHCA21 Elastic Hash Algorithm ● No central metadata ● No Performance Bottleneck ● Eliminates risk scenarios ● Location hashed intelligently on path and filename ● Unique identifiers, similar to md5sum ● The “Elastic” Part ● Files assigned to virtual volumes ● Virtual volumes assigned to multiple bricks ● Volumes easily reassigned on the fly
  • 22. DUSTIN L. BLACK, RHCA22 Translators
  • 23. Demystifying Gluster GlusterFS and RHS for the SysAdmin Distribution and Replication
  • 24. DUSTIN L. BLACK, RHCA24 Distributed Volume ● Files “evenly” spread across bricks ● File-level RAID 0 ● Server/Disk failure could be catastrophic
  • 25. DUSTIN L. BLACK, RHCA25 Replicated Volume ● Copies files to multiple bricks ● File-level RAID 1
  • 26. DUSTIN L. BLACK, RHCA26 Distributed Replicated Volume ● Distributes files across replicated bricks ● RAID 1 plus improved read performance
  • 27. DUSTIN L. BLACK, RHCA27 Geo Replication ● Asynchronous across LAN, WAN, or Internet ● Master-Slave model -- Cascading possible ● Continuous and incremental ● Data is passed between defined master and slave only
  • 28. DUSTIN L. BLACK, RHCA28 Replicated Volumes vs Geo-replication Replicated Volumes Geo-replication Mirrors data across clusters Mirrors data across geographically distributed clusters Provides high-availability Ensures backing up of data for disaster recovery Synchronous replication (each and every file operation is sent across all the bricks) Asynchronous replication (checks for the changes in files periodically and syncs them on detecting differences)
  • 29. Demystifying Gluster GlusterFS and RHS for the SysAdmin Layered Functionality
  • 30. DUSTIN L. BLACK, RHCA30 Striped Volumes ● Individual files split among bricks ● Similar to RAID 0 ● Limited Use Cases – HPC Pre/Post Processing
  • 31. DUSTIN L. BLACK, RHCA31 Distributed Striped Volume ● Files striped across two or more nodes ● Striping plus scalability
  • 32. DUSTIN L. BLACK, RHCA32 Striped Replicated Volume ● RHS 2.0 / GlusterFS 3.3+ ● Similar to RAID 10 (1+0)
  • 33. DUSTIN L. BLACK, RHCA33 Distributed Striped Replicated Volume ● RHS 2.0 / GlusterFS 3.3+ ● Limited Use Cases – Map Reduce
  • 34. Demystifying Gluster GlusterFS and RHS for the SysAdmin Data Access
  • 35. DUSTIN L. BLACK, RHCA35 GlusterFS Native Client (FUSE) ● FUSE kernel module allows the filesystem to be built and operated entirely in userspace ● Specify mount to any GlusterFS node ● Native Client fetches volfile from mount server, then communicates directly with all nodes to access data ● Recommended for high concurrency and high write performance ● Load is inherently balanced across distributed volumes
  • 36. DUSTIN L. BLACK, RHCA36 NFS ● Standard NFS v3 clients ● Note: Mount with vers=3 option ● Standard automounter is supported ● Mount to any node, or use a load balancer ● GlusterFS NFS server includes Network Lock Manager (NLM) to synchronize locks across clients ● Better performance for reading many small files from a single client ● Load balancing must be managed externally
  • 37. DUSTIN L. BLACK, RHCA37 SMB/CIFS ● GlusterFS volume is first mounted with the Native Client ● Redundantly on the GlusterFS peer -or- ● On an external server ● Native mount point is then shared via Samba ● Must be setup on each node you wish to connect to via CIFS ● Load balancing must be managed externally
  • 38. Demystifying Gluster GlusterFS and RHS for the SysAdmin General Administration
  • 39. DUSTIN L. BLACK, RHCA39 Preparing a Brick # lvcreate -L 100G -n lv_brick1 vg_server1 # mkfs -t xfs -i size=512 /dev/vg_server1/lv_brick1 # mkdir /brick1 # mount /dev/vg_server1/lv_brick1 /brick1 # echo '/dev/vg_server1/lv_brick1 /brick1 xfs defaults 1 2' >> /etc/fstab
  • 40. DUSTIN L. BLACK, RHCA40 Adding Nodes (peers) and Volumes gluster> peer probe server3 gluster> peer status Number of Peers: 2 Hostname: server2 Uuid: 5e987bda-16dd-43c2-835b-08b7d55e94e5 State: Peer in Cluster (Connected) Hostname: server3 Uuid: 1e0ca3aa-9ef7-4f66-8f15-cbc348f29ff7 State: Peer in Cluster (Connected) gluster> volume create my-dist-vol server2:/brick2 server3:/brick3 gluster> volume info my-dist-vol Volume Name: my-dist-vol Type: Distribute Status: Created Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: server2:/brick2 Brick2: server3:/brick3 gluster> volume start my-dist-vol Distributed Volume Peer Probe
  • 41. DUSTIN L. BLACK, RHCA41 Distributed Striped Replicated Volume gluster> volume create test-volume replica 2 stripe 2 transport tcp server1:/exp1 server1:/exp2 server2:/exp3 server2:/exp4 server3:/exp5 server3:/exp6 server4:/exp7 server4:/exp8 Multiple bricks of a replicate volume are present on the same server. This setup is not optimal. Do you still want to continue creating the volume? (y/n) y Creation of volume test-volume has been successful. Please start the volume to access data. <- test-volume<- test-volume <- distributed files -><- distributed files -> stripe 2stripe 2 replica 2replica 2
  • 42. DUSTIN L. BLACK, RHCA42 Distributed Striped Replicated Volume gluster> volume info test-volume Volume Name: test-volume Type: Distributed-Striped-Replicate Volume ID: 8f8b8b59-d1a1-42fe-ae05-abe2537d0e2d Status: Created Number of Bricks: 2 x 2 x 2 = 8 Transport-type: tcp Bricks: Brick1: server1:/exp1 Brick2: server2:/exp3 Brick3: server1:/exp2 Brick4: server2:/exp4 Brick5: server3:/exp5 Brick6: server4:/exp7 Brick7: server3:/exp6 Brick8: server4:/exp8 gluster> volume create test-volume stripe 2 replica 2 transport tcp server1:/exp1 server2:/exp3 server1:/exp2 server2:/exp4 server3:/exp5 server4:/exp7 server3:/exp6 server4:/exp8 Creation of volume test-volume has been successful. Please start the volume to access data.
  • 43. DUSTIN L. BLACK, RHCA43 Manipulating Bricks in a Volume gluster> volume add-brick my-dist-vol server4:/brick4 gluster> volume remove-brick my-dist-vol server2:/brick2 start gluster> volume remove-brick my-dist-vol server2:/brick2 status Node Rebalanced-files size scanned failures status --------- ----------- ----------- ----------- ----------- ------------ localhost 16 16777216 52 0 in progress 192.168.1.1 13 16723211 47 0 in progress gluster> volume remove-brick my-dist-vol server2:/brick2 commit gluster> volume rebalance my-dist-vol fix-layout start gluster> volume rebalance my-dist-vol start gluster> volume rebalance my-dist-vol status Node Rebalanced-files size scanned failures status --------- ----------- ----------- ----------- ----------- ------------ localhost 112 15674 170 0 completed 10.16.156.72 140 3423 321 2 completed
  • 44. DUSTIN L. BLACK, RHCA44 Migrating Data / Replacing Bricks gluster> volume replace-brick my-dist-vol server3:/brick3 server5:/brick5 start gluster> volume replace-brick my-dist-vol server3:/brick3 server5:/brick5 status Current File = /usr/src/linux-headers-2.6.31-14/block/Makefile Number of files migrated = 10567 Migration complete gluster> volume replace-brick my-dist-vol server3:/brick3 server5:/brick5 commit
  • 45. DUSTIN L. BLACK, RHCA45 Volume Options gluster> volume set my-dist-vol auth.allow 192.168.1.* gluster> volume set my-dist-vol auth.reject 10.* gluster> volume set my-dist-vol nfs.volume-access read-only gluster> volume set my-dist-vol nfs.disable on gluster> volume set my-dist-vol features.read-only on gluster> volume set my-dist-vol performance.cache-size 67108864 gluster> volume set my-dist-vol auth.allow 192.168.1.* gluster> volume set my-dist-vol auth.reject 10.* NFS Auth Other advanced options
  • 46. DUSTIN L. BLACK, RHCA46 Volume Top Command gluster> volume set my-dist-vol auth.allow 192.168.1.* gluster> volume set my-dist-vol auth.reject 10.* gluster> volume top my-dist-vol read brick server3:/brick3 list-cnt 3 Brick: server:/export/dir1 ==========Read file stats======== read filename call count 116 /clients/client0/~dmtmp/SEED/LARGE.FIL 64 /clients/client0/~dmtmp/SEED/MEDIUM.FIL 54 /clients/client2/~dmtmp/SEED/LARGE.FIL ● Many top commands are available for analysis of files, directories, and bricks ● Read and write performance test commands available ● Perform active dd tests and measure throughput
  • 47. DUSTIN L. BLACK, RHCA47 Volume Profiling gluster> volume set my-dist-vol auth.allow 192.168.1.* gluster> volume set my-dist-vol auth.reject 10.* gluster> volume profile my-dist-vol start gluster> volume profile my-dist-vol info Brick: Test:/export/2 Cumulative Stats: Block 1b+ 32b+ 64b+ Size: Read: 0 0 0 Write: 908 28 8 ... %-latency Avg- Min- Max- calls Fop latency Latency Latency ___________________________________________________________ 4.82 1132.28 21.00 800970.00 4575 WRITE 5.70 156.47 9.00 665085.00 39163 READDIRP 11.35 315.02 9.00 1433947.00 38698 LOOKUP 11.88 1729.34 21.00 2569638.00 7382 FXATTROP 47.35 104235.02 2485.00 7789367.00 488 FSYNC ------------------ Duration : 335 BytesRead : 94505058 BytesWritten : 195571980
  • 48. DUSTIN L. BLACK, RHCA48 Geo-Replication # ssh-keygen -f /var/lib/glusterd/geo-replication/secret.pem # ssh-copy-id -i /var/lib/glusterd/geo-replication/secret.pem repluser@slavehost1 gluster> volume geo-replication my-dist-vol repluser@slavehost1::my-dist-repl start Starting geo-replication session between my-dist-vol & slavehost1:my-dist-repl has been successful gluster> volume geo-replication my-dist-vol status MASTER SLAVE STATUS -------------------------------------------------------------------------------- my-dist-vol ssh://repluser@slavehost1::my-dist-repl OK Setup SSH Keys Replicate Via SSH to Remote GlusterFS Volume gluster> volume info my-dist-vol ... Options Reconfigured: geo-replication.indexing: on Output of volume info Now Reflects Replication
  • 49. Demystifying Gluster GlusterFS and RHS for the SysAdmin Use Cases
  • 50. DUSTIN L. BLACK, RHCA50 Common Solutions ● Media / Content Distribution Network (CDN) ● Backup / Archive / Disaster Recovery (DR) ● Large Scale File Server ● Home directories ● High Performance Computing (HPC) ● Infrastructure as a Service (IaaS) storage layer
  • 51. DUSTIN L. BLACK, RHCA51 Hadoop – Map Reduce ● Access data within and outside of Hadoop ● No HDFS name node single point of failure / bottleneck ● Seamless replacement for HDFS ● Scales with the massive growth of big data
  • 52. DUSTIN L. BLACK, RHCA52 CIC Electronic Signature Solutions ● Challenge ● Must leverage economics of the cloud ● Storage performance in the cloud too slow ● Need to meet demanding client SLA’s ● Solution ● Red Hat Storage Software Appliance ● Amazon EC2 and Elastic Block Storage (EBS) ● Benefits ● Faster development and delivery of new products ● SLA’s met with headroom to spare ● Accelerated cloud migration ● Scale-out for rapid and simple expansion ● Data is highly available for 24/7 client access Hybrid Cloud: Electronic Signature Solutions ● Reduced time-to- market for new products ● Meeting all client SLAs ● Accelerating move to the cloud
  • 53. DUSTIN L. BLACK, RHCA53 Pandora Internet Radio ● Challenge ● Explosive user & title growth ● As many as 12 file formats for each song ● ‘Hot’ content and long tail ● Solution ● Three data centers, each with a six-node GlusterFS cluster ● Replication for high availability ● 250+ TB total capacity ● Benefits ● Easily scale capacity ● Centralized management; one administrator to manage day-to-day operations ● No changes to application ● Higher reliability Private Cloud: Media Serving ● 1.2 PB of audio served per week ● 13 million files ● Over 50 GB/sec peak traffic
  • 54. DUSTIN L. BLACK, RHCA54 Brightcove • Over 1 PB currently in Gluster • Separate 4 PB project in the works Private Cloud: Media Serving ● Challenge ● Explosive customer & title growth ● Massive video in multiple locations ● Costs rising, esp. with HD formats ● Solution ● Complete scale-out based on commodity DAS/JBOD and GlusterFS ● Replication for high availability ● 1PB total capacity ● Benefits ● Easily scale capacity ● Centralized management; one administrator to manage day-to-day operations ● Higher reliability ● Path to multi-site
  • 55. DUSTIN L. BLACK, RHCA55 Pattern Energy • Rapid and advance weather predictions • Maximizing energy assets • Cost savings and avoidance High Performance Computing for Weather Prediction ● Challenge ● Need to deliver rapid advance weather predictions ● Identify wind and solar abundance in advance ● More effectively perform preventative maintenance and repair ● Solution ● 32 HP compute nodes ● Red Hat SSA for high throughput and availability ● 20TB+ total capacity ● Benefits ● Predicts solar and wind patterns 3 to 5 days in advance ● Maximize energy production and repair times ● Avoid costs of outsourcing weather predictions ● Solution has paid for itself many times over
  • 56. Demystifying Gluster GlusterFS and RHS for the SysAdmin Common Pitfalls
  • 57. DUSTIN L. BLACK, RHCA57 Split-Brain Syndrome ● Communication lost between replicated peers ● Clients write separately to multiple copies of a file ● No automatic fix ● May be subjective which copy is right – ALL may be! ● Admin determines the “bad” copy and removes it ● Self-heal will correct the volume ● Trigger a recursive stat to initiate ● Proactive self-healing in RHS 2.0 / GlusterFS 3.3
  • 58. DUSTIN L. BLACK, RHCA58 Quorum Enforcement ● Disallows writes (EROFS) on non-quorum peers ● Significantly reduces files affected by split-brain ● Preferred when data integrity is the priority ● Not preferred when application integrity is the priority
  • 59. Demystifying Gluster GlusterFS and RHS for the SysAdmin Do it!
  • 60. 60 Do it! ● Build a test environment in VMs in just minutes! ● Get the bits: ● Fedora 17 has GlusterFS packages natively (3.2) ● RHS appliance eval. ISO available on RHN (3.3) ● Go upstream: www.gluster.org (3.3)
  • 61. Thank You! ● [email protected] ● [email protected] ● RHS: www.redhat.com/storage/ ● GlusterFS: www.gluster.org ● TAM: access.redhat.com/support/offerings/tam/ @Glusterorg @RedHatStorage Gluster Red Hat Storage Demystifying Gluster GlusterFS and RHS for the SysAdmin Slides Available at: http://people.redhat.com/dblack/lceu2012