k8s分布式存储-ceph

已于 2024-08-08 15:12:23 修改
阅读量2k 收藏 35
点赞数 35
CC 4.0 BY-SA版权
文章标签： kubernetes 分布式 ceph
于 2024-08-08 15:04:46 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_43876317/article/details/141026818

文章目录

Ceph

deploy-ceph部署

投入使用ceph前，要知道一个很现实得问题，ceph对低版本内核得客户端使用非常不友好，低内核是指小于等于3.10.0-862，默认的centos7.5及以下的系统都是小于此类内核，无法正常使用ceph的文件存储（cephFS）块存储（RBD）。

ceph部署一定要考虑好版本问题，经测试如果想使用ceph16版本，那你的客户端操作系统内核小于3.10.0-862根本用不了，常见的centos7.5以下默认没升级的内核都是小于3.10.0-862，所以这一大批服务器使用ceph提供的rdb存储都会有问题，而且ceph已经不提供centos的16版本的ceph-common组件，也就是说ceph集群部署16版本，常见的客户端centos7系统只能使用15版本的ceph-common，虽说也可以使用，但也存在一定隐患毕竟不是同一版本客户端软件，目前推荐使用ceph15的最高版本，15版本的安装与16相同，只是ceph源不同。

以上说法不正确，ceph版本选择和客户端内核没有关系，是所有版本的ceph都不友好支持内核小于等于3.10.0-862（CentOS7.5）

环境

ubuntu 18.04b版本

ceph 16.10版本

主机名 IP 部署 内容
ceph-master01 public IP：172.26.156.217 内部通讯IP： 10.0.0.217 mon,mgr,osd,ceph-deploy
ceph-master02 public IP：172.26.156.218 内部通讯IP：10.0.0.218 mon,mgr,osd
ceph-master03 public IP：172.26.156.219 内部通讯IP：10.0.0.219 mon,mgr,osd

服务器配置：服务器型号，cpu，内存，磁盘大小及磁盘类型必须完全一致，磁盘最好全ssd盘

1.系统环境初始化

1.1 修改主机名，DNS解析
master01：
hostnamectl  set-hostname  ceph-master01
vi /etc/hostname
ceph-master01

master02：
hostnamectl  set-hostname  ceph-master02
vi /etc/hostname
ceph-master02

master03：
hostnamectl  set-hostname  ceph-master03
vi /etc/hostname
ceph-master03

vi /etc/hosts
10.0.0.217 ceph-master01.example.local ceph-master01
10.0.0.218 ceph-master02.example.local ceph-master02
10.0.0.219 ceph-master03.example.local ceph-master03
1.2 时间同步

所有服务器执行

#修改时区
timedatectl set-timezone Asia/Shanghai

#时间同步
root@ubuntu:~# apt install ntpdate
root@ubuntu:~# ntpdate  ntp.aliyun.com
 1 Sep 20:54:39 ntpdate[9120]: adjust time server 203.107.6.88 offset 0.003441 sec
root@ubuntu:~# crontab  -e 
crontab: installing new crontab
root@ubuntu:~# crontab  -l 
* * * * * ntpdate  ntp.aliyun.com
1.3 配置apt基础源与ceph源

所有服务器执行如下命令自动替换

#基础源
sed -i "s@http://.*archive.ubuntu.com@http://mirrors.tuna.tsinghua.edu.cn@g" /etc/apt/sources.list
sed -i "s@http://.*security.ubuntu.com@http://mirrors.tuna.tsinghua.edu.cn@g" /etc/apt/sources.list
#ceph源
echo "deb https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific bionic main" >> /etc/apt/sources.list.d/ceph.list
echo "deb https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-nautilus  bionic main" >> /etc/apt/sources.list.d/ceph.list

#导入ceph源key，不然不能使用ceph源
wget -q -O- 'https://mirrors.tuna.tsinghua.edu.cn/ceph/keys/release.asc' | sudo apt-key add -

# ceph仓库为https的话需要安装下面，不然无法使用https源
apt install -y apt-transport-https ca-certificates curl software-properties-common

apt update
1.4关闭selinux与防火墙
# ufw disable
1.5 创建 ceph 集群部署用户 cephadmin

推荐使用指定的普通用户部署和运行 ceph 集群，普通用户只要能以非交互方式执行 sudo 命令执行一些特权命令即可，新版的 ceph-deploy 可以指定包含 root 的在内只要可以执 行 sudo 命令的用户，不过仍然推荐使用普通用户，ceph 集群安装完成后会自动创建 ceph 用户(ceph 集群默认会使用 ceph 用户运行各服务进程,如 ceph-osd 等)，因此推荐 使用除了 ceph 用户之外的比如 cephuser**、**cephadmin 这样的普通用户去部署和 管理ceph 集群。

在包含 ceph-deploy 节点的存储节点、mon 节点和 mgr 节点等创建 cephadmin 用户.

groupadd -r -g 2088 cephadmin && useradd -r -m -s /bin/bash -u 2088 -g 2088 cephadmin && echo cephadmin:chinadci888. | chpasswd

各服务器允许 cephadmin 用户以 sudo 执行特权命令:

~# echo "cephadmin ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers
1.6分发密钥

deploy节点要与所有服务器mon，mgr，osd节点免密，本文这里只有三台服务器，mon,mgr,osd都混合一起部署，所以只免密了三台服务器

master01(deploy节点):

su - cephadmin
ssh-keygen
ssh-copy-id   cephadmin@ceph-master01
ssh-copy-id   cephadmin@ceph-master02
ssh-copy-id   cephadmin@ceph-master03

2. ceph部署

2.1 安装 ceph 部署工具
cephadmin@ceph-master01:~$ apt-cache madison ceph-deploy
ceph-deploy |      2.0.1 | https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-nautilus bionic/main amd64 Packages
ceph-deploy |      2.0.1 | https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-nautilus bionic/main i386 Packages
ceph-deploy | 1.5.38-0ubuntu1 | http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic/universe amd64 Packages
ceph-deploy | 1.5.38-0ubuntu1 | http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic/universe i386 Packages
cephadmin@ceph-master01:~$ sudo apt install ceph-deploy
2.2 初始化 mon 节点

Ubuntu 各服务器需要单独安装 Python2(mon，mgr，osd节点所有服务器必须做)：

cephadmin@ceph-master01:~$ sudo apt install python2.7 -y
cephadmin@ceph-master01:~$ sudo ln -sv /usr/bin/python2.7 /usr/bin/python2

ceph-master01:

ceph-deploy new --cluster-network 10.0.0.0/24 --public-network 172.26.0.0/16 ceph-master01 ceph-master02 ceph-master03

–cluster-network: 集群内部之间通讯网络

–public-network：业务客户端使用网络，单独使用网络，规避

~$ mkdir /etc/ceph-cluster
~$ sudo chown  cephadmin:cephadmin /etc/ceph-cluster
~$ cd /etc/ceph-cluster/
cephadmin@ceph-master01:/etc/ceph-cluster$ ceph-deploy new --cluster-network 10.0.0.0/24 --public-network 172.26.0.0/16 ceph-master01  ceph-master02 ceph-master03
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadmin/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy new --cluster-network 10.0.0.0/24 --public-network 172.26.0.0/16 ceph-master01 ceph-master02 ceph-master03
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7efd0a772e10>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  ssh_copykey                   : True
[ceph_deploy.cli][INFO  ]  mon                           : ['ceph-master01', 'ceph-master02', 'ceph-master03']
[ceph_deploy.cli][INFO  ]  func                          : <function new at 0x7efd07a2bbd0>
[ceph_deploy.cli][INFO  ]  public_network                : 172.26.0.0/16
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  cluster_network               : 10.0.0.0/24
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  fsid                          : None
[ceph_deploy.new][DEBUG ] Creating new cluster named ceph
[ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds
[ceph-master01][DEBUG ] connection detected need for sudo
[ceph-master01][DEBUG ] connected to host: ceph-master01 
[ceph-master01][DEBUG ] detect platform information from remote host
[ceph-master01][DEBUG ] detect machine type
[ceph-master01][DEBUG ] find the location of an executable
[ceph-master01][INFO  ] Running command: sudo /bin/ip link show
[ceph-master01][INFO  ] Running command: sudo /bin/ip addr show
[ceph-master01][DEBUG ] IP addresses found: [u'172.26.156.217', u'10.0.0.217']
[ceph_deploy.new][DEBUG ] Resolving host ceph-master01
[ceph_deploy.new][DEBUG ] Monitor ceph-master01 at 172.26.156.217
[ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds
[ceph-master02][DEBUG ] connected to host: ceph-master01 
[ceph-master02][INFO  ] Running command: ssh -CT -o BatchMode=yes ceph-master02
[ceph-master02][DEBUG ] connection detected need for sudo
[ceph-master02][DEBUG ] connected to host: ceph-master02 
[ceph-master02][DEBUG ] detect platform information from remote host
[ceph-master02][DEBUG ] detect machine type
[ceph-master02][DEBUG ] find the location of an executable
[ceph-master02][INFO  ] Running command: sudo /bin/ip link show
[ceph-master02][INFO  ] Running command: sudo /bin/ip addr show
[ceph-master02][DEBUG ] IP addresses found: [u'10.0.0.218', u'172.26.156.218']
[ceph_deploy.new][DEBUG ] Resolving host ceph-master02
[ceph_deploy.new][DEBUG ] Monitor ceph-master02 at 172.26.156.218
[ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds
[ceph-master03][DEBUG ] connected to host: ceph-master01 
[ceph-master03][INFO  ] Running command: ssh -CT -o BatchMode=yes ceph-master03
[ceph-master03][DEBUG ] connection detected need for sudo
[ceph-master03][DEBUG ] connected to host: ceph-master03 
[ceph-master03][DEBUG ] detect platform information from remote host
[ceph-master03][DEBUG ] detect machine type
[ceph-master03][DEBUG ] find the location of an executable
[ceph-master03][INFO  ] Running command: sudo /bin/ip link show
[ceph-master03][INFO  ] Running command: sudo /bin/ip addr show
[ceph-master03][DEBUG ] IP addresses found: [u'172.26.156.219', u'10.0.0.219']
[ceph_deploy.new][DEBUG ] Resolving host ceph-master03
[ceph_deploy.new][DEBUG ] Monitor ceph-master03 at 172.26.156.219
[ceph_deploy.new][DEBUG ] Monitor initial members are ['ceph-master01', 'ceph-master02', 'ceph-master03']
[ceph_deploy.new][DEBUG ] Monitor addrs are [u'172.26.156.217', u'172.26.156.218', u'172.26.156.219']
[ceph_deploy.new][DEBUG ] Creating a random mon key...
[ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring...
[ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf...

cephadmin@ceph-master01:/etc/ceph-cluster$ ll
total 36
drwxr-xr-x  2 cephadmin cephadmin  4096 Sep  2 16:50 ./
drwxr-xr-x 91 root      root       4096 Sep  2 16:22 ../
-rw-rw-r--  1 cephadmin cephadmin   326 Sep  2 16:50 ceph.conf
-rw-rw-r--  1 cephadmin cephadmin 17603 Sep  2 16:50 ceph-deploy-ceph.log
-rw-------  1 cephadmin cephadmin    73 Sep  2 16:50 ceph.mon.keyring

此步骤必须执行，否 ceph 集群的后续安装步骤会报错。

cephadmin@ceph-master01:/etc/ceph-cluster$ ceph-deploy install --no-adjust-repos --nogpgcheck ceph-master01 ceph-master02 ceph-master03

--no-adjust-repos #不修改已有的 apt 仓库源(默认会使用官方仓库) 
--nogpgcheck #不进行校验 

cephadmin@ceph-master01:/etc/ceph-cluster$ ceph-deploy install --no-adjust-repos --nogpgcheck ceph-master01 ceph-master02 ceph-master03
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadmin/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy install --no-adjust-repos --nogpgcheck ceph-master01 ceph-master02 ceph-master03
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  testing                       : None
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f59e4913e60>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  dev_commit                    : None
[ceph_deploy.cli][INFO  ]  install_mds                   : False
[ceph_deploy.cli][INFO  ]  stable                        : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  adjust_repos                  : False
[ceph_deploy.cli][INFO  ]  func                          : <function install at 0x7f59e51c5b50>
[ceph_deploy.cli][INFO  ]  install_mgr                   : False
[ceph_deploy.cli][INFO  ]  install_all                   : False
[ceph_deploy.cli][INFO  ]  repo                          : False
[ceph_deploy.cli][INFO  ]  host                          : ['ceph-master01', 'ceph-master02', 'ceph-master03']
[ceph_deploy.cli][INFO  ]  install_rgw                   : False
[ceph_deploy.cli][INFO  ]  install_tests                 : False
[ceph_deploy.cli][INFO  ]  repo_url                      : None
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  install_osd                   : False
[ceph_deploy.cli][INFO  ]  version_kind                  : stable
[ceph_deploy.cli][INFO  ]  install_common                : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  dev                           : master
[ceph_deploy.cli][INFO  ]  nogpgcheck                    : True
[ceph_deploy.cli][INFO  ]  local_mirror                  : None
[ceph_deploy.cli][INFO  ]  release                       : None
[ceph_deploy.cli][INFO  ]  install_mon                   : False
[ceph_deploy.cli][INFO  ]  gpg_url                       : None
[ceph_deploy.install][DEBUG ] Installing stable version mimic on cluster ceph hosts ceph-master01 ceph-master02 ceph-master03
[ceph_deploy.install][DEBUG ] Detecting platform for host ceph-master01 ...
[ceph-master01][DEBUG ] connection detected need for sudo
[ceph-master01][DEBUG ] connected to host: ceph-master01 
[ceph-master01][DEBUG ] detect platform information from remote host
[ceph-master01][DEBUG ] detect machine type
[ceph_deploy.install][INFO  ] Distro info: Ubuntu 18.04 bionic
[ceph-master01][INFO  ] installing Ceph on ceph-master01
[ceph-master01][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q update
[ceph-master01][DEBUG ] Hit:1 https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific bionic InRelease
[ceph-master01][DEBUG ] Hit:2 https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-nautilus bionic InRelease
[ceph-master01][DEBUG ] Hit:3 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic InRelease
[ceph-master01][DEBUG ] Hit:4 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-updates InRelease
[ceph-master01][DEBUG ] Hit:5 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-backports InRelease
[ceph-master01][DEBUG ] Hit:6 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-security InRelease
[ceph-master01][DEBUG ] Reading package lists...
[ceph-master01][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q --no-install-recommends install ca-certificates apt-transport-https
[ceph-master01][DEBUG ] Reading package lists...
[ceph-master01][DEBUG ] Building dependency tree...
[ceph-master01][DEBUG ] Reading state information...
[ceph-master01][DEBUG ] ca-certificates is already the newest version (20211016~18.04.1).
[ceph-master01][DEBUG ] apt-transport-https is already the newest version (1.6.14).
[ceph-master01][DEBUG ] 0 upgraded, 0 newly installed, 0 to remove and 202 not upgraded.
[ceph-master01][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q update
[ceph-master01][DEBUG ] Hit:1 https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific bionic InRelease
[ceph-master01][DEBUG ] Hit:2 https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-nautilus bionic InRelease
[ceph-master01][DEBUG ] Hit:3 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic InRelease
[ceph-master01][DEBUG ] Hit:4 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-updates InRelease
[ceph-master01][DEBUG ] Hit:5 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-backports InRelease
[ceph-master01][DEBUG ] Hit:6 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-security InRelease
[ceph-master01][DEBUG ] Reading package lists...
[ceph-master01][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q --no-install-recommends install ceph ceph-osd ceph-mds ceph-mon radosgw
[ceph-master01][DEBUG ] Reading package lists...
[ceph-master01][DEBUG ] Building dependency tree...
[ceph-master01][DEBUG ] Reading state information...
[ceph-master01][DEBUG ] ceph is already the newest version (16.2.10-1bionic).
[ceph-master01][DEBUG ] ceph-mds is already the newest version (16.2.10-1bionic).
[ceph-master01][DEBUG ] ceph-mon is already the newest version (16.2.10-1bionic).
[ceph-master01][DEBUG ] ceph-osd is already the newest version (16.2.10-1bionic).
[ceph-master01][DEBUG ] radosgw is already the newest version (16.2.10-1bionic).
[ceph-master01][DEBUG ] 0 upgraded, 0 newly installed, 0 to remove and 202 not upgraded.
[ceph-master01][INFO  ] Running command: sudo ceph --version
[ceph-master01][DEBUG ] ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable)
[ceph_deploy.install][DEBUG ] Detecting platform for host ceph-master02 ...
[ceph-master02][DEBUG ] connection detected need for sudo
[ceph-master02][DEBUG ] connected to host: ceph-master02 
[ceph-master02][DEBUG ] detect platform information from remote host
[ceph-master02][DEBUG ] detect machine type
[ceph_deploy.install][INFO  ] Distro info: Ubuntu 18.04 bionic
[ceph-master02][INFO  ] installing Ceph on ceph-master02
[ceph-master02][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q update
[ceph-master02][DEBUG ] Hit:1 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic InRelease
[ceph-master02][DEBUG ] Hit:2 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-updates InRelease
[ceph-master02][DEBUG ] Get:3 https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific bionic InRelease [8,572 B]
[ceph-master02][DEBUG ] Hit:4 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-backports InRelease
[ceph-master02][DEBUG ] Get:5 https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-nautilus bionic InRelease [8,560 B]
[ceph-master02][DEBUG ] Hit:6 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-security InRelease
[ceph-master02][DEBUG ] Fetched 17.1 kB in 1s (13.1 kB/s)
[ceph-master02][DEBUG ] Reading package lists...
[ceph-master02][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q --no-install-recommends install ca-certificates apt-transport-https
[ceph-master02][DEBUG ] Reading package lists...
[ceph-master02][DEBUG ] Building dependency tree...
[ceph-master02][DEBUG ] Reading state information...
[ceph-master02][DEBUG ] ca-certificates is already the newest version (20211016~18.04.1).
[ceph-master02][DEBUG ] apt-transport-https is already the newest version (1.6.14).
[ceph-master02][DEBUG ] 0 upgraded, 0 newly installed, 0 to remove and 202 not upgraded.
[ceph-master02][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q update
[ceph-master02][DEBUG ] Hit:1 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic InRelease
[ceph-master02][DEBUG ] Hit:2 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-updates InRelease
[ceph-master02][DEBUG ] Hit:3 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-backports InRelease
[ceph-master02][DEBUG ] Hit:4 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-security InRelease
[ceph-master02][DEBUG ] Get:5 https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific bionic InRelease [8,572 B]
[ceph-master02][DEBUG ] Get:6 https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-nautilus bionic InRelease [8,560 B]
[ceph-master02][DEBUG ] Fetched 17.1 kB in 1s (12.5 kB/s)
[ceph-master02][DEBUG ] Reading package lists...
[ceph-master02][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q --no-install-recommends install ceph ceph-osd ceph-mds ceph-mon radosgw
[ceph-master02][DEBUG ] Reading package lists...
[ceph-master02][DEBUG ] Building dependency tree...
[ceph-master02][DEBUG ] Reading state information...
[ceph-master02][DEBUG ] ceph is already the newest version (16.2.10-1bionic).
[ceph-master02][DEBUG ] ceph-mds is already the newest version (16.2.10-1bionic).
[ceph-master02][DEBUG ] ceph-mon is already the newest version (16.2.10-1bionic).
[ceph-master02][DEBUG ] ceph-osd is already the newest version (16.2.10-1bionic).
[ceph-master02][DEBUG ] radosgw is already the newest version (16.2.10-1bionic).
[ceph-master02][DEBUG ] 0 upgraded, 0 newly installed, 0 to remove and 202 not upgraded.
[ceph-master02][INFO  ] Running command: sudo ceph --version
[ceph-master02][DEBUG ] ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable)
[ceph_deploy.install][DEBUG ] Detecting platform for host ceph-master03 ...
[ceph-master03][DEBUG ] connection detected need for sudo
[ceph-master03][DEBUG ] connected to host: ceph-master03 
[ceph-master03][DEBUG ] detect platform information from remote host
[ceph-master03][DEBUG ] detect machine type
[ceph_deploy.install][INFO  ] Distro info: Ubuntu 18.04 bionic
[ceph-master03][INFO  ] installing Ceph on ceph-master03
[ceph-master03][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q update
[ceph-master03][DEBUG ] Hit:1 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic InRelease
[ceph-master03][DEBUG ] Hit:2 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-updates InRelease
[ceph-master03][DEBUG ] Hit:3 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-backports InRelease
[ceph-master03][DEBUG ] Hit:4 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-security InRelease
[ceph-master03][DEBUG ] Get:5 https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific bionic InRelease [8,572 B]
[ceph-master03][DEBUG ] Get:6 https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-nautilus bionic InRelease [8,560 B]
[ceph-master03][DEBUG ] Fetched 17.1 kB in 2s (8,636 B/s)
[ceph-master03][DEBUG ] Reading package lists...
[ceph-master03][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q --no-install-recommends install ca-certificates apt-transport-https
[ceph-master03][DEBUG ] Reading package lists...
[ceph-master03][DEBUG ] Building dependency tree...
[ceph-master03][DEBUG ] Reading state information...
[ceph-master03][DEBUG ] ca-certificates is already the newest version (20211016~18.04.1).
[ceph-master03][DEBUG ] apt-transport-https is already the newest version (1.6.14).
[ceph-master03][DEBUG ] 0 upgraded, 0 newly installed, 0 to remove and 202 not upgraded.
[ceph-master03][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q update
[ceph-master03][DEBUG ] Hit:1 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic InRelease
[ceph-master03][DEBUG ] Hit:2 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-updates InRelease
[ceph-master03][DEBUG ] Hit:3 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-backports InRelease
[ceph-master03][DEBUG ] Hit:4 http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-security InRelease
[ceph-master03][DEBUG ] Get:5 https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific bionic InRelease [8,572 B]
[ceph-master03][DEBUG ] Get:6 https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-nautilus bionic InRelease [8,560 B]
[ceph-master03][DEBUG ] Fetched 17.1 kB in 1s (14.3 kB/s)
[ceph-master03][DEBUG ] Reading package lists...
[ceph-master03][INFO  ] Running command: sudo env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get --assume-yes -q --no-install-recommends install ceph ceph-osd ceph-mds ceph-mon radosgw
[ceph-master03][DEBUG ] Reading package lists...
[ceph-master03][DEBUG ] Building dependency tree...
[ceph-master03][DEBUG ] Reading state information...
[ceph-master03][DEBUG ] ceph is already the newest version (16.2.10-1bionic).
[ceph-master03][DEBUG ] ceph-mds is already the newest version (16.2.10-1bionic).
[ceph-master03][DEBUG ] ceph-mon is already the newest version (16.2.10-1bionic).
[ceph-master03][DEBUG ] ceph-osd is already the newest version (16.2.10-1bionic).
[ceph-master03][DEBUG ] radosgw is already the newest version (16.2.10-1bionic).
[ceph-master03][DEBUG ] 0 upgraded, 0 newly installed, 0 to remove and 202 not upgraded.
[ceph-master03][INFO  ] Running command: sudo ceph --version
[ceph-master03][DEBUG ] ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable)

此 过 程 会 在 指 定 的 ceph node 节 点 按 照 串 行 的 方 式 逐 个 服 务 器 安 装 ceph-base

ceph-common 等组件包：

在这里插入图片描述

2.3 安装ceph-mon服务
2.3.1 ceph-mon节点安装ceph-mon
cephadmin@ceph-master01:/etc/ceph-cluster# apt-cache madison ceph-mon
  ceph-mon | 16.2.10-1bionic | https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific bionic/main amd64 Packages
  ceph-mon | 14.2.22-1bionic | https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-nautilus bionic/main amd64 Packages
  ceph-mon | 12.2.13-0ubuntu0.18.04.10 | http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-updates/main amd64 Packages
  ceph-mon | 12.2.13-0ubuntu0.18.04.10 | http://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic-security/main amd64 Packages
  ceph-mon
最低0.47元/天 解锁文章

200万优质内容无限畅学
Kubernetes云原生存储解决方案之 Rook Ceph实践探究
lldhsds的专栏
09-29 1853
kubernetes下三位一体的云原生分布式存储解决方案
有状态容器实践：k8s集成ceph分布式存储
11-28
在部署容器集群的过程中，很多用户都在头疼存储的问题：数据怎样持久存储？数据怎样随容器迁移？
速通Ceph分布式存储（含超详细图解）
最新发布
weixin_45631123的博客
06-23 778
云计算存储架构图示例fill:#333;color:#333;color:#333;fill:none;important;important;important;important;important;important;important;important;important;important;important;important;important;important;important;important;important;important;important;important;
Kubernetes（K8s）上使用分布式存储（Distributed Storage）
多头注意力探索Now
09-11 2089
分布式存储在K8s上的应用
Ceph对接K8S
屈帅波的技术博客
01-02 2642
1.PV、PVC概述 管理存储是管理计算的一个明显问题。PersistentVolume子系统为用户和管理员提供了一个API，用于抽象如何根据消费方式提供存储的详细信息。于是引入了两个新的API资源：PersistentVolume和PersistentVolumeClaim PersistentVolume（PV）是集群中已由管理员配置的一段网络存储。 集群中的资源就像一个节点是一个集群资源。 ...
K8S对接Ceph分布式存储
秦子腾
08-25 2526
Ceph 是一个开源的分布式存储系统，用于提供高性能、高可靠性的对象存储、块存储和文件系统，它通过去中心化的架构，自动处理数据复制和恢复，旨在提供弹性扩展和容错能力。Ceph包含一下几种存储类型：（1）块存储(RBD)：用途： 提供了高性能的块级存储，类似于传统的磁盘驱动器。通常用于虚拟机的存储或数据库的存储。接口： 通过 Ceph 的 RADOS（Reliable Autonomic Distributed Object Store）接口提供块设备。
【云原生】Ceph 在 k8s中应用
匠人精神，持之以恒！
01-07 4100
分布式存储系统 Ceph 介绍与环境部署分布式存储系统 Ceph 实战操作Kubernetes（k8s）最新版最完整版环境部署+master高可用实现（k8sV1.24.1+dashboard+harbor）Rook是一个开源的云原生存储编排工具，提供平台、框架和对各种存储解决方案的支持，以和云原生环境进行本地集成。Rook 将存储软件转变成自我管理、自我扩展和自我修复的存储服务，通过自动化部署、启动、配置、供应、扩展、升级、迁移、灾难恢复、监控和资源管理来实现。
云原生-基于k8s部署-分布式底层存储-Rook-Ceph
06-13
通过这个工具，你可以查看集群状态、监控安装进度、配置Ceph-Dashboard以及进行硬盘格式化和Osd的添加或删除等操作。 在使用过程中，可能会遇到一些问题，例如删除集群时卡住，这时可以按照教程提供的命令进行操作...
K8s-ceph-csi-rbd连接资源
06-15
在K8s-ceph-csi-rbd场景中，`rbdcsi`是实现了CSI接口的Ceph RBD驱动。它允许Kubernetes集群直接使用Ceph RBD卷，提供了无缝的存储集成。为了实现这个，你需要： 1. **部署Ceph RBD CSI Driver**：这包括部署csi-...
k8s-rook-ceph
03-02
`k8s-rook-ceph-master` 可能包含这些配置文件，用于初始化和管理 Rook-Ceph 集群。 4. **持久化卷（Persistent Volumes, PV）与持久卷声明（Persistent Volume Claims, PVC）** 在 Kubernetes 中，PV 代表集群中...
k8s 使用ceph
discsthnew的博客
03-25 3831
简单使用 通过StorageClass动态分配 存储卷分配方式 静态分配 动态分配 踩坑 简单使用 关于k8s 简单实用ceph, 以下两篇文章给出了详细步骤和要注意的事项： https://github.com/kubernetes/kubernetes/tree/master/examples/volumes/cephfs http://tonybai.com/201...
（k8s)kubernetes 使用 Ceph RBD 作为后端存储（静态供给、动态供给）
Richardlygo的博客
05-10 512
允许用户创建虚拟块设备，并将其映射到客户端系统中，就像本地磁盘一样使用。存储集群中的一个重要组件，它提供了块级别的存储访问。需要绑定一个存储池，创建一个名为。静态供给方式需要提前创建好。命令，所以在开始前需要安装。，并初始化该存储池给。
k8s的ceph
fengge55的博客
03-30 4782
ceph安装 地址：https://rook.io/docs/rook/v1.8/quickstart.html 特性丰富 1，支持三种存储接口：块存储、文件存储、对象存储。 2，支持自定义接口，支持多种语言驱动。 基本概念 Ceph OSD Object Storage Device是ceph的核心组件，用于存储数据，处理数据的复制、恢复、回填、再均衡，并通过检查其他OSD守护进程的心跳来向Ceph Monitors提供一些监控信息。ceph集群每台机器上的每块盘需要运行一个OSD进程。 每个OSD
k8s使用ceph存储
梵修
06-22 2562
另外keyring文件也可以保存在secret中，在pod中通过secret来引用keyring文件。k8s集群中的pod使用rbd时可以直接通过pod的volume进行挂载，也可以以pv形式挂载使用。ceph添加授权，需要创建两个用户，一个挂载rbd时使用，另一个挂载cephfs时使用。创建pod到集群中，等待pod就绪后可以进入pod验证。将pod创建到集群中，等待pod就绪后进入pod验证。将pod创建到集群中，等待pod就绪后进入pod验证。将pod创建到集群中，等待pod就绪后进入pod验证。
k8s挂载使用ceph集群
k8seasz的博客
05-22 4053
我们都知道ceph集群是一个开源的分布式存储系统，支持对象存储，块存储，文件系统。本篇文章就是要讲解下k8s下如何挂载使用ceph的块存储系统。 示意图 一：ceph集群创建存储池及存储镜像 #在ceph机器中创建名字为lilh-rbd-pool 的存储池 ceph osd pool create lilh-rbd-pool 9 9 #创建完成后查看ceph机器存储池信息 ceph osd pool ls #启用快存储 ceph osd pool applica...
K8S实战集训第二课 K8S 存储 之 Ceph 分布式存储系统
fegus的博客
05-30 2033
文章目录为什么要用CephCeph架构介绍Ceph核心概念RADOSLibradosCrushPoolPGObjectCeph核心组件OSDMonitorMDSMgrRGWAdminCeph三种存储类型1、 块存储（RBD）2、文件存储（CephFS）3、对象存储（Object）(适合更新变动较少的数据)一、Ceph版本选择Ceph版本来源介绍Luminous新版本特性二、安装前准备三、安装内网yum源四、安装Ceph集群五、ceph.confRBD介绍RBD常用命令RBD配置操作RBD挂载到操作系统快照配
k8s使用ceph实现动态持久化存储
weixin_34034261的博客
12-17 422
简介 本文章介绍如何使用ceph为k8s提供动态申请pv的功能。ceph 提供底层存储功能，cephfs方式支持k8s的pv的3种访问模式ReadWriteOnce，ReadOnlyMany ，ReadWriteMany ，RBD支持ReadWriteOnce，ReadOnlyMany两种模式 访问模式只是能力描述，并不是强制执行的，对于没有按pvc声明...
K8s 使用 CephFS 作为后端存储（静态供给、动态供给）
小毕超博客
04-27 1720
CephFS是 Ceph 中基于RADOS（可扩展分布式对象存储）构建，通过将文件数据划分为对象并分布到集群中的多个存储节点上来实现高可用性和可扩展性。
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值