转载自：https://cloud.tencent.com/developer/article/1010539
1、Ceph 介绍
Ceph 是一个开源的分布式存储系统，包括对象存储、块设备、文件系统。它具有高可靠性、安装方便、管理简便、能够轻松管理海量数据。Ceph 存储集群具备了企业级存储的能力，它通过组织大量节点，节点之间靠相互通讯来复制数据、并动态地重分布数据，从而达到高可用分布式存储功能。Ceph 经过七年的发展，现在越来越完善，吸引了更多的参与者，相信以后会更加流行。
2、环境、软件准备
本次演示环境，我是在虚拟机 Linux Centos7 上操作，通过虚拟机完成存储集群搭建，以下是安装的软件及版本：
- Centos：release 7.4.1708 (Core)
- Ceph：jewel-10.2.10
- Openssh-server：version 7.4
- NTP
注意： 这篇文章只涉及 Ceph 存储集群搭建过程，不在详细阐述 Ceph 的体系架构，以及各个组件的详细情况，具体可参考 官方系统结构 中有详细描述文档。Ceph 官方文档 中建议安装一个 ceph-deploy 管理节点和一个三节点的 Ceph 存储集群来研究 Ceph 的基本特性，结构图如下：
不过，这里受限于本机内存限制，虚拟机不能开的太多，所以这里我少一个 mon.node1 节点，将该节点 Monitor 功能迁移到 admin-node 节点上，所以集群结构图变成如下：
Ceph 分布式存储集群有三大组件组成，分为：Ceph Monitor、Ceph OSD、Ceph MDS，后边使用对象存储和块存储时，MDS 非必须安装，只有当使用 Cephfs 文件存储时，才需要安装。这里我们暂时不安装 MDS。
3、Ceph 预检
3.1 配置节点Host
为了方便后边安装，以及 ssh 方式连接各个节点，我们先修改一下各个节点的 Hostname 以及配置 Hosts 如下：
admin-node (10.222.77.213)
$ cat /etc/hostname
admin
$ cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.222.77.213 admin
10.222.77.242 node0
10.222.77.253 node1
-----------------------------
node0（10.222.77.242）
$ cat /etc/hostname
admin
$ cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.222.77.242 node0
10.222.77.213 admin
10.222.77.253 node1
-----------------------------
node1（10.222.77.253）
$ cat /etc/hostname
admin
$ cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.222.77.253 node1
10.222.77.213 admin
10.222.77.242 node0
3.2 安装部署工具 ceph-deploy
Ceph 提供了部署工具 ceph-deploy 来方便安装 Ceph 集群，我们只需要在 ceph-deploy 节点上安装即可，这里对应的就是 admin-node 节点。把 Ceph 仓库添加到 ceph-deploy 管理节点，然后安装 ceph-deploy。因为系统是 Centos7 版本，所以配置如下：
# ceph-deploy (admin-node) 上执行
# yum 配置其他依赖包
$ sudo yum install -y yum-utils && sudo yum-config-manager --add-repo https://dl.fedoraproject.org/pub/epel/7/x86_64/ && sudo yum install --nogpgcheck -y epel-release && sudo rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 && sudo rm /etc/yum.repos.d/dl.fedoraproject.org*
# 添加 Ceph 源
$ sudo vim /etc/yum.repos.d/ceph.repo
[Ceph-noarch]
name=Ceph noarch packages
baseurl=http://download.ceph.com/rpm-jewel/el7/noarch
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://download.ceph.com/keys/release.asc
priority=1
# 安装 ceph-deploy
$ sudo yum update && sudo yum install ceph-deploy
3.3 安装 NTP 和 Openssh
官方建议在所有 Ceph 节点上安装 NTP 服务（特别是 Ceph Monitor 节点），以免因时钟漂移导致故障。
# yum 安装 ntp
sudo yum install ntp ntpdate ntp-doc
# 校对系统时钟
ntpdate 0.cn.pool.ntp.org
后续操作，ceph-deploy 节点需要使用 ssh 方式登录各个节点完成 ceph 安装配置工作，所以要确保各个节点上有可用 SSH 服务。
# yum 安装 openssh
$ sudo yum install openssh-server
# 查看 ssh 版本
$ ssh -V
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017
3.4 创建 Ceph 部署用户
ceph-deploy 工具必须以普通用户登录 Ceph 节点，且此用户拥有无密码使用 sudo 的权限，因为它需要在安装软件及配置文件的过程中，不必输入密码。官方建议所有 Ceph 节点上给 ceph-deploy 创建一个特定的用户，而且不要使用 ceph 这个名字。这里为了方便，我们使用 cephd 这个账户作为特定的用户，而且每个节点上（admin-node、node0、node1）上都需要创建该账户，并且拥有 sudo 权限。
# 在 Ceph 集群各节点进行如下操作
# 创建 ceph 特定用户
$ sudo useradd -d /home/cephd -m cephd
$ sudo passwd cephd
# 添加 sudo 权限
$ echo "cephd ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/cephd
$ sudo chmod 0440 /etc/sudoers.d/cephd
接下来在 ceph-deploy 节点（admin-node）上，切换到 cephd 用户，生成 SSH 密钥并把其公钥分发到各 Ceph 节点上，注意使用 cephd 账户生成，且提示输入密码时，直接回车，因为它需要免密码登录到各个节点。
# ceph-deploy (admin-node) 上执行
# 生成 ssh 密钥
$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/cephd/.ssh/id_rsa):
Created directory '/home/cephd/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/cephd/.ssh/id_rsa.
Your public key has been saved in /home/cephd/.ssh/id_rsa.pub.
The key fingerprint is:
...
# 将公钥复制到 node0 节点
$ ssh-copy-id cephd@node0
/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/cephd/.ssh/id_rsa.pub"
The authenticity of host 'node0 (10.222.77.242)' can't be established.
ECDSA key fingerprint is MD5:3c:e0:a7:a0:e6:3c:dc:c0:df:28:dc:87:16:2d:0f:c6.
Are you sure you want to continue connecting (yes/no)? yes
/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
cephd@node0's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'cephd@node0'"
and check to make sure that only the key(s) you wanted were added.
# 将公钥复制到 node1 节点
$ ssh-copy-id cephd@node1
/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/cephd/.ssh/id_rsa.pub"
The authenticity of host 'node1 (10.222.77.253)' can't be established.
ECDSA key fingerprint is MD5:3c:e0:a7:a0:e6:3c:dc:c0:df:28:dc:87:16:2d:0f:c6.
Are you sure you want to continue connecting (yes/no)? yes
/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
cephd@node1's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'cephd@node1'"
and check to make sure that only the key(s) you wanted were added.
复制完毕，测试一下在 ceph-deploy 管理节点免密码登录各个节点。
$ ssh node0
Last login: Fri Dec 8 15:50:08 2017 from admin
$ ssh node1
Last login: Fri Dec 8 15:49:27 2017 from admin
测试没有问题，接下来，修改 ceph-deploy 管理节点上的 ~/.ssh/config 文件，这样无需每次执行 ceph-deploy 都要指定 –username cephd 。这样做同时也简化了 ssh 和 scp 的用法。
$ cat ~/.ssh/config
Host node0
Hostname node0
User cephd
Host node1
Hostname node1
User cephd
注意，此时再执行 ssh node0 会提示报错 Bad owner or permissions on /home/cephd/.ssh/config。原因是 config 文件权限问题，修改权限 sudo chmod 600 config 即可解决。
3.5 其他网络配置
官网文档中指定 Ceph 的各 OSD 进程通过网络互联并向 Monitors 上报自己的状态，所以要保证网络为开启状态，不过某些发行版（如 CentOS ）默认关闭网络接口。所以我们需要保证集群各个节点系统网络接口是开启的。
# 在 Ceph 集群各节点进行如下操作
$ sudo cat /etc/sysconfig/network-scripts/ifcfg-enp0s3
TYPE="Ethernet"
BOOTPROTO="dhcp"
DEFROUTE="yes"
PEERDNS="yes"
PEERROUTES="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_PEERDNS="yes"
IPV6_PEERROUTES="yes"
IPV6_FAILURE_FATAL="no"
IPV6_ADDR_GEN_MODE="stable-privacy"
NAME="enp0s3"
UUID="3e68d5a3-f9a6-4c83-9969-706f7e3b0bc2"
DEVICE="enp0s3"
ONBOOT="yes" # 这里要设置为 yes
注意：这里因为在我安装的虚拟机集群中网卡为 enp0s3，所以需要修改 /etc/sysconfig/network-scripts/ifcfg-enp0s3 文件，请根据自己系统网卡名去修改对应配置文件。
SELINUX 设置，在 CentOS 系统上， SELinux 默认为 Enforcing 开启状态，为了方便安装，建议把 SELinux 设置为 Permissive 或者 disabled。
# 在 Ceph 集群各节点进行如下操作
# 临时生效设置
$ sudo setenforce 0
# 永久生效设置
$ sudo cat /etc/selinux/config
SELINUX=disabled # 这里设置为 Permissive | disabled
SELINUXTYPE=targeted
开放所需端口设置，Ceph Monitors 之间默认使用 6789 端口通信， OSD 之间默认用 6800:7300 这个范围内的端口通信，所以我们需要调整防火墙设置，开放所需端口，允许相应的入站请求。
# 防火墙设置
$ sudo firewall-cmd --zone=public --add-port=6789/tcp --permanent
# 当然我们也可以关闭防火墙
$ sudo systemctl stop firewalld.service #停止 firewall
$ sudo systemctl disable firewalld.service #禁止 firewall 开机启动
4、Ceph 存储集群搭建
好了，经过上边一系列的预检设置后，我们就可以开始 Ceph 存储集群搭建了，集群结构为 admin-node (ceph-deploy、Monitor)、node0（osd.0）、node1（osd.1）。首先要提一下的是，如果我们在安装过程中出现了问题，需要重新操作的时候，例如想清理我搭建的这个集群的话，可以使用以下命令。
# ceph-deploy (admin-node) 上执行
# 清理配置
ceph-deploy purgedata admin node0 node1
ceph-deploy forgetkeys
# 清理 Ceph 安装包
ceph-deploy purge admin node0 node1
好了，现在开始搭建。首先 Cephd 用户创建一个目录 ceph-cluster 并进入到该目录执行一系列操作。因为我们设计的 monitor 节点在 admin-node 节点上，所以，执行如下命令。
# 创建执行目录
$ mkdir ~/ceph-cluster && cd ~/ceph-cluster
# 创建集群
$ ceph-deploy new admin
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephd/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.39): /bin/ceph-deploy new admin
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] func : <function new at 0xf24938>
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0xf1f6c8>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] ssh_copykey : True
[ceph_deploy.cli][INFO ] mon : ['admin']
[ceph_deploy.cli][INFO ] public_network : None
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] cluster_network : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] fsid : None
[ceph_deploy.new][DEBUG ] Creating new cluster named ceph
[ceph_deploy.new][INFO ] making sure passwordless SSH succeeds
...
[ceph_deploy.new][DEBUG ] Resolving host admin
[ceph_deploy.new][DEBUG ] Monitor admin at 10.222.77.213
[ceph_deploy.new][DEBUG ] Monitor initial members are ['admin']
[ceph_deploy.new][DEBUG ] Monitor addrs are ['10.222.77.213']
[ceph_deploy.new][DEBUG ] Creating a random mon key...
[ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring...
[ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf...
此时，我们会发现 ceph-deploy 会在 ceph-cluster 目录下生成几个文件，ceph.conf 为 ceph 配置文件，ceph-deploy-ceph.log 为 ceph-deploy 日志文件，ceph.mon.keyring 为 ceph monitor 的密钥环。
$ ll ceph-cluster
-rw-rw-r--. 1 cephd cephd 196 12月 7 14:46 ceph.conf
-rw-rw-r--. 1 cephd cephd 3694 12月 7 14:46 ceph-deploy-ceph.log
-rw-------. 1 cephd cephd 73 12月 7 14:46 ceph.mon.keyring
$ cat ceph.conf
[global]
fsid = 363c3cf4-dba4-4f13-8ec5-ed93416f4e26
mon_initial_members = admin
mon_host = 10.222.77.213
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
接下来，我们需要修改下 ceph.conf 配置文件，增加副本数为 2，因为我们有两个 osd 节点。
$ cat ceph.conf
[global]
fsid = 363c3cf4-dba4-4f13-8ec5-ed93416f4e26
mon_initial_members = admin
mon_host = 10.222.77.213
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
osd pool default size = 2 #增加默认副本数为 2
然后，我们需要通过 ceph-deploy 在各个节点安装 ceph。
$ ceph-deploy install admin node0 node1
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephd/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.39): /bin/ceph-deploy install admin node0 node1
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] testing : None
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0xde1c68>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] dev_commit : None
[ceph_deploy.cli][INFO ] install_mds : False
[ceph_deploy.cli][INFO ] stable : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] adjust_repos : True
[ceph_deploy.cli][INFO ] func : <function install at 0xd5b140>
[ceph_deploy.cli][INFO ] install_mgr : False
[ceph_deploy.cli][INFO ] install_all : False
[ceph_deploy.cli][INFO ] repo : False
[ceph_deploy.cli][INFO ] host : ['admin', 'node0', 'node1']
[ceph_deploy.cli][INFO ] install_rgw : False
[ceph_deploy.cli][INFO ] install_tests : False
[ceph_deploy.cli][INFO ] repo_url : None
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] install_osd : False
[ceph_deploy.cli][INFO ] version_kind : stable
[ceph_deploy.cli][INFO ] install_common : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] dev : master
[ceph_deploy.cli][INFO ] nogpgcheck : False
[ceph_deploy.cli][INFO ] local_mirror : None
[ceph_deploy.cli][INFO ] release : None
[ceph_deploy.cli][INFO ] install_mon : False
[ceph_deploy.cli][INFO ] gpg_url : None
[ceph_deploy.install][DEBUG ] Installing stable version jewel on cluster ceph hosts admin node0 node1
[ceph_deploy.install][DEBUG ] Detecting platform for host admin ...
...
此过程需要等待一段时间，因为 ceph-deploy 会 SSH 登录到各 node 上去，依次执行安装 ceph 依赖的组件包。
漫长的等待安装完毕之后，接下来需要初始化 monitor 节点并收集所有密钥。
$ ceph-deploy mon create-initial
...
ceph_deploy.mon][ERROR ] RuntimeError: config file /etc/ceph/ceph.conf exists with different content; use --overwrite-conf to overwrite
[ceph_deploy][ERROR ] GenericError: Failed to create 1 monitors
...
不过很遗憾，执行过程中报错了。查看原因应该是已经存在了 /etc/ceph/ceph.conf 配置文件了，解决方案就是加上 --overwrite-conf 参数，覆盖已存在的配置。
$ ceph-deploy --overwrite-conf mon create-initial
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephd/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.39): /bin/ceph-deploy --overwrite-conf mon create-initial
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] overwrite_conf : True
[ceph_deploy.cli][INFO ] subcommand : create-initial
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x1d9e758>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] func : <function mon at 0x1d8ea28>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] keyrings : None
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts admin
[ceph_deploy.mon][DEBUG ] detecting platform for host admin ...
...
[admin][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --admin-daemon=/var/run/ceph/ceph-mon.admin.asok mon_status
[admin][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-admin/keyring auth get client.admin
[admin][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-admin/keyring auth get client.bootstrap-mds
[admin][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-admin/keyring auth get client.bootstrap-mgr
[admin][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-admin/keyring auth get-or-create client.bootstrap-mgr mon allow profile bootstrap-mgr
[admin][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-admin/keyring auth get client.bootstrap-osd
[admin][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-admin/keyring auth get client.bootstrap-rgw
[ceph_deploy.gatherkeys][INFO ] Storing ceph.client.admin.keyring
[ceph_deploy.gatherkeys][INFO ] Storing ceph.bootstrap-mds.keyring
[ceph_deploy.gatherkeys][INFO ] Storing ceph.bootstrap-mgr.keyring
[ceph_deploy.gatherkeys][INFO ] keyring 'ceph.mon.keyring' already exists
[ceph_deploy.gatherkeys][INFO ] Storing ceph.bootstrap-osd.keyring
[ceph_deploy.gatherkeys][INFO ] Storing ceph.bootstrap-rgw.keyring
[ceph_deploy.gatherkeys][INFO ] Destroy temp directory /tmp/tmpLv4mug
执行完毕后，会在当前目录下生成一系列的密钥环，应该是各组件之间访问所需要的认证信息吧。
$ ll ~/ceph-cluster
-rw-------. 1 cephd cephd 113 12月 7 15:13 ceph.bootstrap-mds.keyring
-rw-------. 1 cephd cephd 71 12月 7 15:13 ceph.bootstrap-mgr.keyring
-rw-------. 1 cephd cephd 113 12月 7 15:13 ceph.bootstrap-osd.keyring
-rw-------. 1 cephd cephd 113 12月 7 15:13 ceph.bootstrap-rgw.keyring
-rw-------. 1 cephd cephd 129 12月 7 15:13 ceph.client.admin.keyring
-rw-rw-r--. 1 cephd cephd 222 12月 7 14:47 ceph.conf
-rw-rw-r--. 1 cephd cephd 120207 12月 7 15:13 ceph-deploy-ceph.log
-rw-------. 1 cephd cephd 73 12月 7 14:46 ceph.mon.keyring
到此，ceph monitor 已经成功启动了。接下来需要创建 OSD 了，OSD 是最终数据存储的地方，这里我们准备了两个 OSD 节点，分别为 osd.0 和 osd.1。官方建议为 OSD 及其日志使用独立硬盘或分区作为存储空间，不过本机虚拟机上不具备条件，但是我们可以在虚拟机本地磁盘上创建目录，来作为 OSD 的存储空间。
# ceph-deploy (admin-node) 上执行
$ ssh node0
$ sudo mkdir /var/local/osd0
$ sudo chown -R ceph:ceph /var/local/osd0
$ exit
$ ssh node1
$ sudo mkdir /var/local/osd1
$ sudo chown -R ceph:ceph /var/local/osd1
$ exit
注意：这里执行了 chown -R ceph:ceph 操作，将 osd0 和 osd1 目录的权限赋予 ceph:ceph，否则，接下来执行 ceph-deploy osd activate ... 时会报权限错误。
接下来，我们需要 ceph-deploy 节点执行 prepare OSD 操作，目的是分别在各个 OSD 节点上创建一些后边激活 OSD 需要的信息。
$ ceph-deploy --overwrite-conf osd prepare node0:/var/local/osd0 node1:/var/local/osd1
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephd/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.39): /bin/ceph-deploy --overwrite-conf osd prepare node0:/var/local/osd0 node1:/var/local/osd1
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] block_db : None
[ceph_deploy.cli][INFO ] disk : [('node0', '/var/local/osd0', None), ('node1', '/var/local/osd1', None)]
[ceph_deploy.cli][INFO ] dmcrypt : False
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] bluestore : None
[ceph_deploy.cli][INFO ] block_wal : None
[ceph_deploy.cli][INFO ] overwrite_conf : True
[ceph_deploy.cli][INFO ] subcommand : prepare
[ceph_deploy.cli][INFO ] dmcrypt_key_dir : /etc/ceph/dmcrypt-keys
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x117d950>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] fs_type : xfs
[ceph_deploy.cli][INFO ] filestore : None
[ceph_deploy.cli][INFO ] func : <function osd at 0x116a320>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] zap_disk : False
[ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks node0:/var/local/osd0: node1:/var/local/osd1:
[node0][DEBUG ] connection detected need for sudo
[node0][DEBUG ] connected to host: node0
[node0][DEBUG ] detect platform information from remote host
[node0][DEBUG ] detect machine type
[node0][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO ] Distro info: CentOS Linux 7.3.1611 Core
[ceph_deploy.osd][DEBUG ] Deploying osd to node0
...
[node0][INFO ] checking OSD status...
[node0][DEBUG ] find the location of an executable
[node0][INFO ] Running command: sudo /bin/ceph --cluster=ceph osd stat --format=json
[ceph_deploy.osd][DEBUG ] Host node0 is now ready for osd use.
[node1][DEBUG ] connection detected need for sudo
[node1][DEBUG ] connected to host: node1
[node1][DEBUG ] detect platform information from remote host
[node1][DEBUG ] detect machine type
[node1][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO ] Distro info: CentOS Linux 7.3.1611 Core
[ceph_deploy.osd][DEBUG ] Deploying osd to node1
...
[node1][INFO ] checking OSD status...
[node1][DEBUG ] find the location of an executable
[node1][INFO ] Running command: sudo /bin/ceph --cluster=ceph osd stat --format=json
[ceph_deploy.osd][DEBUG ] Host node1 is now ready for osd use.
OK 接下来，我们需要激活 activate OSD。
$ ceph-deploy osd activate node0:/var/local/osd0 node1:/var/local/osd1
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephd/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.39): /bin/ceph-deploy osd activate node0:/var/local/osd0 node1:/var/local/osd1
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] subcommand : activate
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x2045950>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] func : <function osd at 0x2032320>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] disk : [('node0', '/var/local/osd0', None), ('node1', '/var/local/osd1', None)]
[ceph_deploy.osd][DEBUG ] Activating cluster ceph disks node0:/var/local/osd0: node1:/var/local/osd1:
[node0][DEBUG ] connection detected need for sudo
[node0][DEBUG ] connected to host: node0
[node0][DEBUG ] detect platform information from remote host
[node0][DEBUG ] detect machine type
[node0][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO ] Distro info: CentOS Linux 7.3.1611 Core
[ceph_deploy.osd][DEBUG ] activating host node0 disk /var/local/osd0
[ceph_deploy.osd][DEBUG ] will use init type: systemd
...
[node0][INFO ] checking OSD status...
[node0][DEBUG ] find the location of an executable
[node0][INFO ] Running command: sudo /bin/ceph --cluster=ceph osd stat --format=json
[node0][INFO ] Running command: sudo systemctl enable ceph.target
[node1][DEBUG ] connection detected need for sudo
[node1][DEBUG ] connected to host: node1
[node1][DEBUG ] detect platform information from remote host
[node1][DEBUG ] detect machine type
[node1][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO ] Distro info: CentOS Linux 7.3.1611 Core
[ceph_deploy.osd][DEBUG ] activating host node1 disk /var/local/osd1
[ceph_deploy.osd][DEBUG ] will use init type: systemd
...
[node1][INFO ] checking OSD status...
[node1][DEBUG ] find the location of an executable
[node1][INFO ] Running command: sudo /bin/ceph --cluster=ceph osd stat --format=json
[node1][INFO ] Running command: sudo systemctl enable ceph.target
看日志，激活也没有问题，最后一步，通过 ceph-deploy admin 将配置文件和 admin 密钥同步到各个节点，以便在各个 Node 上使用 ceph 命令时，无需指定 monitor 地址和 ceph.client.admin.keyring 密钥。
$ ceph-deploy admin admin node0 node1
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephd/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.39): /bin/ceph-deploy admin admin node0 node1
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x15a93b0>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] client : ['admin', 'node0', 'node1']
[ceph_deploy.cli][INFO ] func : <function admin at 0x1500cf8>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to admin
[admin][DEBUG ] connection detected need for sudo
[admin][DEBUG ] connected to host: admin
[admin][DEBUG ] detect platform information from remote host
[admin][DEBUG ] detect machine type
[admin][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node0
[node0][DEBUG ] connection detected need for sudo
[node0][DEBUG ] connected to host: node0
[node0][DEBUG ] detect platform information from remote host
[node0][DEBUG ] detect machine type
[node0][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node1
[node1][DEBUG ] connection detected need for sudo
[node1][DEBUG ] connected to host: node1
[node1][DEBUG ] detect platform information from remote host
[node1][DEBUG ] detect machine type
[node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
同时为了确保对 ceph.client.admin.keyring 有正确的操作权限，所以还需要增加权限设置。
$ sudo chmod +r /etc/ceph/ceph.client.admin.keyring
至此，Ceph 存储集群已经搭建完毕了，我们可以查看那一下集群是否启动成功！
# 查看集群状态
$ ceph -s
cluster 363c3cf4-dba4-4f13-8ec5-ed93416f4e26
health HEALTH_OK
monmap e1: 1 mons at {admin=10.222.77.213:6789/0}
election epoch 6, quorum 0 admin
osdmap e15: 2 osds: 2 up, 2 in
flags sortbitwise,require_jewel_osds
pgmap v9276: 64 pgs, 1 pools, 863 MB data, 226 objects
33440 MB used, 33574 MB / 67015 MB avail
64 active+clean
# 或者查看集群健康状况
$ ceph health
HEALTH_OK
# 查看集群 OSD 信息
$ ceph osd tree
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 0.06400 root default
-2 0.03200 host node0
0 0.03200 osd.0 up 1.00000 1.00000
-3 0.03200 host node1
1 0.03200 osd.1 up 1.00000 1.00000
好了，通过参考官方文档中的 Ceph 安装和集群搭建说明，一步一步的完成整个存储集群的搭建，文档写的很详细，非常容易上手操作。本次就先记录到这，下一篇我们继续研究学习 Ceph 存储体系中的对象存储、块设备和文件系统。
参考资料