Ceph集群

Ceph集群

官方网站:
http://ceph.com
https://github.com/ceph

环境:
CentOS7.2
ceph10.2.5

192.168.8.254 (admin/client node)
192.168.8.101 (mon node)
192.168.8.102 (ceph node)
192.168.8.103 (ceph node)




基础环境
http://docs.ceph.com/docs/master/start/
一.安装ceph-deploy(admin node)
yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
cat >/etc/yum.repos.d/ceph.repo <<HERE
[ceph-noarch]
name=Ceph noarch packages
baseurl=https://download.ceph.com/rpm-jewel/el7/noarch/
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://download.ceph.com/keys/release.asc
HERE
提示:ceph-deploy仅仅是一个python脚本，用来管理和部署ceph。
版本选择最新的LTS版 Jewel http://docs.ceph.com/docs/master/releases/
yum -y install ceph-deploy

二.基本设置
1.ntp,ssh(所有节点)
yum -y install ntp ntpdate openssh-server openssh-client
ssh通常默认己安装，ntp指向内网ntp server,具体配置略
2.创建ceph-deploy管理用户(所有节点)
useradd -r -m ceph
echo ceph:ceph|chpasswd
echo "ceph ALL = (root) NOPASSWD:ALL" > /etc/sudoers.d/ceph
chmod 0440 /etc/sudoers.d/ceph
3.免密码登录(admin node)
su - ceph
ssh-keygen -t rsa -N '' -f ~/.ssh/id_rsa -q -b 2048
ssh-copy-id 192.168.8.254
ssh-copy-id 192.168.8.101
ssh-copy-id 192.168.8.102
ssh-copy-id 192.168.8.103
在admin节点上用ceph用户在所有节点上做免密码认证
提示:避免每次ceph-deploy时指定--username参数，可以添加如下
cat >~/.ssh/config <<HERE
Host 192.168.8.254
   Hostname client-node
   User ceph
Host 192.168.8.254
   Hostname admin-node
   User ceph
Host 192.168.8.101
   Hostname node1
   User ceph
Host 192.168.8.102
   Hostname node2
   User ceph
Host 192.168.8.103
   Hostname node3
   User ceph
HERE
chmod 0644 ~/.ssh/config
4.ceph-deploy管理用户不强制tty登录(所有节点)
sed -i '/Defaults    requiretty/c Defaults:ceph !requiretty' /etc/sudoers


存储集群
添加hosts记录，方便后续操作
192.168.8.254 client-node client.example.com
192.168.8.254 admin-node admin.example.com
192.168.8.101 node1 node1.example.com
192.168.8.102 node2 node2.example.com
192.168.8.103 node3 node3.example.com

集群管理都在admin node节点上操作
一.初始化集群
http://docs.ceph.com/docs/master/start/quick-ceph-deploy/
1.创建monitor
su - ceph
mkdir my-cluster
cd my-cluster
ceph-deploy new node1

[ceph@router my-cluster]$ pwd
/home/ceph/my-cluster
[ceph@router my-cluster]$ ls
ceph.conf  ceph-deploy-ceph.log  ceph.mon.keyring
注意:monitor在整个集群中应以奇数个数存在
2.修改默认replica数量
http://docs.ceph.com/docs/master/rados/configuration/mon-config-ref/
cat ~/my-cluster/ceph.conf
[global]
fsid = 9a776b16-1e40-45de-a66b-afd4ba689fd5
mon_initial_members = node1
mon_host = 192.168.8.101
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

osd pool default size = 2
3.安装ceph
ceph-deploy purgedata admin-node node1 node2 node3
ceph-deploy install admin-node node1 node2 node3
提示:如果己做好了本地yum源，也可直接在节点上执行 yum -y install ceph ceph-radosgw来安装
4.初始化monitor节点
ceph-deploy mon create-initial

[ceph@router my-cluster]$ ls
ceph.bootstrap-mds.keyring  ceph.bootstrap-rgw.keyring  ceph.conf             ceph.mon.keyring
ceph.bootstrap-osd.keyring  ceph.client.admin.keyring   ceph-deploy-ceph.log
初始化完成会生成4个keyring文件
5.添加osd磁盘
node2 /dev/vdb
node3 /dev/vdb
kvm虚拟，各添加10T的磁盘/dev/vdb
6.激活osd磁盘
ceph-deploy osd prepare node2:/dev/vdb node3:/dev/vdb
ceph-deploy osd activate node2:/dev/vdb1 node3:/dev/vdb1
ceph-deploy osd list node2 node3
提示: 可以直接ceph-deploy osd create完成上述的prepare和activate。

重复做实验可以先还原后再创建，如

删除现有分区表和磁盘内容

ceph-deploy disk zap node1:sdb node1:sdc node1:sdd

ceph-deploy disk zap node2:sdb node2:sdc node2:sdd

ceph-deploy disk zap node3:sdb node3:sdc node3:sdd

准备并激活磁盘

ceph-deploy osd create node1:sdb node1:sdc node1:sdd

ceph-deploy osd create node2:sdb node2:sdc node2:sdd

ceph-deploy osd create node3:sdb node3:sdc node3:sdd

磁盘默认会格式化为带Journal xfs文件系统的两个分区

7.同步配置文件到集群节点
ceph-deploy admin admin-node node1 node2 node3
实际上是把~/my-cluster/ceph.conf等配置文件同步到了各节点的/etc/ceph/目录下
[ceph@router my-cluster]$ ls /etc/ceph/
ceph.client.admin.keyring  ceph.conf  rbdmap  tmpSmajsW
sudo chmod +r /etc/ceph/ceph.client.admin.keyring
保证配置文件有正确的权限，很重要，可能同步后要再次添加ceph用户的读权限
8.检查ceph集群状态
[ceph@router my-cluster]$ ceph -s
    cluster 9a776b16-1e40-45de-a66b-afd4ba689fd5
     health HEALTH_OK
     monmap e1: 1 mons at {node1=192.168.8.101:6789/0}
            election epoch 3, quorum 0 node1
     osdmap e11: 3 osds: 2 up, 2 in
            flags sortbitwise,require_jewel_osds
      pgmap v21: 64 pgs, 1 pools, 0 bytes data, 0 objects
            70076 kB used, 20465 GB / 20466 GB avail
                  64 active+clean

默认的pg数太小，会提示HEALTH_WARN,可以手动调大

ceph osd pool set rbd pg_num 256

ceph osd pool set rbd pgp_num 256

二.扩展集群

1.增加osd
ceph-deploy osd create node1:/dev/vdb
2.增加metadata server
ceph-deploy mds create node1
3.增加monitor
http://docs.ceph.com/docs/master/rados/deployment/ceph-deploy-mon/
http://docs.ceph.com/docs/master/rados/configuration/network-config-ref/
i.添加网络配置
[ceph@router my-cluster]$ cat ceph.conf
[global]
fsid = 708da694-8089-4cc5-926a-6642a3e38100
mon_initial_members = node1
mon_host = 192.168.8.101
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

osd pool default size = 2
public network = 192.168.8.0/24
注意:往己存在的cluster里添加monitor时，需要修改配置文件ceph.conf在global章节中指定public network或者mon.章节中指定public addr，配置文件中写成代下划线的public_network =也是可以的。
ii.同步配置并添加monitor
ceph-deploy --overwrite-conf mon add node2
ceph-deploy --overwrite-conf mon add node3
ceph quorum_status --format json-pretty
[ceph@router my-cluster]$ ceph quorum_status --format json-pretty
{
    "election_epoch": 6,
    "quorum": [
        0,
        1,
        2
    ],
    "quorum_names": [
        "node1",
        "node2",
        "node3"
    ],
    "quorum_leader_name": "node1",
    "monmap": {
        "epoch": 3,
        "fsid": "708da694-8089-4cc5-926a-6642a3e38100",
        "modified": "2017-02-18 17:00:31.312958",
        "created": "2017-02-18 16:26:45.649393",
        "mons": [
            {
                "rank": 0,
                "name": "node1",
                "addr": "192.168.8.101:6789/0"
            },
            {
                "rank": 1,
                "name": "node2",
                "addr": "192.168.8.102:6789/0"
            },
            {
                "rank": 2,
                "name": "node3",
                "addr": "192.168.8.103:6789/0"
            }
        ]
    }
}

错误1:
[node3][INFO  ] Running command: sudo systemctl enable ceph.target
[node3][INFO  ] Running command: sudo systemctl enable ceph-mon@node3
[node3][INFO  ] Running command: sudo systemctl start ceph-mon@node3
[node3][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node3.asok mon_status
[node3][ERROR ] admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
[node3][WARNIN] node3 is not defined in `mon initial members`
[node3][WARNIN] monitor node3 does not exist in monmap
[node3][WARNIN] neither `public_addr` nor `public_network` keys are defined for monitors
[node3][WARNIN] monitors may not be able to form quorum
[node3][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.node3.asok mon_status
[node3][ERROR ] admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
[node3][WARNIN] monitor: mon.node3, might not be running yet
原因:
配置文件没有添加网络项

补充:手动添加monitor
ceph-deploy非常好用，官方也推荐使用。当然在手动配置时也可以参考ceph-deploy运行中提示的命令。
1.查看当前mon节点
ceph mon stat
2.添加mon.node2节点，首先在ceph.conf添加mon.node2配置
    [mon.node2] 
        host                       = node2
        mon addr              = 192.168.8.102:6789
3.获取mon的keyring
ceph auth get mon. -o /tmp/keyring
4.获取mon的map
ceph mon getmap -o /tmp/mapfile
5.初始化mon.node2节点的数据目录
 ceph-mon -i node2 --mkfs --monmap /tmp/mapfile --keyring /tmp/keyring 
6.添加mon.node2节点至ceph cluster
ceph mon add node2 192.168.8.102:6789
7.启动mon.node2
ceph-mon -i node2 --public-addr 192.168.8.102:6789


Ceph Object Storage
http://docs.ceph.com/docs/master/start/quick-rgw/
一.创建rgw
ceph-deploy rgw create node1
二.客户端存储文件---rados
1.创建存储池
rados mkpool data
rados lspools
2.上传文件
echo 'test ceph object store' >/tmp/foo.txt
rados put object_foo /tmp/foo.txt --pool=data
3.查看文件
rados -p data ls
rados get object_foo foo --pool=data
ceph osd map data object_foo
4.删除文件
rados rm object_foo --pool=data #删除存储池中的
rados rmpool data data --yes-i-really-really-mean-it #完全删除存储池，非常危险


Ceph Block Device
http://docs.ceph.com/docs/master/start/quick-rbd/
一.安装ceph客户端
ceph-deploy install client-node
ceph-deploy admin client-node
sudo chmod +r /etc/ceph/ceph.client.admin.keyring
实际上在客户端上yum -y install ceph ceph-radosgw运行即可
二.配置
1.创建image
rbd create data/myimage --size 102400 --object-size 8M
2.映射
sudo rbd map data/myimage --name client.admin

报错:
rbd: sysfs write failed
RBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable".
In some cases useful info is found in syslog - try "dmesg | tail" or so.
rbd: map failed: (6) No such device or address
原因:
http://docs.ceph.com/docs/master/man/8/rbd/

--image-format format-id

Specifies which object layout to use. The default is 2.

• format 1 - (deprecated) Use the original format for a new rbd image. This format is understood by all versions of librbd and the kernel rbd module, but does not support newer features like cloning.
• format 2 - Use the second rbd format, which is supported by librbd and kernel since version 3.11 (except for striping). This adds support for cloning and is more easily extensible to allow more features in the future.

--image-feature feature-name

Specifies which RBD format 2 feature should be enabled when creating an image. Multiple features can be enabled by repeating this option multiple times. The following features are supported:

• layering: layering support
• striping: striping v2 support
• exclusive-lock: exclusive locking support
• object-map: object map support (requires exclusive-lock)
• fast-diff: fast diff calculations (requires object-map)
• deep-flatten: snapshot flatten support
• journaling: journaled IO support (requires exclusive-lock)

解决:
因为内核版本过低，只启用支持的功能
rbd remove data/myimage
rbd create data/myimage --size 10G --image-format 2 --image-feature layering
sudo rbd map data/myimage
3.格式化
sudo mkfs.ext4 -m0 /dev/rbd/data/myimage
4.挂载
sudo mkdir /mnt/ceph-block-device
sudo mount /dev/rbd/data/myimage /mnt/ceph-block-device
sudo file -s /dev/rbd0
df -h /mnt/ceph-block-device/



Ceph FS
http://docs.ceph.com/docs/master/start/quick-cephfs/
一.创建文件系统
ceph osd pool create cephfs_data $(ceph osd pool get data all|grep -w 'pg_num:'|awk -F: '{print $2}')
ceph osd pool create cephfs_metadata $(ceph osd pool get data all|grep -w 'pg_num:'|awk -F: '{print $2}')
ceph fs new mycephfs cephfs_data cephfs_metadata
二.创建密钥文件
grep -w key /etc/ceph/ceph.client.admin.keyring|awk -F"= " '{print $2}' >/home/ceph/.admin.secret
三.挂载
方式一.kernel driver
sudo mkdir /mnt/mycephfs
sudo mount -t ceph 192.168.8.101:6789:/ /mnt/mycephfs/ -o name=admin,secretfile=/home/ceph/.admin.secret
方式二.FUSE
sudo mkdir /home/ceph/mycephfs
sudo ceph-fuse -m 192.168.8.101:6789 /home/ceph/mycephfs