003 ansible部署ceph集群

介绍：在上一次的deploy部署ceph，虽然出了结果，最后的结果并没有满足最初的目的，现在尝试使用ansible部署一遍，看是否会有问题

一、环境准备

ceph1充当部署节点，ceph2，ceph3，ceph4充当ceph集群节点

IP	主机名	节点	系统
172.25.250.10	ceph1	ceph-ansible	Red Hat release 7.4
172.25.250.11	ceph2	mon、mgr、osd	Red Hat release 7.4
172.25.250.12	ceph3	mon、mgr、osd	Red Hat release 7.4
172.25.250.10	ceph4	mon、mgr、osd	Red Hat release 7.4

1.1 ceph1配置hosts

172.25.250.10  ceph1     
172.25.250.11  ceph2     
172.25.250.12  ceph3   
172.25.250.13  ceph4

1.2 配置免密登录

[root@ceph1 ~]#  ssh-keygen

[root@ceph1 ~]# ssh-copy-id -i .ssh/id_rsa.pub ceph1

[root@ceph1 ~]# ssh-copy-id -i .ssh/id_rsa.pub ceph2

[root@ceph1 ~]# ssh-copy-id -i .ssh/id_rsa.pub ceph3

[root@ceph1 ~]# ssh-copy-id -i .ssh/id_rsa.pub ceph4

二，部署ceph集群

2.1部署节点安装ansible

注：再部署之前，一般需要配置yum源和时间服务器ntp配置

[root@ceph1 ~]# yum -y install ansible

[root@ceph1 ~]# cd /etc/ansible/

[root@ceph1 ansible]# ll

2.2 定义hosts

[root@ceph1 ansible]# vim hosts

[mons]
ceph2
ceph3
ceph4

[mgrs]
ceph2
ceph3
ceph4

[osds]
ceph2
ceph3
ceph4

[clients]
ceph1

2.3 安装ceph-ansible

[root@ceph1 ansible]#  yum -y install ceph-ansible

[root@ceph1 ansible]#  cd /usr/share/ceph-ansible/

[root@ceph1 ceph-ansible]# ls

[root@ceph1 ceph-ansible]#  ansible all -m ping

[root@ceph1 ceph-ansible]# cd /usr/share/ceph-ansible/group_vars/

[root@ceph1 group_vars]# cp mons.yml.sample mons.yml

[root@ceph1 group_vars]# cp mgrs.yml.sample mgrs.yml

[root@ceph1 group_vars]# cp osds.yml.sample osds.yml

[root@ceph1 group_vars]# cp clients.yml.sample clients.yml

[root@ceph1 group_vars]# cp all.yml.sample all.yml

[root@ceph1 group_vars]# vim all.yml

fetch_directory: ~/ceph-ansible-keys
ceph_repository_type: "cdn"
ceph_origin: repository
ceph_repository: rhcs
ceph_rhcs_version: 3
monitor_interface: eth0
journal_size: 5120
public_network: 172.25.250.0/24
cluster_network: "{{ public_network }}"

2.4 定义OSD

[root@ceph1 group_vars]# vim osds.yml

devices:
   - /dev/vdb
   - /dev/vdc
   - /dev/vdd
osd_scenario: collocated

 [root@ceph1 group_vars]# grep -Ev "^$|^s*#"  *.yml

2.5 定义ansible的入口文件

[root@ceph1 group_vars]# cd ..

[root@ceph1 ceph-ansible]# cp site.yml.sample site.yml

[root@ceph1 ceph-ansible]# vim site.yml

- hosts:
  - mons
#  - agents
  - osds
#  - mdss
#  - rgws
#  - nfss
#  - restapis
#  - rbdmirrors
  - clients
  - mgrs
#  - iscsi-gws

 2.6 安装

 [root@ceph1 ceph-ansible]# ansible-playbook  site.yml

安装如果有问题，会有报错，根据报错信息去修改yml文件，然后重新执行，出现下面的结果表示已经安装完成

PLAY RECAP ****************************************************************
ceph1                      : ok=42   changed=4    unreachable=0    failed=0   
ceph2                      : ok=213  changed=9    unreachable=0    failed=0   
ceph3                      : ok=196  changed=3    unreachable=0    failed=0   
ceph4                      : ok=190  changed=28   unreachable=0    failed=0   

2.7 检测验证

[root@ceph2 ~]# ceph -s

[root@ceph2 ~]# ceph osd tree

 

所有的文件都存放在/etc/ceph下

[root@ceph2 ~]# cd /etc/ceph/

[root@ceph2 ceph]# ll

ceph.conf是配置文件的名字，是因为没有指定集群的名字默认为ceph，配置文件的名字应为：cluster_name.conf

[root@ceph2 ceph]# cat ceph.conf

[global]
fsid = 35a91e48-8244-4e96-a7ee-980ab989d20d
mon initial members = ceph2,ceph3,ceph4
mon host = 172.25.250.11,172.25.250.12,172.25.250.13
public network = 172.25.250.0/24
cluster network = 172.25.250.0/24
[osd]
osd mkfs type = xfs
osd mkfs options xfs = -f -i size=2048
osd mount options xfs = noatime,largeio,inode64,swalloc
osd journal size = 5120

查看磁盘，有一个日志分区和数据分区

[root@ceph2 ceph]# fdisk -l

查看ceph进程

[root@ceph2 ceph]# ps aux|grep ceph

2.8 节点扩充

如果需要扩充节点，可以直接在部署节点的hosts文件，添加上主机名，执行部署，就会自己在原来的基础上继续扩充

如，把ceph1的三个磁盘扩充到osd，则配置如下：

[root@ceph1 ansible]# vim hosts

[mons]
ceph2
ceph3
ceph4

[mgrs]
ceph2
ceph3
ceph4

[osds]
ceph2
ceph3
ceph4
ceph1

[clients]
ceph1

再执行部署

[root@ceph1 ceph-ansible]# ansible-playbook  site.yml

如果部署错误，删除需要下面这个yml文件

[root@ceph1 ceph-ansible]#  vim infrastructure-playbooks/purge-cluster.yml

部分注释内容：
# This playbook purges Ceph
# It removes: packages, configuration files and ALL THE DATA
#
# Use it like this:
# ansible-playbook purge-cluster.yml
#     Prompts for confirmation to purge, defaults to no and
#     doesn't purge the cluster. yes purges the cluster.
#
# ansible-playbook -e ireallymeanit=yes|no purge-cluster.yml
#     Overrides the prompt using -e option. Can be used in
#     automation scripts to avoid interactive prompt.

 执行删除：

[root@ceph1 ceph-ansible]# ansible-playbook infrastructure-playbooks/purge-cluster.yml

---

 另一个环境安装

 一、环境准备，

每个虚拟机只有一块多余的磁盘做osd

IP	主机名	节点	系统	硬盘
172.25.254.130	ceph1	ceph-ansible	CentOS Linux release 7.4.1708 (Core)	/dev/sdb,dev/sdc,dev/sdd
172.25.254.131	ceph2	mon、mgr、osd	CentOS Linux release 7.4.1708 (Core)	/dev/sdb,dev/sdc,dev/sdd
172.25.254.132	ceph3	mon、mgr、osd	CentOS Linux release 7.4.1708 (Core)	/dev/sdb,dev/sdc,dev/sdd
172.25.254.133	ceph4	mon、mgr、osd	CentOS Linux release 7.4.1708 (Core)	/dev/sdb,dev/sdc,dev/sdd

1.1 配置hosts

172.25.254.130  ceph1     
172.25.254.131  ceph2     
172.25.254.132  ceph3   
172.25.254.133  ceph4

1.2 配置免密登录

root@ceph1 ~]#  ssh-keygen

[root@ceph1 ~]# ssh-copy-id -i .ssh/id_rsa.pub ceph1

[root@ceph1 ~]# ssh-copy-id -i .ssh/id_rsa.pub ceph2

[root@ceph1 ~]# ssh-copy-id -i .ssh/id_rsa.pub ceph3

[root@ceph1 ~]# ssh-copy-id -i .ssh/id_rsa.pub ceph4

1.3 内核进行升级

rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm
yum --disablerepo=* --enablerepo=elrepo-kernel repolist
yum --disablerepo=* --enablerepo=elrepo-kernel list kernel*
yum --disablerepo=* --enablerepo=elrepo-kernel install -y kernel-ml.x86_64
awk -F' '$1=="menuentry " {print $2}' /etc/grub2.cfg
grub2-set-default 0
reboot
rpm -qa|grep kernel|grep 3.10|xargs yum remove -y 
yum --disablerepo=* --enablerepo=elrepo-kernel install -y kernel-ml-tools.x86_64 
rpm -qa|grep kernel
systemctl stop firewalld
systemctl disable firewalld 

二、部署

2.1 下载包

[root@ceph1 ~]# wget -c https://github.com/ceph/ceph-ansible/archive/v3.1.7.tar.gz

[root@ceph1 ~]# tar xf v3.1.7.tar.gz

[root@ceph1 ~]# cd ceph-ansible-3.1.7

2.3 修改inventory，添加主机信息

[root@ceph1 ceph-ansible-3.1.7]# vim hosts

[mons]
ceph2
ceph3
ceph4

[osds]
ceph2
ceph3
ceph4

[mgrs]
ceph2
ceph3
ceph4

[mdss]
ceph2
ceph3
ceph4

[clients]
ceph1
ceph2
ceph3
ceph4

[root@ceph1 ceph-ansible-3.1.7]# ansible all -m ping -i hosts

2.4 修改all.yml写入如下内容

[root@ceph1 ceph-ansible-3.1.7]# cp group_vars/all.yml.sample group_vars/all.yml

[root@ceph1 ceph-ansible-3.1.7]# cp group_vars/osds.yml.sample group_vars/osds.yml

[root@ceph1 ceph-ansible-3.1.7]# cp site.yml.sample site.yml

[root@ceph1 ceph-ansible-3.1.7]# vim group_vars/all.yml

ceph_origin: repository
ceph_repository: community
ceph_mirror: http://mirrors.aliyun.com/ceph
ceph_stable_key: http://mirrors.aliyun.com/ceph/keys/release.asc
ceph_stable_release: luminous
ceph_stable_repo: "{{ ceph_mirror }}/rpm-{{ ceph_stable_release }}"

fsid: 54d55c64-d458-4208-9592-36ce881cbcb7 ##通过uuidgen生成
generate_fsid: false

cephx: true

public_network: 172.25.254.0/20
cluster_network: 172.25.254.0/20
monitor_interface: ens33

ceph_conf_overrides:
    global:
      rbd_default_features: 7
      auth cluster required: cephx
      auth service required: cephx
      auth client required: cephx
      osd journal size: 2048
      osd pool default size: 3
      osd pool default min size: 1
      mon_pg_warn_max_per_osd: 1024
      osd pool default pg num: 128
      osd pool default pgp num: 128
      max open files: 131072
      osd_deep_scrub_randomize_ratio: 0.01

    mgr:
      mgr modules: dashboard

    mon:
      mon_allow_pool_delete: true

    client:
      rbd_cache: true
      rbd_cache_size: 335544320
      rbd_cache_max_dirty: 134217728
      rbd_cache_max_dirty_age: 10

    osd:
      osd mkfs type: xfs
      ms_bind_port_max: 7100
      osd_client_message_size_cap: 2147483648
      osd_crush_update_on_start: true
      osd_deep_scrub_stride: 131072
      osd_disk_threads: 4
      osd_map_cache_bl_size: 128
      osd_max_object_name_len: 256
      osd_max_object_namespace_len: 64
      osd_max_write_size: 1024
      osd_op_threads: 8

      osd_recovery_op_priority: 1
      osd_recovery_max_active: 1
      osd_recovery_max_single_start: 1
      osd_recovery_max_chunk: 1048576
      osd_recovery_threads: 1
      osd_max_backfills: 4
      osd_scrub_begin_hour: 23
      osd_scrub_end_hour: 7

[root@ceph1 ceph-ansible-3.1.7]# vim group_vars/osds.yml

devices:
  - /dev/sdb
  - /dev/sdc
  - /dev/sdd
osd_scenario: collocated
osd_objectstore: bluestore

[root@ceph1 ceph-ansible-3.1.7]# vim site.yml

# Defines deployment design and assigns role to server groups
- hosts:
  - mons
#  - agents
  - osds
  - mdss
#  - rgws
#  - nfss
#  - rbdmirrors
  - clients
  - mgrs

2.5 执行部署 

[root@ceph1 ceph-ansible-3.1.7]# ansible-playbook site.yml -i hosts

-bash: ansible-playbook: command not found

[root@ceph1 ceph-ansible-3.1.7]# yum install ansible -y

[root@ceph1 ceph-ansible-3.1.7]# ansible-playbook site.yml -i hosts

成功安装

注：如果是不会过程出错，先清空集群，在进行部署

cp infrastructure-playbooks/purge-cluster.yml purge-cluster.yml # 必须copy到项目根目录下
ansible-playbook -i hosts purge-cluster.yml 

不升级内核，也可以安装成功，但是没有测试集群的其他性能！

参考链接：https://yq.aliyun.com/articles/624202