综合云平台

接下来是存储的部署, GlusterFS

主机: 192.168.1.201 / 192.168.1.202 / 192.168.1.203

系统: CentOS 7.4

CPU: 2 vCPU

Memory: 4 G

1. 安装 yum repo

# lvm
$ ansible glusterfs -m lvg -a "vg=gluster-vg pvs=/dev/sdb,/dev/sdc,/dev/sdd "
# 查看 vg 大小, 这里是 120G, 所以分为三个 lv ,每个给 35GB
$ vgs
  VG         #PV #LV #SN Attr   VSize    VFree   
  gluster-vg   3   0   0 wz--n- <119.99g <119.99g
# 划分 lv
$ ansible glusterfs -v -m lvol -a "lv=ovirt size=35G vg=gluster-vg"
$ ansible glusterfs -v -m lvol -a "lv=openshift size=35G vg=gluster-vg"
$ ansible glusterfs -v -m lvol -a "lv=openstack size=35G vg=gluster-vg"

2. 准备 bricks

# lvm
$ ansible glusterfs -m lvg -a "vg=gluster-vg pvs=/dev/sdb,/dev/sdc,/dev/sdd "
# 查看 vg 大小, 这里是 120G, 所以分为三个 lv ,每个给 35GB
$ vgs
  VG         #PV #LV #SN Attr   VSize    VFree   
  gluster-vg   3   0   0 wz--n- <119.99g <119.99g
# 划分 lv
$ ansible glusterfs -v -m lvol -a "lv=ovirt size=35G vg=gluster-vg"
$ ansible glusterfs -v -m lvol -a "lv=openshift size=35G vg=gluster-vg"
$ ansible glusterfs -v -m lvol -a "lv=openstack size=35G vg=gluster-vg"
# 查看一下结果
$ lvs
  LV        VG         Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  openshift gluster-vg -wi-a-----  35.00g                                                    
  openstack gluster-vg -wi-a-----  35.00g                                                    
  ovirt     gluster-vg -wi-a-----  35.00g
# 分区
$ ansible glusterfs -v -m filesystem -a "dev=/dev/gluster-vg/ovirt fstype=xfs opts='-i size=512'"
$ ansible glusterfs -v -m filesystem -a "dev=/dev/gluster-vg/openshift fstype=xfs opts='-i size=512'"
$ ansible glusterfs -v -m filesystem -a "dev=/dev/gluster-vg/openstack fstype=xfs opts='-i size=512'"
# 创建挂载目录
$ ansible glusterfs -v -m file -a "state=directory path=/bricks/ovirt"
$ ansible glusterfs -v -m file -a "state=directory path=/bricks/openshift"
$ ansible glusterfs -v -m file -a "state=directory path=/bricks/openstack"
# 修改 fstab
$ ansible glusterfs -v -m lineinfile -a "path=/etc/fstab line='/dev/gluster-vg/ovirt /bricks/ovirt xfs defaults 1 2'"
$ ansible glusterfs -v -m lineinfile -a "path=/etc/fstab line='/dev/gluster-vg/openshift /bricks/openshift xfs defaults 1 2'"
$ ansible glusterfs -v -m lineinfile -a "path=/etc/fstab line='/dev/gluster-vg/openstack /bricks/openstack xfs defaults 1 2'"
# 挂载
$ ansible glusterfs -v -m shell -a "mount -a"
# 检查
$ ansible glusterfs -v -m shell -a "df -hT"

3. 安装 GlusterFS 软件包

$ ansible glusterfs -m yum -a "name=glusterfs-server"
# 防火墙, 一般情况下加 glusterfs 的服务, 但是根据官网描述, 关掉会比较方便, 毕竟分布式存储是一个"只对内部提供访问"的服务
# "You can run gluster with iptables rules, but it's up to you to decide how you'll configure those rules. By default, glusterd will listen on tcp/24007 but opening that port isn't enough on the gluster nodes. Each time you add a brick, it will open a new port (that you'll be able to see with "gluster volume status")"
# 2017-12-6 更新: 还是添加 glusterfs 的防火墙服务吧, 不然老是会连接不上, 要开启防火墙然后再关掉才能连接
$ ansible glusterfs -m firewalld -a "service=glusterfs state=enabled permanent=true"
$ ansible glusterfs -m service -a "name=firewalld state=restarted enabled=yes"
# 启动 GlusterFS 服务吧
$ ansible glusterfs -m service -a "name=glusterd state=started enabled=yes"
# 检查服务状态
$ ansible glusterfs -a "systemctl status glusterd"

GlusterFS 就装好了, 接下来就可以创建 Volume 分享出去了

另外, 上面的过程可以直接跑个 playbook 来完成:

---
- hosts: glusterfs
  tasks:
    - name: pv / vg create
      lvg:
        vg: gluster-vg
        pvs: /dev/sdb,/dev/sdc,/dev/sdd

    - name: lv create
      lvol: lv={{ item }} size=35G vg=gluster-vg
      with_items:
        - ovirt
        - openshift
        - openstack

    - name: filesystem
      filesystem: dev=/dev/gluster-vg/{{ item }} fstype=xfs opts='-i size=512'
      with_items:
        - ovirt
        - openshift
        - openstack

    - name: create mount directories
      file: state=directory path=/bricks/{{ item }}
      with_items:
        - ovirt
        - openshift
        - openstack

    - name: modify fstab
      lineinfile: path=/etc/fstab line='/dev/gluster-vg/{{ item }} /bricks/{{ item }} xfs defaults 1 2'
      with_items:
        - ovirt
        - openshift
        - openstack

    - name: mount
      shell: mount -a
    - name: check mounts
      shell: df -hT

    - name: install glusterfs
      yum:
        name: glusterfs-server

    - name: disable firewalld
      service:
        name: firewalld
        state: stopped
        enabled: no
    - name: enable and start glusterfs
      service:
        name: glusterd
        state: started
        enabled: yes

    - name: check daemon status
      shell: systemctl status glusterd