K8S中使用gfs当存储

K8S中使用gfs当存储

一开始是参考《再也不踩坑的Kubernetes实战指南》中的以容器方式去安装GFS集群的，但是在安装Heketi的过程中卡了将近几十分钟也没有好，就换成来EndPoints安装。

环境信息

三台机器，操作系统CentOS 7.8或者阿里云服务器。此处我都安装测试过是可行的

k8s-master 192.168.1.180
k8s-node1 192.168.1.151
k8s-node2 192.168.1.152

1、搭建glusterFS集群

执行

[root@k8s-master ]# yum install centos-release-gluster
[root@k8s-master ]# yum install -y glusterfs glusterfs-server glusterfs-fuse glusterfs-rdma

配置 GlusterFS 集群

[root@k8s-master ]# systemctl start glusterd.service
[root@k8s-master ]# systemctl enable glusterd.service

主节点执行

[root@k8s-master ]# gluster peer probe k8s-master
[root@k8s-master ]#  gluster peer probe k8s-node1
[root@k8s-master ]# gluster peer probe k8s-node2

创建数据目录

[root@k8s-master ]# mkdir -p /opt/gfs_data

创建复制卷

[root@k8s-master ]# gluster volume create k8s-volume replica 3 k8s-master:/opt/gfs_data k8s-node1:/opt/gfs_data k8s-node2:/opt/gfs_data force

启动卷

[root@k8s-master ]# gluster volume start k8s-volume

查询卷状态

[root@k8s-master ~]# gluster volume status
Status of volume: k8s-volume
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick k8s-master:/opt/gfs_data              49152     0          Y       13987
Brick k8s-node1:/opt/gfs_data               49152     0          Y       13216
Brick k8s-node2:/opt/gfs_data               49152     0          Y       13932
Self-heal Daemon on localhost               N/A       N/A        Y       14014
Self-heal Daemon on k8s-node1               N/A       N/A        Y       13233
Self-heal Daemon on k8s-node2               N/A       N/A        Y       13949

Task Status of Volume k8s-volume
------------------------------------------------------------------------------
There are no active volume tasks
[root@k8s-master ~]#

[root@k8s-master ~]# gluster volume info

Volume Name: k8s-volume
Type: Replicate
Volume ID: fce4a311-6afb-428b-90dd-e0e9c335de3e
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: k8s-master:/opt/gfs_data
Brick2: k8s-node1:/opt/gfs_data
Brick3: k8s-node2:/opt/gfs_data
Options Reconfigured:
cluster.granular-entry-heal: on
storage.fips-mode-rchecksum: on
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
[root@k8s-master ~]#

验证glusterFS集群可用

选择其中一台主机执行

yum install -y glusterfs glusterfs-fuse
mount -t glusterfs 192.168.1.180:/k8s-volume /mnt
cd /mnt
touch test

2、使用glusterfs(以下均在k8s master节点执行)

创建glusterfs的endpoints：kubectl apply -f glusterfs-cluster.yaml

apiVersion: v1
kind: Endpoints
metadata:
  name: glusterfs-cluster
  namespace: default
subsets:
- addresses:
  - ip: 192.168.1.180
  - ip: 192.168.1.151
  - ip: 192.168.1.152
  ports:
  - port: 30003
    protocol: TCP

创建pv使用glusterfs：kubectl apply -f glusterfs-pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: glusterfs-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteMany
  glusterfs:
    endpoints: glusterfs-cluster
    path: k8s-volume
    readOnly: false

创建pvc声明：kubectl apply -f glusterfs-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: glusterfs-pvc
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 2Gi

创建应用使用pvc：kubectl apply -f nginx_deployment.yaml

[root@k8s-master ]# cat nginx_deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      name: nginx
  template:
    metadata:
      labels:
        name: nginx
    spec:
      containers:
        - name: nginx
          image: nginx
          ports:
            - containerPort: 80
          volumeMounts:
            - name: storage001
              mountPath: "/usr/share/nginx/html"
      volumes:
      - name: storage001
        persistentVolumeClaim:
          claimName: glusterfs-pvc

执行下面命令会发现有test文件

kubectl exec nginx-deployment-6594f875fb-8dnl8 ls /usr/share/nginx/html/