linux运维、架构之路-K8s数据管理

一、Volume介绍

容器和Pod是短暂的，它们的生命周期可能很短，会被频繁的销毁和创建，存在容器中的数据会被清除，为了持久化保存容器的数据，k8s提供了Volume。
Volume的生命周期独立于容器，它是一个目录，Volume会被mount到Pod，Pod中的所有容器都可以访问这个Volume，和Docker Volume类似。
Volume支持的类型：emptyDir、hostPath、AWS Elastic Block Store、NFS、Ceph等。

1、emptyDir

       emptyDir是最基础的Volume类型，它对于容器来说是持久的，对于Pod刚不是，当Pod从节点删除时Volume中的数据也会被删除，如果只是容器被销毁而Pod还在，则Volume不受影响。

①创建示例应用

apiVersion: v1
kind: Pod
metadata:
  name: hello-world
spec:
  containers:
  - image: busybox
    name: hello
    volumeMounts:
    - mountPath: /hello_dir   #②hello容器将shared-volume mount到/hello_dir目录 
      name: shared-volume
    args:
    - /bin/sh       #③hello通过echo将数据写到文件hello里
    - -c
    - echo "hello world" >/hello_dir/hello; sleep 30000

  - image: busybox
    name: world    #④world容器将 shared-volume mount到 /world_dir目录
    volumeMounts:
    - mountPath: /world_dir
      name: shared-volume
    args:
    - /bin/sh     #⑤world通过cat从文件hello读取数据
    - -c
    - cat /world_dir/hello; sleep 30000

  volumes:
  - name: shared-volume   #①定义了一个emptyDir类型的Volume shared-volume
    emptyDir: {}

②查看验证

[root@k8s-node1 volume]# kubectl apply -f emptyDir.yaml 
pod "hello-world" created

[root@k8s-node1 volume]# kubectl get pod
NAME          READY     STATUS    RESTARTS   AGE
hello-world   2/2       Running   0          36s

[root@k8s-node1 volume]# kubectl logs hello-world world 
hello world

上述显示容器world成功读到了hello容器写入的数据，验证了两个容器共享emptyDir Volume,其效果相当于执行了docker run -v /hello_dir 和/docker run -v /world_dir

emptyDir是Host主机上创建的临时目录，其优点能够方便地为Pod中的容器提供共享存储，但不具有持久性，如果Pod不存在了，emptyDir也就没有了，适合Pod中容器需要临时共享存储空间的应用场景

2、HostPath

        hostPath Volume的作用是将Docker Host文件系统中已经存在的目录mount给Pod的容器，大部分应用不会使用此类型，因为这实际上增加上了Pod与节点的耦合，有些需要访问k8s内部数据，比如配置文件和二进制库这样的应用需要使用hostPath，例如k8s自身的应用kube-apiserver和kube-controller-manager。

3、外部Storage Provider

       如果k8s部署在AWS、GCE、Azure等公有云上，可以直接使用云盘作为Volume。因资源有限，此处无法演示。

二、PV & PVC

• PV是外部存储系统中的一块存储空间，由管理员创建，与Volume一样具有持久性，生命周期独立于Pod。
• PVC刚是对PV的申请，PVC通常由普通用户创建，需要为Pod分配存储资源时，用户可以创建一个PVC，指明存储资源的容量大小和访问权限等，k8s会查找并提供满足条件的PV。
• k8s支持多种类型的PV，比如AWS EBS、Ceph、NFS等

1、NFS PV

①在k8s-master节点搭建一个NFS服务，目录为/nfsdata

[root@k8s-node1 ~]# showmount -e
Export list for k8s-node1:
/nfsdata *

NFS服务具体部署请参照：https://www.cnblogs.com/yanxinjiang/p/7419469.html

②创建应用示例

apiVersion: v1
kind: PersistentVolume
metadata:
  name: mypv
spec:
  capacity:
    storage: 1Gi    #指定PV的容量为1GB
  accessModes:
    - ReadWriteOnce  #表示PV能为read-write模式mount到单个节点，ReadOnlyMany表示只读，mount到多个节点，ReadWriteMany表示读写可以mount到多节点
  persistentVolumeReclaimPolicy: Recycle  #PV的回收策略，清除PV中的数据，Retain表示管理员手工回收
  storageClassName: nfs    #指定PV的class为nfs，相当于为PV设置了一个分类，PVC可以指定class申请相应的PV
  nfs:
    path: /nfsdata/pv1     #指定PV在NFS服务器上对应的目录
    server: 192.168.56.11

[root@k8s-node1 nfs-pv]# kubectl apply -f nfs-pv1.yaml 
persistentvolume "mypv1" created 
[root@k8s-node1 nfs-pv]# kubectl get pv
NAME      CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM     STORAGECLASS   REASON    AGE   #STATUS为Available，表示mypv1就绪，可以被PVC申请
mypv1     1Gi        RWO            Recycle          Available             nfs                      4s       

③创建PVC

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: mypv
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: nfs

[root@k8s-node1 nfs-pv]# kubectl apply -f nfs-pvc1.yaml 
persistentvolumeclaim "mypvc1" created
[root@k8s-node1 nfs-pv]# kubectl get pvc
NAME      STATUS    VOLUME    CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mypvc1    Bound     mypv1     1Gi        RWO            nfs            13s
[root@k8s-node1 nfs-pv]# kubectl get pv
NAME      CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM            STORAGECLASS   REASON    AGE
mypv1     1Gi        RWO            Recycle          Bound     default/mypvc1   nfs                      22m

结果：mypvc1已经成功绑定到mypv1上面

④在Pod中使用存储

kind: Pod
apiVersion: v1
metadata:
  name: mypod1
spec:
  containers:
    - name: mypod1
      image: busybox
      args:
      - /bin/sh
      - -c
      - sleep 30000
      volumeMounts:
      - mountPath: "/mydata"
        name: mypv
  volumes:
    - name: mypv
      persistentVolumeClaim:
        claimName: mypv

⑤测试验证

[root@k8s-node1 nfs-pv]# kubectl apply -f pod1.yaml
[root@k8s-node1 nfs-pv]# kubectl get pod -o wide
NAME      READY     STATUS    RESTARTS   AGE       IP           NODE
mypod1    1/1       Running   0          11m       10.2.83.14   192.168.56.12
[root@k8s-node1 nfs-pv]# kubectl exec mypod1 touch /mydata/readme.md
[root@k8s-node1 nfs-pv]# ls /nfsdata/pv1/
readme.md

在Pod中创建的文件/mydata已经存储到了NFS服务器目录/nfsdata/pv1中。

⑥回收PV

• 当不需要使用PV时，可用删除PVC回收PV

[root@k8s-node1 nfs-pv]# kubectl delete pvc mypvc1 
persistentvolumeclaim "mypvc1" deleted

因PV的回收策略设置为Recycle，所以数据会被清除，如想保存数据，可以将策略设置为Retain

apiVersion: v1
kind: PersistentVolume
metadata:
  name: mypv1
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain  #Recycle
  storageClassName: nfs
  nfs:
    path: /nfsdata/pv1
    server: 192.168.56.11

[root@k8s-node1 nfs-pv]# kubectl apply -f nfs-pv1.yaml 
persistentvolume "mypv1" configured 
[root@k8s-node1 nfs-pv]# kubectl get pv
NAME      CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM            STORAGECLASS   REASON    AGE
mypv1     1Gi        RWO            Retain           Bound     default/mypvc1   nfs

⑦验证

[root@k8s-node1 nfs-pv]# kubectl apply -f nfs-pvc1.yaml 
[root@k8s-node1 nfs-pv]# kubectl exec mypod1 touch /mydata/hello.world
[root@k8s-node1 nfs-pv]# ls /nfsdata/pv1/
hello.world
[root@k8s-node1 nfs-pv]# kubectl delete pvc mypvc1 
persistentvolumeclaim "mypvc1" deleted
[root@k8s-node1 nfs-pv]# ls /nfsdata/pv1/
hello.world

2、PV动态供给

• 静态供给：我们提前创建PV，通过PVC申请PV并在Pod中使用
• 动态供给：如果没有满足PVC条件的PV，会动态创建PV，不需要提前创建PV，效率高，通过StorageClass实现。     

3、k8s中MySQL数据持久化存储

①创建PV和PVC

apiVersion: v1
kind: PersistentVolume
metadata:
  name: mysql-pv
spec:
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: nfs
  nfs:
    server: 192.168.56.11
    path: /data/k8s/mysql-pv

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: mysql-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
  storageClassName: nfs

查看pv,pvc状态

[root@k8s-node1 mysql]# kubectl apply -f mysql_pv.yaml 
persistentvolume "mysql-pv" created
persistentvolumeclaim "mysql-pvc" created
 
[root@k8s-node1 mysql]# kubectl get pv,pvc
NAME                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM               STORAGECLASS   REASON    AGE
persistentvolume/mysql-pv   2Gi        RWO            Retain           Bound     default/mysql-pvc   nfs                      4s

NAME                              STATUS    VOLUME     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/mysql-pvc   Bound     mysql-pv   2Gi        RWO            nfs            4s

②部署MySQL

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: mysql
spec:
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - name: mysql
        image: mysql:5.6
        imagePullPolicy: IfNotPresent
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: password
        ports:
        - name: mysql
          containerPort: 3306
        volumeMounts:
        - mountPath: /var/lib/mysql
          name: mysql-persistent-storage
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: mysql
spec:
  ports:
  - port: 3306
  selector:
    app: mysql

创建应用查看

[root@k8s-node1 mysql]# kubectl apply -f mysql.yaml 
deployment.apps "mysql" created
[root@k8s-node1 k8s]# kubectl get pod -o wide
NAME                     READY     STATUS    RESTARTS   AGE       IP          NODE
mysql-57c44d9d5f-5v7qc   1/1       Running   0          7m        10.2.8.25   192.168.56.13

③进入MySQL数据库创建测试数据

kubectl exec -it mysql-57c44d9d5f-5v7qc /bin/bash
root@mysql-57c44d9d5f-5v7qc:/# mysql -uroot -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or g.
Your MySQL connection id is 2
Server version: 5.6.46 MySQL Community Server (GPL)

Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or 'h' for help. Type 'c' to clear the current input statement.

mysql> 
mysql> 
mysql> 
mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |

mysql> create database test;
Query OK, 1 row affected (0.01 sec)

mysql> use test;
Database changed

mysql> create table my_id(id int(4));
Query OK, 0 rows affected (0.03 sec)

mysql> insert my_id values(111);
Query OK, 1 row affected (0.01 sec)


mysql> select * from my_id;
+------+
| id   |
+------+
|  111 |
+------+
1 row in set (0.00 sec)

④模拟k8s-node3节点故障

 此时node3节点变为不可用

[root@k8s-node1 mysql]# kubectl get node
NAME            STATUS     ROLES     AGE       VERSION
192.168.56.12   Ready      <none>    6d        v1.10.3
192.168.56.13   NotReady   <none>    6d        v1.10.3

⑤等待一段时间后，k8s会将MySQL迁移至node2节点

[root@k8s-node1 mysql]# kubectl get pod -o wide
NAME                     READY     STATUS    RESTARTS   AGE       IP           NODE
mysql-57c44d9d5f-5v7qc   1/1       Unknown   0          47m       10.2.8.25    192.168.56.13
mysql-57c44d9d5f-6gvh5   1/1       Running   0          18m       10.2.93.74   192.168.56.12
[root@k8s-node1 mysql]# kubectl exec -it mysql-57c44d9d5f-6gvh5 /bin/bash

⑥再次登录MySQL验证数据的一致性

root@mysql-57c44d9d5f-6gvh5:/# mysql -uroot -ppassword
Warning: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or g.
Your MySQL connection id is 1
Server version: 5.6.46 MySQL Community Server (GPL)

Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or 'h' for help. Type 'c' to clear the current input statement.
mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| test               |
+--------------------+
4 rows in set (0.02 sec)
mysql> use test;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
mysql> show tables;
+----------------+
| Tables_in_test |
+----------------+
| my_id          |
+----------------+
1 row in set (0.00 sec)

mysql> select * from my_id;
+------+
| id   |
+------+
|  111 |
+------+
1 row in set (0.00 sec)

结果：MySQL服务恢复，数据无丢失