背景介绍
当我们用K8s管理MySql容器集群时，由于MySql的集群机制是每个实例有自己的存储空间，通过网络进行数据同步，并且数据库应用属于有状态的应用，故采用Deployment方式会很麻烦。
K8s从1.5版本开始使用StatefulSet来编排有状态应用：
-
Pod会被顺序部署和顺序终结：StatefulSet中的各个 Pod会被顺序地创建出来，每个Pod都有一个唯一的ID，在创建后续 Pod 之前，首先要等前面的 Pod 运行成功并进入到就绪状态。删除会销毁StatefulSet 中的每个 Pod，并且按照创建顺序的反序来执行，只有在成功终结后面一个之后，才会继续下一个删除操作。
-
Pod具有唯一网络名称：Pod具有唯一的名称，而且在重启后会保持不变。通过Headless服务，基于主机名，每个 Pod 都有独立的网络地址，这个网域由一个Headless 服务所控制。这样每个Pod会保持稳定的唯一的域名，使得集群就不会将重新创建出的Pod作为新成员。
-
Pod能有稳定的持久存储：StatefulSet中的每个Pod可以有其自己独立的PersistentVolumeClaim对象。即使Pod被重新调度到其它节点上以后，原有的持久磁盘也会被挂载到该Pod。
-
Pod能被通过Headless服务访问到：客户端可以通过服务的域名连接到任意Pod。
使用StatefulSet部署MySql
- configmap.yaml
ConfigMap 提供 my.cnf 覆盖，独立控制 MySQL 主服务器和从服务器的配置。主服务器能够将复制日志提供给从服务器，从服务器拒绝任何不是通过复制进行的写操作。
1 apiVersion: v1 2 kind: ConfigMap 3 metadata: 4 name: mysql 5 labels: 6 app: mysql 7 data: 8 master.cnf: | 9 # Apply this config only on the master. 10 [mysqld] 11 log-bin # 主mysql激活二进制日志 12 #设置时区和字符集 13 default-time-zone='+8:00' 14 character-set-client-handshake=FALSE 15 character-set-server=utf8mb4 16 collation-server=utf8mb4_unicode_ci 17 init_connect='SET NAMES utf8mb4 COLLATE utf8mb4_unicode_ci' 18 slave.cnf: | 19 # Apply this config only on slaves. 20 [mysqld] 21 super-read-only # 从mysql上面设置为只读 22 #设置时区和字符集 23 default-time-zone='+8:00' 24 character-set-client-handshake=FALSE 25 character-set-server=utf8mb4 26 collation-server=utf8mb4_unicode_ci 27 init_connect='SET NAMES utf8mb4 COLLATE utf8mb4_unicode_ci'
[root@localhost k8s_mysql]# kubectl apply -f configmap.yaml configmap/mysql created [root@localhost k8s_mysql]# kubectl get cm NAME DATA AGE mysql 2 107s
- services.yaml
Headless Service 给 StatefulSet 控制器为集合中每个 Pod 创建的 DNS 条目提供了一个宿主。因为 Headless Service 名为 mysql，所以可以通过在同一 Kubernetes 集群和 namespace 中的任何其他 Pod 内解析 <pod-name>.mysql 来访问 Pod。
请注意，只有读取查询才能使用负载平衡的客户端 Service。因为只有一个 MySQL 主服务器，所以客户端应直接连接到 MySQL 主服务器 Pod (通过其在 Headless Service 中的 DNS 条目)以执行写入操作。
1 # Headless service for stable DNS entries of StatefulSet members. 2 apiVersion: v1 3 kind: Service 4 metadata: 5 name: mysql 6 labels: 7 app: mysql 8 spec: 9 ports: 10 - name: mysql 11 port: 3306 12 clusterIP: None 13 selector: 14 app: mysql 15 --- 16 # Client service for connecting to any MySQL instance for reads. 17 # For writes, you must instead connect to the master: mysql-0.mysql. 18 apiVersion: v1 19 kind: Service 20 metadata: 21 name: mysql-read 22 labels: 23 app: mysql 24 spec: 25 ports: 26 - name: mysql 27 port: 3306 28 selector: 29 app: mysql
[root@localhost k8s_mysql]# kubectl apply -f services.yaml service/mysql created service/mysql-read created [root@localhost k8s_mysql]# kubectl get service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 7d17h mysql ClusterIP None <none> 3306/TCP 20s mysql-read ClusterIP 10.43.118.155 <none> 3306/TCP 20s
- statefulset.yaml
1 apiVersion: apps/v1 2 kind: StatefulSet 3 metadata: 4 name: mysql 5 spec: 6 selector: 7 matchLabels: 8 app: mysql 9 serviceName: mysql 10 replicas: 3 11 template: 12 metadata: 13 labels: 14 app: mysql 15 spec: 16 initContainers: 17 - name: init-mysql 18 image: mysql:5.7 19 command: 20 - bash 21 - "-c" 22 - | 23 set -ex 24 # Generate mysql server-id from pod ordinal index. 25 [[ `hostname` =~ -([0-9]+)$ ]] || exit 1 26 ordinal=${BASH_REMATCH[1]} 27 echo [mysqld] > /mnt/conf.d/server-id.cnf 28 # Add an offset to avoid reserved server-id=0 value. 29 echo server-id=$((100 + $ordinal)) >> /mnt/conf.d/server-id.cnf 30 # Copy appropriate conf.d files from config-map to emptyDir. 31 if [[ $ordinal -eq 0 ]]; then 32 cp /mnt/config-map/master.cnf /mnt/conf.d/ 33 else 34 cp /mnt/config-map/slave.cnf /mnt/conf.d/ 35 fi 36 volumeMounts: 37 - name: conf 38 mountPath: /mnt/conf.d 39 - name: config-map 40 mountPath: /mnt/config-map 41 - name: clone-mysql 42 image: gcr.io/google-samples/xtrabackup:1.0 43 command: 44 - bash 45 - "-c" 46 - | 47 set -ex 48 # Skip the clone if data already exists. 49 [[ -d /var/lib/mysql/mysql ]] && exit 0 50 # Skip the clone on master (ordinal index 0). 51 [[ `hostname` =~ -([0-9]+)$ ]] || exit 1 52 ordinal=${BASH_REMATCH[1]} 53 [[ $ordinal -eq 0 ]] && exit 0 54 # Clone data from previous peer. 55 ncat --recv-only mysql-$(($ordinal-1)).mysql 3307 | xbstream -x -C /var/lib/mysql 56 # Prepare the backup. 57 xtrabackup --prepare --target-dir=/var/lib/mysql 58 volumeMounts: 59 - name: data 60 mountPath: /var/lib/mysql 61 subPath: mysql 62 - name: conf 63 mountPath: /etc/mysql/conf.d 64 containers: 65 - name: mysql 66 image: mysql:5.7 67 env: 68 - name: MYSQL_ALLOW_EMPTY_PASSWORD 69 value: "1" 70 ports: 71 - name: mysql 72 containerPort: 3306 73 volumeMounts: 74 - name: data 75 mountPath: /var/lib/mysql 76 subPath: mysql 77 - name: conf 78 mountPath: /etc/mysql/conf.d 79 resources: 80 requests: 81 cpu: 500m 82 memory: 1Gi 83 livenessProbe: 84 exec: 85 command: ["mysqladmin", "ping"] 86 initialDelaySeconds: 30 87 periodSeconds: 10 88 timeoutSeconds: 5 89 readinessProbe: 90 exec: 91 # Check we can execute queries over TCP (skip-networking is off). 92 command: ["mysql", "-h", "127.0.0.1", "-e", "SELECT 1"] 93 initialDelaySeconds: 5 94 periodSeconds: 2 95 timeoutSeconds: 1 96 - name: xtrabackup 97 image: gcr.io/google-samples/xtrabackup:1.0 98 ports: 99 - name: xtrabackup 100 containerPort: 3307 101 command: 102 - bash 103 - "-c" 104 - | 105 set -ex 106 cd /var/lib/mysql 107 108 # Determine binlog position of cloned data, if any. 109 if [[ -f xtrabackup_slave_info && "x$(<xtrabackup_slave_info)" != "x" ]]; then 110 # XtraBackup already generated a partial "CHANGE MASTER TO" query 111 # because we're cloning from an existing slave. (Need to remove the tailing semicolon!) 112 cat xtrabackup_slave_info | sed -E 's/;$//g' > change_master_to.sql.in 113 # Ignore xtrabackup_binlog_info in this case (it's useless). 114 rm -f xtrabackup_slave_info xtrabackup_binlog_info 115 elif [[ -f xtrabackup_binlog_info ]]; then 116 # We're cloning directly from master. Parse binlog position. 117 [[ `cat xtrabackup_binlog_info` =~ ^(.*?)[[:space:]]+(.*?)$ ]] || exit 1 118 rm -f xtrabackup_binlog_info xtrabackup_slave_info 119 echo "CHANGE MASTER TO MASTER_LOG_FILE='${BASH_REMATCH[1]}', 120 MASTER_LOG_POS=${BASH_REMATCH[2]}" > change_master_to.sql.in 121 fi 122 123 # Check if we need to complete a clone by starting replication. 124 if [[ -f change_master_to.sql.in ]]; then 125 echo "Waiting for mysqld to be ready (accepting connections)" 126 until mysql -h 127.0.0.1 -e "SELECT 1"; do sleep 1; done 127 128 echo "Initializing replication from clone position" 129 mysql -h 127.0.0.1 130 -e "$(<change_master_to.sql.in), 131 MASTER_HOST='mysql-0.mysql', 132 MASTER_USER='root', 133 MASTER_PASSWORD='', 134 MASTER_CONNECT_RETRY=10; 135 START SLAVE;" || exit 1 136 # In case of container restart, attempt this at-most-once. 137 mv change_master_to.sql.in change_master_to.sql.orig 138 fi 139 140 # Start a server to send backups when requested by peers. 141 exec ncat --listen --keep-open --send-only --max-conns=1 3307 -c 142 "xtrabackup --backup --slave-info --stream=xbstream --host=127.0.0.1 --user=root" 143 volumeMounts: 144 - name: data 145 mountPath: /var/lib/mysql 146 subPath: mysql 147 - name: conf 148 mountPath: /etc/mysql/conf.d 149 resources: 150 requests: 151 cpu: 100m 152 memory: 100Mi 153 volumes: 154 - name: conf 155 emptyDir: {} 156 - name: config-map 157 configMap: 158 name: mysql 159 volumeClaimTemplates: 160 - metadata: 161 name: data 162 spec: 163 accessModes: ["ReadWriteOnce"] 164 resources: 165 requests: 166 storage: 10Gi
1 [root@localhost k8s_mysql]# kubectl apply -f satefulset.yml 2 [root@localhost k8s_mysql]# kubectl get po -o wide 3 NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES 4 mysql-0 2/2 Running 5 13s 10.42.1.73 localhost.localdomain <none> <none> 5 mysql-1 2/2 Running 0 25s 10.42.1.74 localhost.localdomain <none> <none> 6 mysql-2 1/2 CrashLoopBackOff 8 18m 10.42.1.75 localhost.localdomain <none> <none>
ps：由于我只有一台性能不咋地的虚拟机，因此导致其中一个节点不能运行起来，不过能在此说明问题就行。
- 验证
1 root@mysql-0:/# mysql -h mysql-0.mysql 2 Welcome to the MySQL monitor. Commands end with ; or g. 3 Your MySQL connection id is 1360 4 Server version: 8.0.18 MySQL Community Server - GPL 5 6 Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved. 7 8 Oracle is a registered trademark of Oracle Corporation and/or its 9 affiliates. Other names may be trademarks of their respective 10 owners. 11 12 Type 'help;' or 'h' for help. Type 'c' to clear the current input statement. 13 14 mysql> show databases; 15 +--------------------+ 16 | Database | 17 +--------------------+ 18 | information_schema | 19 | mysql | 20 | performance_schema | 21 | sys | 22 +--------------------+ 23 4 rows in set (0.00 sec) 24 25 mysql> create database addtest; 26 Query OK, 1 row affected (0.01 sec) 27 28 mysql> show databases; 29 +--------------------+ 30 | Database | 31 +--------------------+ 32 | addtest | 33 | information_schema | 34 | mysql | 35 | performance_schema | 36 | sys | 37 +--------------------+ 38 5 rows in set (0.00 sec) 39 40 mysql> exit 41 Bye 42 root@mysql-0:/# mysql -h mysql-1.mysql 43 Welcome to the MySQL monitor. Commands end with ; or g. 44 Your MySQL connection id is 1217 45 Server version: 8.0.18 MySQL Community Server - GPL 46 47 Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved. 48 49 Oracle is a registered trademark of Oracle Corporation and/or its 50 affiliates. Other names may be trademarks of their respective 51 owners. 52 53 Type 'help;' or 'h' for help. Type 'c' to clear the current input statement. 54 55 mysql> show databases; 56 +--------------------+ 57 | Database | 58 +--------------------+ 59 | addtest | 60 | information_schema | 61 | mysql | 62 | performance_schema | 63 | sys | 64 +--------------------+ 65 5 rows in set (0.00 sec) 66 67 mysql> exit