Kubernetes之（十四）StatefulSet控制器
简介
StatefulSet 作为 Controller 为 Pod 提供唯⼀的标识。 它可以保证部署和 scale 的顺序。
StatefulSet是为了解决有状态服务的问题（对应Deployments和ReplicaSets是为⽆状态服务⽽设计） ， 其应⽤场景包括：
- 稳定的持久化存储， 即Pod重新调度后还是能访问到相同的持久化数据， 基于PVC来实现
- 稳定的⽹络标志， 即Pod重新调度后其PodName和HostName不变， 基于Headless Service（即没有Cluster IP的Service） 来实现
- 有序部署， 有序扩展， 即Pod是有顺序的， 在部署或者扩展的时候要依据定义的顺序依次依次进⾏（即从0到N-1，在下⼀个Pod运⾏之前所有之前的Pod必须都是Running和Ready状态） ， 基于init containers来实现有序收缩， 有序删除（即从N-1到0）
从上⾯的应⽤场景可以发现， StatefulSet由以下⼏个部分组成：
- ⽤于定义⽹络标志（DNS domain） 的Headless Service
- ⽤于创建PersistentVolumes的volumeClaimTemplates
- 定义具体应⽤的StatefulSet
StatefulSet中每个Pod的DNS格式为 statefulSetName-{0..N-1}.serviceName.namespace.svc.cluster.local ， 其中
- serviceName 为Headless Service的名字
- 0..N-1 为Pod所在的序号， 从0开始到N-1
- statefulSetName 为StatefulSet的名字
- namespace 为服务所在的namespace， Headless Servic和StatefulSet必须在相同的namespace
- .cluster.local 为Cluster Domain
部署和Scale保证:
- 对于有N个副本的StatefulSet，Pod将按照{0..N-1}的顺序被创建和部署。
- 当删除Pod的时候，将按照逆序来终结，从{N-1..0}
- 对Pod执⾏scale操作之前，它所有的前任必须处于Running和Ready状态。
- 在终⽌Pod前，它所有的继任者必须处于完全关闭状态。
Headless Service：
在deployment中，每一个pod是没有名称，是随机字符串，是无序的。而statefulset中是要求有序的，每一个pod的名称必须是固定的。当节点挂了，重建之后的标识符是不变的，每一个节点的节点名称是不能改变的。pod名称是作为pod识别的唯一标识符，必须保证其标识符的稳定并且唯一。
为了实现标识符的稳定，这时候就需要一个headless service 解析直达到pod，还需要给pod配置一个唯一的名称。
volumeClainTemplate：
大部分有状态副本集都会用到持久存储，比如分布式系统来说，由于数据是不一样的，每个节点都需要自己专用的存储节点。而在deployment中pod模板中创建的存储卷是一个共享的存储卷，多个pod使用同一个存储卷，而statefulset定义中的每一个pod都不能使用同一个存储卷，由此基于pod模板创建pod是不适应的，这就需要引入volumeClainTemplate，当在使用statefulset创建pod时，会自动生成一个PVC，从而请求绑定一个PV，从而有自己专用的存储卷。Pod名称、PVC和PV关系图如下：
StatefulSet使用
使用之前配置的NFS服务器，PV
#nfs服务器
[root@nfs ~]# showmount -e
Export list for nfs:
/data/volumes/v5 10.0.0.0/24
/data/volumes/v4 10.0.0.0/24
/data/volumes/v3 10.0.0.0/24
/data/volumes/v2 10.0.0.0/24
/data/volumes/v1 10.0.0.0/24
#PV
[root@master volume]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pv001 1Gi RWO,RWX Retain Available 3s
pv002 2Gi RWO Retain Available 3s
pv003 2Gi RWO,RWX Retain Available 3s
pv004 4Gi RWO,RWX Retain Available 3s
pv005 4Gi RWO,RWX Retain Available 3s 23h
创建statefulSet
[root@master statefulset]# vim statefulset-v1.yaml
#定义Headless服务
apiVersion: v1
kind: Service
metadata:
name: myapp
labels:
app: myapp
spec:
ports:
- port: 80
name: web
clusterIP: None
selector:
app: myapp-pod
---
#配置statefulset
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: myapp
spec:
serviceName: myapp-svc
replicas: 2
selector:
matchLabels:
app: myapp-pod
template:
metadata:
labels:
app: myapp-pod
spec:
containers:
- name: myapp
image: ikubernetes/myapp:v1
ports:
- name: web
containerPort: 80
volumeMounts:
- name: myappdata
mountPath: /usr/share/nginx/html/
volumeClaimTemplates:
- metadata:
name: myappdata
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 1.5Gi
[root@master statefulset]# kubectl apply -f statefulset-v1.yaml
service/myapp created
statefulset.apps/myapp created
[root@master statefulset]# kubectl get pods
NAME READY STATUS RESTARTS AGE
myapp-0 1/1 Running 0 5s
myapp-1 1/1 Running 0 4s
myapp-2 1/1 Running 0 3s
#此时Pod名称不是乱码是从0~N-1
[root@master statefulset]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 7d
myapp ClusterIP None <none> 80/TCP 31s
[root@master statefulset]# kubectl get sts
NAME READY AGE
myapp 3/3 48s
查看pvc pv资源
[root@master statefulset]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
myappdata-myapp-0 Bound pv002 2Gi RWO 3m56s
myappdata-myapp-1 Bound pv003 2Gi RWO,RWX 2m49s
myappdata-myapp-2 Bound pv004 4Gi RWO,RWX 88s
[root@master statefulset]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pv001 1Gi RWO,RWX Retain Available 22m
pv002 2Gi RWO Retain Bound default/myappdata-myapp-0 22m
pv003 2Gi RWO,RWX Retain Bound default/myappdata-myapp-1 22m
pv004 4Gi RWO,RWX Retain Bound default/myappdata-myapp-2 22m
pv005 4Gi RWO,RWX Retain Available 22m
查看Pod使用的存储卷
[root@master statefulset]# kubectl describe pods myapp-2
......
Volumes:
myappdata:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: myappdata-myapp-2
......
就算删除Pod后重建的Podi依然绑定在myappdata-myapp-2这个PVC上
滚动更新
[root@master statefulset]# vim statefulset-v1.yaml
image: ikubernetes/myapp:v3
[root@master statefulset]# kubectl apply -f statefulset-v1.yaml
service/myapp unchanged
statefulset.apps/myapp configured
[root@master ~]# kubectl get pods -w
NAME READY STATUS RESTARTS AGE
myapp-0 1/1 Running 0 9m18s
myapp-1 1/1 Running 0 9m17s
myapp-2 1/1 Running 0 9m16s
myapp-2 1/1 Terminating 0 9m22s
myapp-2 0/1 Terminating 0 9m23s
myapp-2 0/1 Terminating 0 9m26s
myapp-2 0/1 Terminating 0 9m26s
myapp-2 0/1 Pending 0 0s
myapp-2 0/1 Pending 0 0s
myapp-2 0/1 ContainerCreating 0 0s
myapp-2 1/1 Running 0 2s
myapp-1 1/1 Terminating 0 9m29s
myapp-1 0/1 Terminating 0 9m30s
myapp-1 0/1 Terminating 0 9m31s
myapp-1 0/1 Terminating 0 9m31s
myapp-1 0/1 Pending 0 0s
myapp-1 0/1 Pending 0 0s
myapp-1 0/1 ContainerCreating 0 0s
myapp-1 1/1 Running 0 1s
myapp-0 1/1 Terminating 0 9m33s
myapp-0 0/1 Terminating 0 9m34s
myapp-0 0/1 Terminating 0 9m35s
myapp-0 0/1 Terminating 0 9m35s
myapp-0 0/1 Pending 0 0s
myapp-0 0/1 Pending 0 0s
myapp-0 0/1 ContainerCreating 0 0s
myapp-0 1/1 Running 0 1s
在创建的每一个Pod中，每一个pod自己的名称都是可以被解析的，如下：
[root@master statefulset]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
myapp-0 1/1 Running 0 39s 10.244.2.66 node02 <none> <none>
myapp-1 1/1 Running 0 42s 10.244.1.56 node01 <none> <none>
myapp-2 1/1 Running 0 46s 10.244.2.65 node02 <none> <none>
[root@master statefulset]# kubectl exec -it myapp-1 -- /bin/sh
/ # nslookup myapp-1.myapp-svc.default.svc.cluster.local
nslookup: can't resolve '(null)': Name does not resolve
Name: myapp-1.myapp-svc.default.svc.cluster.local
Address 1: 10.244.1.56 myapp-1.myapp-svc.default.svc.cluster.local
从上面的解析，我们可以看到在容器当中可以通过对Pod的名称进行解析到ip。其解析的域名格式如下：
pod_name.service_name.ns_name.svc.cluster.local
eg: myapp-0.myapp.default.svc.cluster.local
扩展伸缩
#扩展副本到4个
[root@master statefulset]# kubectl scale sts myapp --replicas=4
statefulset.apps/myapp scaled
#查看结果
[root@master ~]# kubectl get pods -w
NAME READY STATUS RESTARTS AGE
myapp-0 1/1 Running 0 6m17s
myapp-1 1/1 Running 0 6m20s
myapp-2 1/1 Running 0 6m24s
myapp-3 0/1 Pending 0 0s
myapp-3 0/1 Pending 0 0s
myapp-3 0/1 Pending 0 0s
myapp-3 0/1 ContainerCreating 0 0s
myapp-3 1/1 Running 0 2s
#查看PV绑定
[root@master statefulset]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pv001 1Gi RWO,RWX Retain Available 38m
pv002 2Gi RWO Retain Bound default/myappdata-myapp-0 38m
pv003 2Gi RWO,RWX Retain Bound default/myappdata-myapp-1 38m
pv004 4Gi RWO,RWX Retain Bound default/myappdata-myapp-2 38m
pv005 4Gi RWO,RWX Retain Bound default/myappdata-myapp-3
#打补丁方式缩容
[root@master statefulset]# kubectl patch sts myapp -p '{"spec":{"replicas":2}}'
statefulset.apps/myapp patched
[root@master ~]# kubectl get pods -w
NAME READY STATUS RESTARTS AGE
myapp-0 1/1 Running 0 9m26s
myapp-1 1/1 Running 0 9m29s
myapp-2 1/1 Running 0 13s
myapp-3 1/1 Running 0 11s
myapp-3 1/1 Terminating 0 16s
myapp-3 0/1 Terminating 0 18s
myapp-3 0/1 Terminating 0 22s
myapp-3 0/1 Terminating 0 22s
myapp-2 1/1 Terminating 0 24s
myapp-2 0/1 Terminating 0 24s
myapp-2 0/1 Terminating 0 30s
myapp-2 0/1 Terminating 0 30s
更新策略和版本升级
修改更新策略，以partition方式进行更新，更新值为2，只有myapp编号大于等于2的才会进行更新。类似于金丝雀部署方式
[root@master statefulset]# kubectl patch sts myapp -p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":2}}}}'
statefulset.apps/myapp patched
[root@master statefulset]# kubectl describe sts myapp
Name: myapp
Namespace: default
CreationTimestamp: Wed, 03 Apr 2019 16:53:24 +0800
Selector: app=myapp-pod
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"apps/v1","kind":"StatefulSet","metadata":{"annotations":{},"name":"myapp","namespace":"default"},"spec":{"replicas":3,"sele...
版本升级，将image的版本升级为v5，升级后对比myapp-2和myapp-1的image版本是不同的。这样就实现了金丝雀发布的效果。
[root@master statefulset]# kubectl get sts -o wide
NAME READY AGE CONTAINERS IMAGES
myapp 4/4 23m myapp ikubernetes/myapp:v5
[root@master statefulset]# kubectl get pods myapp-2 -o yaml |grep image
- image: ikubernetes/myapp:v5
imagePullPolicy: IfNotPresent
image: ikubernetes/myapp:v5
imageID: docker-pullable://ikubernetes/myapp@sha256:85d1005d172aa8b97d7f1aa67519132cd450f59d01a607d4b4eaf5bcf402ce52
[root@master statefulset]# kubectl get pods myapp-0 -o yaml |grep image
- image: ikubernetes/myapp:v3
imagePullPolicy: IfNotPresent
image: ikubernetes/myapp:v3
imageID: docker-pullable://ikubernetes/myapp@sha256:b8d74db2515d3c1391c78c5768272b9344428035ef6d72158fd9f6c4239b2c69
将剩余的Pod也更新版本，只需要将更新策略的partition值改为0即可，如下：
[root@master statefulset]# kubectl patch sts myapp -p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":0}}}}'
statefulset.apps/myapp patched
[root@master statefulset]# kubectl get pods myapp-0 -o yaml |grep image - image: ikubernetes/myapp:v5
imagePullPolicy: IfNotPresent
image: ikubernetes/myapp:v5
imageID: docker-pullable://ikubernetes/myapp@sha256:85d1005d172aa8b97d7f1aa67519132cd450f59d01a607d4b4eaf5bcf402ce52
生产中还是不建议把重要应用使用statefulset，如mysql redis 等 。
参考资料
https://www.cnblogs.com/linuxk
马永亮. Kubernetes进阶实战 (云计算与虚拟化技术丛书)
Kubernetes-handbook-jimmysong-20181218