交付redis 到k8s 集群

1. 创建nfs 共享存储

# 在所有主机上：
yum install nfs-utils  rpcbind  -y 

在10.4.7.200 上部署 nfs server

cat[root@hdss7-200 nginxcurl]# cat /etc/exports
/data/nfs-volume 10.4.7.0/24(rw,no_root_squash)
/data/volumes 10.4.7.0/24(rw,no_root_squash)
/data/nfs-volume/redis/pv1 *(rw,all_squash)
/data/nfs-volume/redis/pv2 *(rw,all_squash)
/data/nfs-volume/redis/pv3 *(rw,all_squash)
/data/nfs-volume/redis/pv4 *(rw,all_squash)
/data/nfs-volume/redis/pv5 *(rw,all_squash)
/data/nfs-volume/redis/pv6 *(rw,all_squash)

chmod 777 /data/nfs-volume/redis/pv*   # 不加这个会出现 mount.nfs: access denied by server while mounting的报错。

systemctl start rpcbind
systemctl start nfs

　　

2. 创建PV

在 7.200 上创建6个pv，后面准备申请6个pod 

cat pv.yaml 
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv1
spec:
  capacity:
    storage: 200M
  accessModes:
    - ReadWriteMany
  nfs:
    server: 10.4.7.200
    path: "/data/nfs-volume/redis/pv1"

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv2
spec:
  capacity:
    storage: 200M
  accessModes:
    - ReadWriteMany
  nfs:
    server: 10.4.7.200
    path: "/data/nfs-volume/redis/pv2"

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv3
spec:
  capacity:
    storage: 200M
  accessModes:
    - ReadWriteMany
  nfs:
    server: 10.4.7.200
    path: "/data/nfs-volume/redis/pv3"

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv4
spec:
  capacity:
    storage: 200M
  accessModes:
    - ReadWriteMany
  nfs:
    server: 10.4.7.200
    path: "/data/nfs-volume/redis/pv4"

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv5
spec:
  capacity:
    storage: 200M
  accessModes:
    - ReadWriteMany
  nfs:
    server: 10.4.7.200
    path: "/data/nfs-volume/redis/pv5"

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv6
spec:
  capacity:
    storage: 200M
  accessModes:
    - ReadWriteMany
  nfs:
    server: 10.4.7.200
    path: "/data/nfs-volume/redis/pv6"

kubectl create -f pv.yaml  

　　

3.创建Configmap

cat redis.conf 
appendonly yes
cluster-enabled yes
cluster-config-file /var/lib/redis/nodes.conf
cluster-node-timeout 5000
dir /var/lib/redis
port 6379

kubectl create configmap redis-conf --from-file=redis.conf  #用redis.conf 创建 redis-conf configmap

kubectl describe cm redis-conf  
Name:         redis-conf
Namespace:    default
Labels:       <none>
Annotations:  <none>

Data
====
redis.conf:
----
appendonly yes
cluster-enabled yes
cluster-config-file /var/lib/redis/nodes.conf
cluster-node-timeout 5000
dir /var/lib/redis
port 6379


Events:  <none>

　　

4.创建Redis 集群节点

有状态服务用statefull 控制器

cat stateful.yaml 
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: redis-app
spec:
  serviceName: "redis-service"
  replicas: 6
  template:
    metadata:
      labels:
        app: redis
        appCluster: redis-cluster
    spec:
      terminationGracePeriodSeconds: 20
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - redis
              topologyKey: kubernetes.io/hostname
      containers:
      - name: redis
        image: "harbor.od.com/public/redis:6.0"
        command:
          - "redis-server"
        args:
          - "/etc/redis/redis.conf"
          - "--protected-mode"
          - "no"
        resources:
          requests:
            cpu: "100m"
            memory: "100Mi"
        ports:
            - name: redis
              containerPort: 6379
              protocol: "TCP"
            - name: cluster
              containerPort: 16379
              protocol: "TCP"
        volumeMounts:
          - name: "redis-conf"
            mountPath: "/etc/redis"
          - name: "redis-data"   
            mountPath: "/var/lib/redis"
      volumes:
      - name: "redis-conf"
        configMap:
          name: "redis-conf"
          items:
            - key: "redis.conf"
              path: "redis.conf"
  #这个就是pvc，关联之前创建的pv。            
  volumeClaimTemplates:
  - metadata:
      name: redis-data
    spec:
      accessModes: [ "ReadWriteMany" ]
      resources:
        requests:
          storage: 200M

　　volumemounts 指定两个name 一个是 configmap 的 ，一个是pvc ，名字叫 redis-data, pvc  用来关联pv，这里用来存放redis 的数据的。

查看 redis 的pod 

用 kubectl get pod -o wide --watch 可以创建的过程，会发现依次从0 到 5 逐个创建的，可以发现6个pod 都创建好了，但是现在pod 之间只能通过ip 访问，statefull  设计宗旨是要通过域名访问的，所以下面要

创建service，

 kubectl get pod -o wide --show-labels
NAME                                READY   STATUS              RESTARTS   AGE     IP            NODE                NOMINATED NODE   READINESS GATES   LABELS

pod-vol-pvc                         1/1     Running             0          10d     172.7.21.4    hdss7-21.host.com   <none>           <none>            <none>
redis-app-0                         1/1     Running             0          28h     172.7.21.8    hdss7-21.host.com   <none>           <none>            app=redis,appCluster=redis-cluster,controller-revision-hash=redis-app-58dbd4d55c,statefulset.kubernetes.io/pod-name=redis-app-0
redis-app-1                         1/1     Running             0          28h     172.7.22.12   hdss7-22.host.com   <none>           <none>            app=redis,appCluster=redis-cluster,controller-revision-hash=redis-app-58dbd4d55c,statefulset.kubernetes.io/pod-name=redis-app-1
redis-app-2                         1/1     Running             0          28h     172.7.21.9    hdss7-21.host.com   <none>           <none>            app=redis,appCluster=redis-cluster,controller-revision-hash=redis-app-58dbd4d55c,statefulset.kubernetes.io/pod-name=redis-app-2
redis-app-3                         1/1     Running             0          28h     172.7.22.15   hdss7-22.host.com   <none>           <none>            app=redis,appCluster=redis-cluster,controller-revision-hash=redis-app-58dbd4d55c,statefulset.kubernetes.io/pod-name=redis-app-3
redis-app-4                         1/1     Running             0          28h     172.7.21.10   hdss7-21.host.com   <none>           <none>            app=redis,appCluster=redis-cluster,controller-revision-hash=redis-app-58dbd4d55c,statefulset.kubernetes.io/pod-name=redis-app-4
redis-app-5                         1/1     Running             0          28h     172.7.22.16   hdss7-22.host.com   <none>           <none>            app=redis,appCluster=redis-cluster,controller-revision-hash=redis-app-58dbd4d55c,statefulset.kubernetes.io/pod-name=redis-app-5
web-0                               0/1     ContainerCreating   0          10d     <none>        hdss7-21.host.com   <none>           <none>            app=nginx,controller-revision-hash=web-7f49674c6,statefulset.kubernetes.io/pod-name=web-0

　　查看pv pvc，发现pvc 和pv 已经绑定了，pod 也关联上了pv。

[root@hdss7-21 ~]# kubectl get pv,pvc -o wide 
NAME                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS        CLAIM                            STORAGECLASS   REASON   AGE   VOLUMEMODE
persistentvolume/nfs-pv1   200M       RWX            Retain           Bound         default/redis-data-redis-app-4                           44h   Filesystem
persistentvolume/nfs-pv3   200M       RWX            Retain           Bound         default/redis-data-redis-app-3                           44h   Filesystem
persistentvolume/nfs-pv4   200M       RWX            Retain           Bound         default/redis-data-redis-app-5                           44h   Filesystem
persistentvolume/nfs-pv5   200M       RWX            Retain           Bound         default/redis-data-redis-app-1                           44h   Filesystem
persistentvolume/nfs-pv6   200M       RWX            Retain           Available                                                              44h   Filesystem
persistentvolume/nfs-pv2   200M       RWX            Retain           Bound         default/redis-data-redis-app-2                           44h   Filesystem
persistentvolume/pv001     1Gi        RWO,RWX        Retain           Bound         default/redis-data-redis-app-0                           10d   Filesystem
persistentvolume/pv002     2Gi        RWO            Retain           Terminating   default/www-web-0                                        10d   Filesystem
persistentvolume/pv004     4Gi        RWO,RWX        Retain           Bound         default/mypvc                                            10d   Filesystem

NAME                                           STATUS   VOLUME    CAPACITY   ACCESS MODES   STORAGECLASS   AGE   VOLUMEMODE
persistentvolumeclaim/mypvc                    Bound    pv004     4Gi        RWO,RWX                       10d   Filesystem
persistentvolumeclaim/redis-data-redis-app-0   Bound    pv001     1Gi        RWO,RWX                       46h   Filesystem
persistentvolumeclaim/redis-data-redis-app-1   Bound    nfs-pv5   200M       RWX                           28h   Filesystem
persistentvolumeclaim/redis-data-redis-app-2   Bound    nfs-vp2   200M       RWX                           46h   Filesystem
persistentvolumeclaim/redis-data-redis-app-3   Bound    nfs-pv3   200M       RWX                           44h   Filesystem
persistentvolumeclaim/redis-data-redis-app-4   Bound    nfs-pv1   200M       RWX                           44h   Filesystem
persistentvolumeclaim/redis-data-redis-app-5   Bound    nfs-pv4   200M       RWX                           44h   Filesystem

　　

　　

5. 创建redis 的service

用来实现pod 之间域名访问， statefull  控制器的service  的 clusterIP: None ，这个要注意！service 通过标签  

app: redis
appCluster: redis-cluster   找到pod 

[root@hdss7-200 redis]# cat service.yaml 
apiVersion: v1
kind: Service
metadata:
  name: redis-service
  labels:
    app: redis
spec:
  ports:
  - name: redis-port
    port: 6379
  clusterIP: None
  selector:
    app: redis
    appCluster: redis-cluster

　　

查看 service

名字 redis-service 是刚创建的，这时候我们就可以通过  pod 名字. service 名字  访问pod 了，例如 redis-app-1.redis-service  这样的。

kubectl get svc -o wide --show-labels
NAME                   TYPE        CLUSTER-IP        EXTERNAL-IP   PORT(S)    AGE   SELECTOR                             LABELS
kubernetes             ClusterIP   192.168.0.1       <none>        443/TCP    71d   <none>                               component=apiserver,provider=kubernetes
nginx                  ClusterIP   None              <none>        80/TCP     11d   app=nginx                            app=nginx
redis-service          ClusterIP   None              <none>        6379/TCP   28h   app=redis,appCluster=redis-cluster   app=redis

　　新版busybox 的坑

由于redis 镜像里没有nslookup 命令，这里起一个busybox 镜像来验证。这里有个坑，最新的busybox 镜像会解析失败，当时我还以为我的coredns 有问题了，后来发现是 busybox 镜像问题，用 1.28.4 就没有问题了。

参考： 

nslookup redis-app-1.redis-service  # 格式是 pod 名字.service名字.namespace名字，如果是default 可以不用谢namespace 名字。

kubectl run -i --tty --image busybox:1.28.4 dns-test --restart=Never --rm /bin/sh
/ # nslookup redis-app-1.redis-service
Server:    192.168.0.2
Address 1: 192.168.0.2 coredns.kube-system.svc.cluster.local

Name:      redis-app-1.redis-service
Address 1: 172.7.22.12 172-7-22-12.lightspeed.bcvloh.sbcglobal.net
/ # 
/ # nslookup redis-app-0.redis-service
Server:    192.168.0.2
Address 1: 192.168.0.2 coredns.kube-system.svc.cluster.local

Name:      redis-app-0.redis-service
Address 1: 172.7.21.8 172-7-21-8.lightspeed.bcvloh.sbcglobal.net
/ # cat /etc/resolv.conf 
nameserver 192.168.0.2
search default.svc.cluster.local svc.cluster.local cluster.local host.com
options ndots:5

　　发现都可以解析了。

6.初始化Redis集群

先安装 redis-trib.rb 命令的ruby 依赖环境，在10.4.7.21 上操作。  redis-trib.rb  在redis包 的二进制文件里面。

wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
gem sources --add http://mirrors.aliyun.com/rubygems/ --remove https://rubygems.org/
yum install ruby rubygems  -y　 && gem install redis -v 3.3.5       //安装版本3.3.5，针对redis3.0.6,redis-trib.rb reshard 操作可以成功
#gem list      //查看已经安装的软件列表

　　创建redis-cluster 集群----- 有三个主从节点，

--replicas 1 表示集群中的每个主节点创建一个从节点，前面三个是主节点，后面三个是slave 节点。

redis-trib.rb create --replicas 1 172.7.21.8:6379  172.7.21.9:6379 172.7.21.10:6379   172.7.22.14:6379 172.7.22.12:6379 172.7.22.15:6379 

　　创建后随便登录一台查看，可以看到集群中有6个节点了。剩下的操作就和普通redis-cluster 操作一样了。

redis-cli -h 172.7.22.15 -c 
172.7.22.15:6379> CLUSTER NODES
3b69afaaa38f700c057b3c0310765dc0082d71dd 172.7.22.12:6379@16379 slave cdb68a9c2c68657d344ce81a10529d0c760ad411 0 1593896053552 3 connected
cdb68a9c2c68657d344ce81a10529d0c760ad411 172.7.21.9:6379@16379 master - 0 1593896054564 3 connected 8192-16383
dec1a6f7cf8668ea987fe06b5ca05df5f054d198 172.7.22.15:6379@16379 myself,slave 4ba7e926b9e6c7edc6461fd101db6cb6a37ad0ed 0 1593896054000 6 connected
ad855dc2ba9f2c3c409c05216068845e9a9f435e 172.7.21.10:6379@16379 slave 20c5a6057994439b84ac92ec3bbd8d76b1c80557 0 1593896053556 5 connected
20c5a6057994439b84ac92ec3bbd8d76b1c80557 172.7.21.8:6379@16379 master - 0 1593896054869 5 connected 286-8191
4ba7e926b9e6c7edc6461fd101db6cb6a37ad0ed 172.7.22.14:6379@16379 master - 0 1593896053852 7 connected 0-285
172.7.22.15:6379> CLUSTER INFO
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:7
cluster_my_epoch:7
cluster_stats_messages_ping_sent:9415
cluster_stats_messages_pong_sent:9545
cluster_stats_messages_sent:18960
cluster_stats_messages_ping_received:9545
cluster_stats_messages_pong_received:9415
cluster_stats_messages_update_received:2
cluster_stats_messages_received:18962

　　

 7. 创建提供给集群内部的service

这里是为了外部访问redis 用的。

cat ../redis/redissvc.yaml 
apiVersion: v1
kind: Service
metadata:
  name: redis-access-service
  labels:
    app: redis
spec:
  ports:
  - name: redis-port
    protocol: "TCP"
    port: 6379
    targetPort: 6379
  selector:
    app: redis
    appCluster: redis-cluster

　　

[root@hdss7-21 ~]# kubectl get svc -o wide --show-labels
NAME                   TYPE        CLUSTER-IP        EXTERNAL-IP   PORT(S)    AGE    SELECTOR                             LABELS
kubernetes             ClusterIP   192.168.0.1       <none>        443/TCP    75d    <none>                               component=apiserver,provider=kubernetes
nginx                  ClusterIP   None              <none>        80/TCP     14d    app=nginx                            app=nginx
redis-access-service   ClusterIP   192.168.169.232   <none>        6379/TCP   5d3h   app=redis,appCluster=redis-cluster   app=redis
redis-service          ClusterIP   None              <none>        6379/TCP   3d8h   app=redis,appCluster=redis-cluster   app=redis

　　访问  192.168.169.232 ,  发现和连接redis  pod 看到是一样的效果！

redis-cli -c -h 192.168.169.232
192.168.169.232:6379> CLUSTER NODES
3b69afaaa38f700c057b3c0310765dc0082d71dd 172.7.22.12:6379@16379 slave cdb68a9c2c68657d344ce81a10529d0c760ad411 0 1593895154063 3 connected
cdb68a9c2c68657d344ce81a10529d0c760ad411 172.7.21.9:6379@16379 master - 0 1593895154594 3 connected 8192-16383
dec1a6f7cf8668ea987fe06b5ca05df5f054d198 172.7.22.15:6379@16379 myself,slave 4ba7e926b9e6c7edc6461fd101db6cb6a37ad0ed 0 1593895154000 6 connected
ad855dc2ba9f2c3c409c05216068845e9a9f435e 172.7.21.10:6379@16379 slave 20c5a6057994439b84ac92ec3bbd8d76b1c80557 0 1593895155601 5 connected
20c5a6057994439b84ac92ec3bbd8d76b1c80557 172.7.21.8:6379@16379 master - 0 1593895155089 5 connected 286-8191
4ba7e926b9e6c7edc6461fd101db6cb6a37ad0ed 172.7.22.14:6379@16379 master - 0 1593895154000 7 connected 0-285
192.168.169.232:6379> CLUSTER INFO
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:7
cluster_my_epoch:7
cluster_stats_messages_ping_sent:8984
cluster_stats_messages_pong_sent:9113
cluster_stats_messages_sent:18097
cluster_stats_messages_ping_received:9113
cluster_stats_messages_pong_received:8984
cluster_stats_messages_update_received:2
cluster_stats_messages_received:18099