zoukankan      html  css  js  c++  java
  • Kerbernetes的StatefulSet与Operator

                Kerbernetes的StatefulSet与Operator

                                         作者:尹正杰

    版权声明:原创作品,谢绝转载!否则将追究法律责任。 

    一.StatefulSet概述

    1>.StatefulSet与Operator的关系

      管理一组有状态pod服务的部署(deployment)和扩展(scaling),并保证这些pod的顺序和唯一性。
        与部署一样,StatefulSet管理基于相同容器规范的pod。
        与部署不同,statefulSet为每个pod维护一个粘性标识。这些pod是从同一个规范创建的,但不能互换:每个pod都有一个peristent标识符,可以在任何重新调度中维护。
    
      事实上,使用StatefulSet可以很轻松的帮咱们完成部署,但是扩展的话需要我们自己编写封装扩容集群或者缩容集群脚本代码的程序,这个程序我们统称为Operator。而且我们编写的脚本还需要进行大量测试,否则出现任意一个bug,将会在线上带来不堪设想的后果。
     
      于是,有一家公司叫CoreOS的公司对StatefulSet进行二次封装开源了Operator SDK(Software Development Kit)接口,用户可以借助于Operator SDK的API比较轻松定制出Operator控制器。
    
      这意味着第三方的程序员再去开发云原生应用不是针对Kubernetes云原生API,而是针对CoreOS公司的Operator SDK。换句话说,Kubernetes的大本营被人开了一道后门,从后方侵入快速拿下一半的云原生领土,而这个领土的所属方叫CoreOS。而CoreOS已经是Redhat旗下的产品,而Redhat又是IBM旗下的产品。
    
      在国外,微软(Microsoft),亚马逊(Amazon)已经是云计算的领头羊了,而IBM作为一个老牌的基础设施公司虽说也有自己的云计算产品但影响力却远远不如前两者公司。因为大公司通常是很难接收革新的,它们有自己成熟的产品也有充分的市场份额,有一个其它产品时竞争对手时,它们通常把竞争对手视为眼中钉肉中刺,而不是拥抱和开发它们,IBM就错失了很多良机。IBM之所以愿意买RedHat,原因之一可能是因为红帽公司旗下的CoreOS公司。
    
      IBM公司地位发展也给咱们运维人员一定的启发,这就好比你已经轻车熟路的使用K8S,突然间又出现了一个颠覆性的产品的苗头之后,你一定不是第一个愿意接收它的人,会有一种抵触心理,此时若很多人抱着开放的心胸去学习了该产品,恍然之间觉得自己的技能要被淘汰了不得不去学习的时候,那些先行者早已在这个行业称为巨头了。
    
      目前大部分应用也有自己的Operator开源项目,比较火热的应用在Github有对应的开源地址。博主推荐阅读:https://github.com/operator-framework/awesome-operators。

    2>.StatefulSet的应用场景

      statefulset对于需要以下一项或多项的应用程序很有价值:
        (1)有唯一固定的网络标识符;
        (2)固定唯一的持久存储;
        (3)有序优雅的部署和扩展;
        (4)有序右眼的删除和终止;
        (5)有序自动的执行滚动更新;
    
      StatefulSet的限制条件(Limitations):
        (1)各Pod用到的存储卷必须使用由StorageClass动态供给给或由管理事先创建好的PV;
        (2)删除StatefulSet或缩减其规模导致Pod被删除时不应该自动删除(这个需要我们定义pvc指定其回收策略为"Retain")其存储卷以确保数据安全;
        (3)StatefulSet控制器依赖于一个事先存在的Headless Serivce对象实现Pod对象的持久,唯一的标识符配置;次Headless Service需要由用户手动配置。

    3>.StatefulSet的Pod 标识符(Identity)

      有序索引(Ordinal Index):
        对于一个有N个副本的StatefulSet,StatefulSet中的每个Pod将被分配一个整数序数,从0到N-1,在这个集合上是唯一的。
    
      稳定网络ID(Stabel Network ID):
        StatefulSet中的每个Pod都从StatefulSet的名称和Pod的序号派生其主机名。
        StatefulSet可以使用无头服务(Headless Service)来控制其Pods的域。此服务管理的域的任务格式为:"$(service name).$(namespace).svc.cluster.local",其中“cluster.local”是群集域。
        创建每个Pod时,它将获得一个匹配的DNS子域,其格式为:"$(podname).$(governing service domain)",其中governing service由StatefulSet上的serviceName字段定义。

    4>.Pod管理策略(Management Policies)

      在Kubernetes 1.7及更高版本中,StatefulSet允许您放宽其排序保证,同时通过其.spec.podManagementPolicay字段保持其唯一性和标识保证。
    
      有序(OrderedReady)Pod管理(Management):
        OrderedReady pod management是statefulset的默认设置。换句话说,就是当一个Pod创建成功后再回去创建下一个Pod,即有序创建Pod。
      并行的(Parallel)Pod管理(Management):     并行Pod管理告诉StatefulSet控制器并行启动或终止所有Pod,不要等到Pod运行并准备就绪或完全终止后再启动或终止另一个Pod。

    5>.更新策略(Update Strategies)

      在Kubernetes 1.7及更高版本中,StatefulSet的.spec.updateStrategy字段允许您配置和禁用StatefulSet中Pods的容器、标签、资源请求/限制和注释的自动滚动更新。
    
      ON Delete:
        由用户手动删除现有的Pod对象从而触发其更新过程。
    
      Rolling Updates:
        默认策略,它通过自动更新机制完成更新过程,启动更新过程时,它自动删除每个Pod对象并以新配置进行重建,更新顺序同删除StatefulSet时的逆向操作机制,一次删除并更新一个Pod对象。

      可以通过指定.spec.update strategy.RollingUpdate.partition对RollingUpdate更新策略进行分区。

      如果指定了分区,则当StatefulSet的.pec.template被更新时,序号大于或等于分区的所有pod都将被更新。

      序号小于分区的所有pod都将不会更新,即使它们被删除,也将在以前的版本中重新创建。

    二.StatefulSet实战案例

    1>.构建NFS网络文件系统

    [root@test110.yinzhengjie.org.cn ~]# yum -y install nfs-utils
    Loaded plugins: fastestmirror
    Loading mirror speeds from cached hostfile
     * base: mirror.bit.edu.cn
     * extras: mirrors.tuna.tsinghua.edu.cn
     * updates: mirrors.tuna.tsinghua.edu.cn
    base                                                                                                                                                                                                                                                   | 3.6 kB  00:00:00     
    docker-ce-stable                                                                                                                                                                                                                                       | 3.5 kB  00:00:00     
    extras                                                                                                                                                                                                                                                 | 2.9 kB  00:00:00     
    updates                                                                                                                                                                                                                                                | 2.9 kB  00:00:00     
    updates/7/x86_64/primary_db                                                                                                                                                                                                                            | 6.7 MB  00:00:01     
    Resolving Dependencies
    --> Running transaction check
    ---> Package nfs-utils.x86_64 1:1.3.0-0.65.el7 will be installed
    --> Processing Dependency: libtirpc >= 0.2.4-0.7 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: gssproxy >= 0.7.0-3 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: rpcbind for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: quota for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: libnfsidmap for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: keyutils for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: libtirpc.so.1()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: libnfsidmap.so.0()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Running transaction check
    ---> Package gssproxy.x86_64 0:0.7.0-26.el7 will be installed
    --> Processing Dependency: libini_config >= 1.3.1-31 for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libverto-module-base for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libref_array.so.1(REF_ARRAY_0.1.1)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libini_config.so.3(INI_CONFIG_1.2.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libini_config.so.3(INI_CONFIG_1.1.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libref_array.so.1()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libini_config.so.3()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libcollection.so.2()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libbasicobjects.so.0()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    ---> Package keyutils.x86_64 0:1.5.8-3.el7 will be installed
    ---> Package libnfsidmap.x86_64 0:0.25-19.el7 will be installed
    ---> Package libtirpc.x86_64 0:0.2.4-0.16.el7 will be installed
    ---> Package quota.x86_64 1:4.01-19.el7 will be installed
    --> Processing Dependency: quota-nls = 1:4.01-19.el7 for package: 1:quota-4.01-19.el7.x86_64
    --> Processing Dependency: tcp_wrappers for package: 1:quota-4.01-19.el7.x86_64
    ---> Package rpcbind.x86_64 0:0.2.0-48.el7 will be installed
    --> Running transaction check
    ---> Package libbasicobjects.x86_64 0:0.1.1-32.el7 will be installed
    ---> Package libcollection.x86_64 0:0.7.0-32.el7 will be installed
    ---> Package libini_config.x86_64 0:1.3.1-32.el7 will be installed
    --> Processing Dependency: libpath_utils.so.1(PATH_UTILS_0.2.1)(64bit) for package: libini_config-1.3.1-32.el7.x86_64
    --> Processing Dependency: libpath_utils.so.1()(64bit) for package: libini_config-1.3.1-32.el7.x86_64
    ---> Package libref_array.x86_64 0:0.1.5-32.el7 will be installed
    ---> Package libverto-libevent.x86_64 0:0.2.5-4.el7 will be installed
    ---> Package quota-nls.noarch 1:4.01-19.el7 will be installed
    ---> Package tcp_wrappers.x86_64 0:7.6-77.el7 will be installed
    --> Running transaction check
    ---> Package libpath_utils.x86_64 0:0.2.1-32.el7 will be installed
    --> Finished Dependency Resolution
    
    Dependencies Resolved
    
    ==============================================================================================================================================================================================================================================================================
     Package                                                                 Arch                                                         Version                                                                Repository                                                  Size
    ==============================================================================================================================================================================================================================================================================
    Installing:
     nfs-utils                                                               x86_64                                                       1:1.3.0-0.65.el7                                                       base                                                       412 k
    Installing for dependencies:
     gssproxy                                                                x86_64                                                       0.7.0-26.el7                                                           base                                                       110 k
     keyutils                                                                x86_64                                                       1.5.8-3.el7                                                            base                                                        54 k
     libbasicobjects                                                         x86_64                                                       0.1.1-32.el7                                                           base                                                        26 k
     libcollection                                                           x86_64                                                       0.7.0-32.el7                                                           base                                                        42 k
     libini_config                                                           x86_64                                                       1.3.1-32.el7                                                           base                                                        64 k
     libnfsidmap                                                             x86_64                                                       0.25-19.el7                                                            base                                                        50 k
     libpath_utils                                                           x86_64                                                       0.2.1-32.el7                                                           base                                                        28 k
     libref_array                                                            x86_64                                                       0.1.5-32.el7                                                           base                                                        27 k
     libtirpc                                                                x86_64                                                       0.2.4-0.16.el7                                                         base                                                        89 k
     libverto-libevent                                                       x86_64                                                       0.2.5-4.el7                                                            base                                                       8.9 k
     quota                                                                   x86_64                                                       1:4.01-19.el7                                                          base                                                       179 k
     quota-nls                                                               noarch                                                       1:4.01-19.el7                                                          base                                                        90 k
     rpcbind                                                                 x86_64                                                       0.2.0-48.el7                                                           base                                                        60 k
     tcp_wrappers                                                            x86_64                                                       7.6-77.el7                                                             base                                                        78 k
    
    Transaction Summary
    ==============================================================================================================================================================================================================================================================================
    Install  1 Package (+14 Dependent packages)
    
    Total download size: 1.3 M
    Installed size: 3.6 M
    Downloading packages:
    (1/15): keyutils-1.5.8-3.el7.x86_64.rpm                                                                                                                                                                                                                |  54 kB  00:00:00     
    (2/15): libcollection-0.7.0-32.el7.x86_64.rpm                                                                                                                                                                                                          |  42 kB  00:00:00     
    (3/15): libini_config-1.3.1-32.el7.x86_64.rpm                                                                                                                                                                                                          |  64 kB  00:00:00     
    (4/15): gssproxy-0.7.0-26.el7.x86_64.rpm                                                                                                                                                                                                               | 110 kB  00:00:00     
    (5/15): libnfsidmap-0.25-19.el7.x86_64.rpm                                                                                                                                                                                                             |  50 kB  00:00:00     
    (6/15): libref_array-0.1.5-32.el7.x86_64.rpm                                                                                                                                                                                                           |  27 kB  00:00:00     
    (7/15): libpath_utils-0.2.1-32.el7.x86_64.rpm                                                                                                                                                                                                          |  28 kB  00:00:00     
    (8/15): libverto-libevent-0.2.5-4.el7.x86_64.rpm                                                                                                                                                                                                       | 8.9 kB  00:00:00     
    (9/15): libtirpc-0.2.4-0.16.el7.x86_64.rpm                                                                                                                                                                                                             |  89 kB  00:00:00     
    (10/15): libbasicobjects-0.1.1-32.el7.x86_64.rpm                                                                                                                                                                                                       |  26 kB  00:00:00     
    (11/15): quota-4.01-19.el7.x86_64.rpm                                                                                                                                                                                                                  | 179 kB  00:00:00     
    (12/15): quota-nls-4.01-19.el7.noarch.rpm                                                                                                                                                                                                              |  90 kB  00:00:00     
    (13/15): tcp_wrappers-7.6-77.el7.x86_64.rpm                                                                                                                                                                                                            |  78 kB  00:00:00     
    (14/15): nfs-utils-1.3.0-0.65.el7.x86_64.rpm                                                                                                                                                                                                           | 412 kB  00:00:00     
    (15/15): rpcbind-0.2.0-48.el7.x86_64.rpm                                                                                                                                                                                                               |  60 kB  00:00:00     
    ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    Total                                                                                                                                                                                                                                         2.9 MB/s | 1.3 MB  00:00:00     
    Running transaction check
    Running transaction test
    Transaction test succeeded
    Running transaction
      Installing : libbasicobjects-0.1.1-32.el7.x86_64                                                                                                                                                                                                                       1/15 
      Installing : libref_array-0.1.5-32.el7.x86_64                                                                                                                                                                                                                          2/15 
      Installing : libcollection-0.7.0-32.el7.x86_64                                                                                                                                                                                                                         3/15 
      Installing : libtirpc-0.2.4-0.16.el7.x86_64                                                                                                                                                                                                                            4/15 
      Installing : rpcbind-0.2.0-48.el7.x86_64                                                                                                                                                                                                                               5/15 
      Installing : 1:quota-nls-4.01-19.el7.noarch                                                                                                                                                                                                                            6/15 
      Installing : tcp_wrappers-7.6-77.el7.x86_64                                                                                                                                                                                                                            7/15 
      Installing : 1:quota-4.01-19.el7.x86_64                                                                                                                                                                                                                                8/15 
      Installing : keyutils-1.5.8-3.el7.x86_64                                                                                                                                                                                                                               9/15 
      Installing : libnfsidmap-0.25-19.el7.x86_64                                                                                                                                                                                                                           10/15 
      Installing : libpath_utils-0.2.1-32.el7.x86_64                                                                                                                                                                                                                        11/15 
      Installing : libini_config-1.3.1-32.el7.x86_64                                                                                                                                                                                                                        12/15 
      Installing : libverto-libevent-0.2.5-4.el7.x86_64                                                                                                                                                                                                                     13/15 
      Installing : gssproxy-0.7.0-26.el7.x86_64                                                                                                                                                                                                                             14/15 
      Installing : 1:nfs-utils-1.3.0-0.65.el7.x86_64                                                                                                                                                                                                                        15/15 
      Verifying  : libtirpc-0.2.4-0.16.el7.x86_64                                                                                                                                                                                                                            1/15 
      Verifying  : libverto-libevent-0.2.5-4.el7.x86_64                                                                                                                                                                                                                      2/15 
      Verifying  : 1:quota-4.01-19.el7.x86_64                                                                                                                                                                                                                                3/15 
      Verifying  : gssproxy-0.7.0-26.el7.x86_64                                                                                                                                                                                                                              4/15 
      Verifying  : libpath_utils-0.2.1-32.el7.x86_64                                                                                                                                                                                                                         5/15 
      Verifying  : libnfsidmap-0.25-19.el7.x86_64                                                                                                                                                                                                                            6/15 
      Verifying  : keyutils-1.5.8-3.el7.x86_64                                                                                                                                                                                                                               7/15 
      Verifying  : 1:nfs-utils-1.3.0-0.65.el7.x86_64                                                                                                                                                                                                                         8/15 
      Verifying  : tcp_wrappers-7.6-77.el7.x86_64                                                                                                                                                                                                                            9/15 
      Verifying  : libcollection-0.7.0-32.el7.x86_64                                                                                                                                                                                                                        10/15 
      Verifying  : libref_array-0.1.5-32.el7.x86_64                                                                                                                                                                                                                         11/15 
      Verifying  : libbasicobjects-0.1.1-32.el7.x86_64                                                                                                                                                                                                                      12/15 
      Verifying  : rpcbind-0.2.0-48.el7.x86_64                                                                                                                                                                                                                              13/15 
      Verifying  : libini_config-1.3.1-32.el7.x86_64                                                                                                                                                                                                                        14/15 
      Verifying  : 1:quota-nls-4.01-19.el7.noarch                                                                                                                                                                                                                           15/15 
    
    Installed:
      nfs-utils.x86_64 1:1.3.0-0.65.el7                                                                                                                                                                                                                                           
    
    Dependency Installed:
      gssproxy.x86_64 0:0.7.0-26.el7        keyutils.x86_64 0:1.5.8-3.el7       libbasicobjects.x86_64 0:0.1.1-32.el7     libcollection.x86_64 0:0.7.0-32.el7    libini_config.x86_64 0:1.3.1-32.el7    libnfsidmap.x86_64 0:0.25-19.el7    libpath_utils.x86_64 0:0.2.1-32.el7   
      libref_array.x86_64 0:0.1.5-32.el7    libtirpc.x86_64 0:0.2.4-0.16.el7    libverto-libevent.x86_64 0:0.2.5-4.el7    quota.x86_64 1:4.01-19.el7             quota-nls.noarch 1:4.01-19.el7         rpcbind.x86_64 0:0.2.0-48.el7       tcp_wrappers.x86_64 0:7.6-77.el7      
    
    Complete!
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# yum -y install nfs-utils
    [root@test110.yinzhengjie.org.cn ~]# mkdir -pv /yinzhengjie/data/volume{1,2,3,4,5}
    mkdir: created directory ‘/yinzhengjie/data’
    mkdir: created directory ‘/yinzhengjie/data/volume1’
    mkdir: created directory ‘/yinzhengjie/data/volume2’
    mkdir: created directory ‘/yinzhengjie/data/volume3’
    mkdir: created directory ‘/yinzhengjie/data/volume4’
    mkdir: created directory ‘/yinzhengjie/data/volume5’
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# ll /yinzhengjie/data/ -R
    /yinzhengjie/data/:
    total 0
    drwxr-xr-x 2 root root 6 Feb 10 06:04 volume1
    drwxr-xr-x 2 root root 6 Feb 10 06:04 volume2
    drwxr-xr-x 2 root root 6 Feb 10 06:04 volume3
    drwxr-xr-x 2 root root 6 Feb 10 06:04 volume4
    drwxr-xr-x 2 root root 6 Feb 10 06:04 volume5
    
    /yinzhengjie/data/volume1:
    total 0
    
    /yinzhengjie/data/volume2:
    total 0
    
    /yinzhengjie/data/volume3:
    total 0
    
    /yinzhengjie/data/volume4:
    total 0
    
    /yinzhengjie/data/volume5:
    total 0
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# mkdir -pv /yinzhengjie/data/volume{1,2,3,4,5}
    [root@test110.yinzhengjie.org.cn ~]# vim /etc/exports
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# cat /etc/exports
    /yinzhengjie/data/volume1    172.200.0.0/21(rw,no_root_squash)
    /yinzhengjie/data/volume2    172.200.0.0/21(rw,no_root_squash)
    /yinzhengjie/data/volume3    172.200.0.0/21(rw,no_root_squash)
    /yinzhengjie/data/volume4    172.200.0.0/21(rw,no_root_squash)
    /yinzhengjie/data/volume5    172.200.0.0/21(rw,no_root_squash)
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# ll /yinzhengjie/data/
    total 0
    drwxr-xr-x 2 polkitd root 22 Feb 10 06:40 volume1
    drwxr-xr-x 2 root    root  6 Feb 10 06:04 volume2
    drwxr-xr-x 2 root    root  6 Feb 10 06:04 volume3
    drwxr-xr-x 2 root    root  6 Feb 10 06:04 volume4
    drwxr-xr-x 2 root    root  6 Feb 10 06:04 volume5
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# exportfs -rav
    exporting 172.200.0.0/21:/yinzhengjie/data/volume5
    exporting 172.200.0.0/21:/yinzhengjie/data/volume4
    exporting 172.200.0.0/21:/yinzhengjie/data/volume3
    exporting 172.200.0.0/21:/yinzhengjie/data/volume2
    exporting 172.200.0.0/21:/yinzhengjie/data/volume1
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# vim /etc/exports
    [root@test110.yinzhengjie.org.cn ~]# systemctl status nfs
    ● nfs-server.service - NFS server and services
       Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; disabled; vendor preset: disabled)
       Active: inactive (dead)
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# systemctl start nfs
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# systemctl enable nfs
    Created symlink from /etc/systemd/system/multi-user.target.wants/nfs-server.service to /usr/lib/systemd/system/nfs-server.service.
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# systemctl status nfs
    ● nfs-server.service - NFS server and services
       Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; enabled; vendor preset: disabled)
      Drop-In: /run/systemd/generator/nfs-server.service.d
               └─order-with-mounts.conf
       Active: active (exited) since Mon 2020-02-10 06:08:52 CST; 6s ago
     Main PID: 6116 (code=exited, status=0/SUCCESS)
       CGroup: /system.slice/nfs-server.service
    
    Feb 10 06:08:52 test110.yinzhengjie.org.cn systemd[1]: Starting NFS server and services...
    Feb 10 06:08:52 test110.yinzhengjie.org.cn systemd[1]: Started NFS server and services.
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# ss -ntl | grep 2049
    LISTEN     0      64           *:2049                     *:*                  
    LISTEN     0      64          :::2049                    :::*                  
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# iptables -vnL
    Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
     pkts bytes target     prot opt in     out     source               destination         
    
    Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
     pkts bytes target     prot opt in     out     source               destination         
    
    Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
     pkts bytes target     prot opt in     out     source               destination         
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# iptables -vnL -t nat
    Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
     pkts bytes target     prot opt in     out     source               destination         
    
    Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
     pkts bytes target     prot opt in     out     source               destination         
    
    Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
     pkts bytes target     prot opt in     out     source               destination         
    
    Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
     pkts bytes target     prot opt in     out     source               destination         
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# systemctl start nfs

    2>.在K8S集群的每一个节点安装nfs客户端驱动

    [root@master200.yinzhengjie.org.cn ~]# yum -y install nfs-utils
    Loaded plugins: fastestmirror
    Loading mirror speeds from cached hostfile
     * base: mirror.bit.edu.cn
     * extras: mirror.bit.edu.cn
     * updates: mirror.bit.edu.cn
    base                                                                                                                                                                                                                                                   | 3.6 kB  00:00:00     
    docker-ce-stable                                                                                                                                                                                                                                       | 3.5 kB  00:00:00     
    extras                                                                                                                                                                                                                                                 | 2.9 kB  00:00:00     
    kubernetes                                                                                                                                                                                                                                             | 1.4 kB  00:00:00     
    updates                                                                                                                                                                                                                                                | 2.9 kB  00:00:00     
    Resolving Dependencies
    --> Running transaction check
    ---> Package nfs-utils.x86_64 1:1.3.0-0.65.el7 will be installed
    --> Processing Dependency: libtirpc >= 0.2.4-0.7 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: gssproxy >= 0.7.0-3 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: rpcbind for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: quota for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: libnfsidmap for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: keyutils for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: libtirpc.so.1()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: libnfsidmap.so.0()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Running transaction check
    ---> Package gssproxy.x86_64 0:0.7.0-26.el7 will be installed
    --> Processing Dependency: libini_config >= 1.3.1-31 for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libverto-module-base for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libref_array.so.1(REF_ARRAY_0.1.1)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libini_config.so.3(INI_CONFIG_1.2.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libini_config.so.3(INI_CONFIG_1.1.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libref_array.so.1()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libini_config.so.3()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libcollection.so.2()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libbasicobjects.so.0()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    ---> Package keyutils.x86_64 0:1.5.8-3.el7 will be installed
    ---> Package libnfsidmap.x86_64 0:0.25-19.el7 will be installed
    ---> Package libtirpc.x86_64 0:0.2.4-0.16.el7 will be installed
    ---> Package quota.x86_64 1:4.01-19.el7 will be installed
    --> Processing Dependency: quota-nls = 1:4.01-19.el7 for package: 1:quota-4.01-19.el7.x86_64
    --> Processing Dependency: tcp_wrappers for package: 1:quota-4.01-19.el7.x86_64
    ---> Package rpcbind.x86_64 0:0.2.0-48.el7 will be installed
    --> Running transaction check
    ---> Package libbasicobjects.x86_64 0:0.1.1-32.el7 will be installed
    ---> Package libcollection.x86_64 0:0.7.0-32.el7 will be installed
    ---> Package libini_config.x86_64 0:1.3.1-32.el7 will be installed
    --> Processing Dependency: libpath_utils.so.1(PATH_UTILS_0.2.1)(64bit) for package: libini_config-1.3.1-32.el7.x86_64
    --> Processing Dependency: libpath_utils.so.1()(64bit) for package: libini_config-1.3.1-32.el7.x86_64
    ---> Package libref_array.x86_64 0:0.1.5-32.el7 will be installed
    ---> Package libverto-libevent.x86_64 0:0.2.5-4.el7 will be installed
    ---> Package quota-nls.noarch 1:4.01-19.el7 will be installed
    ---> Package tcp_wrappers.x86_64 0:7.6-77.el7 will be installed
    --> Running transaction check
    ---> Package libpath_utils.x86_64 0:0.2.1-32.el7 will be installed
    --> Finished Dependency Resolution
    
    Dependencies Resolved
    
    ==============================================================================================================================================================================================================================================================================
     Package                                                                 Arch                                                         Version                                                                Repository                                                  Size
    ==============================================================================================================================================================================================================================================================================
    Installing:
     nfs-utils                                                               x86_64                                                       1:1.3.0-0.65.el7                                                       base                                                       412 k
    Installing for dependencies:
     gssproxy                                                                x86_64                                                       0.7.0-26.el7                                                           base                                                       110 k
     keyutils                                                                x86_64                                                       1.5.8-3.el7                                                            base                                                        54 k
     libbasicobjects                                                         x86_64                                                       0.1.1-32.el7                                                           base                                                        26 k
     libcollection                                                           x86_64                                                       0.7.0-32.el7                                                           base                                                        42 k
     libini_config                                                           x86_64                                                       1.3.1-32.el7                                                           base                                                        64 k
     libnfsidmap                                                             x86_64                                                       0.25-19.el7                                                            base                                                        50 k
     libpath_utils                                                           x86_64                                                       0.2.1-32.el7                                                           base                                                        28 k
     libref_array                                                            x86_64                                                       0.1.5-32.el7                                                           base                                                        27 k
     libtirpc                                                                x86_64                                                       0.2.4-0.16.el7                                                         base                                                        89 k
     libverto-libevent                                                       x86_64                                                       0.2.5-4.el7                                                            base                                                       8.9 k
     quota                                                                   x86_64                                                       1:4.01-19.el7                                                          base                                                       179 k
     quota-nls                                                               noarch                                                       1:4.01-19.el7                                                          base                                                        90 k
     rpcbind                                                                 x86_64                                                       0.2.0-48.el7                                                           base                                                        60 k
     tcp_wrappers                                                            x86_64                                                       7.6-77.el7                                                             base                                                        78 k
    
    Transaction Summary
    ==============================================================================================================================================================================================================================================================================
    Install  1 Package (+14 Dependent packages)
    
    Total download size: 1.3 M
    Installed size: 3.6 M
    Downloading packages:
    (1/15): keyutils-1.5.8-3.el7.x86_64.rpm                                                                                                                                                                                                                |  54 kB  00:00:00     
    (2/15): gssproxy-0.7.0-26.el7.x86_64.rpm                                                                                                                                                                                                               | 110 kB  00:00:00     
    (3/15): libini_config-1.3.1-32.el7.x86_64.rpm                                                                                                                                                                                                          |  64 kB  00:00:00     
    (4/15): libbasicobjects-0.1.1-32.el7.x86_64.rpm                                                                                                                                                                                                        |  26 kB  00:00:00     
    (5/15): libpath_utils-0.2.1-32.el7.x86_64.rpm                                                                                                                                                                                                          |  28 kB  00:00:00     
    (6/15): libcollection-0.7.0-32.el7.x86_64.rpm                                                                                                                                                                                                          |  42 kB  00:00:00     
    (7/15): libref_array-0.1.5-32.el7.x86_64.rpm                                                                                                                                                                                                           |  27 kB  00:00:00     
    (8/15): libtirpc-0.2.4-0.16.el7.x86_64.rpm                                                                                                                                                                                                             |  89 kB  00:00:00     
    (9/15): libverto-libevent-0.2.5-4.el7.x86_64.rpm                                                                                                                                                                                                       | 8.9 kB  00:00:00     
    (10/15): quota-4.01-19.el7.x86_64.rpm                                                                                                                                                                                                                  | 179 kB  00:00:00     
    (11/15): libnfsidmap-0.25-19.el7.x86_64.rpm                                                                                                                                                                                                            |  50 kB  00:00:00     
    (12/15): rpcbind-0.2.0-48.el7.x86_64.rpm                                                                                                                                                                                                               |  60 kB  00:00:00     
    (13/15): nfs-utils-1.3.0-0.65.el7.x86_64.rpm                                                                                                                                                                                                           | 412 kB  00:00:00     
    (14/15): tcp_wrappers-7.6-77.el7.x86_64.rpm                                                                                                                                                                                                            |  78 kB  00:00:00     
    (15/15): quota-nls-4.01-19.el7.noarch.rpm                                                                                                                                                                                                              |  90 kB  00:00:00     
    ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    Total                                                                                                                                                                                                                                         1.4 MB/s | 1.3 MB  00:00:00     
    Running transaction check
    Running transaction test
    Transaction test succeeded
    Running transaction
      Installing : libbasicobjects-0.1.1-32.el7.x86_64                                                                                                                                                                                                                       1/15 
      Installing : libref_array-0.1.5-32.el7.x86_64                                                                                                                                                                                                                          2/15 
      Installing : libcollection-0.7.0-32.el7.x86_64                                                                                                                                                                                                                         3/15 
      Installing : libtirpc-0.2.4-0.16.el7.x86_64                                                                                                                                                                                                                            4/15 
      Installing : rpcbind-0.2.0-48.el7.x86_64                                                                                                                                                                                                                               5/15 
      Installing : 1:quota-nls-4.01-19.el7.noarch                                                                                                                                                                                                                            6/15 
      Installing : tcp_wrappers-7.6-77.el7.x86_64                                                                                                                                                                                                                            7/15 
      Installing : 1:quota-4.01-19.el7.x86_64                                                                                                                                                                                                                                8/15 
      Installing : keyutils-1.5.8-3.el7.x86_64                                                                                                                                                                                                                               9/15 
      Installing : libnfsidmap-0.25-19.el7.x86_64                                                                                                                                                                                                                           10/15 
      Installing : libpath_utils-0.2.1-32.el7.x86_64                                                                                                                                                                                                                        11/15 
      Installing : libini_config-1.3.1-32.el7.x86_64                                                                                                                                                                                                                        12/15 
      Installing : libverto-libevent-0.2.5-4.el7.x86_64                                                                                                                                                                                                                     13/15 
      Installing : gssproxy-0.7.0-26.el7.x86_64                                                                                                                                                                                                                             14/15 
      Installing : 1:nfs-utils-1.3.0-0.65.el7.x86_64                                                                                                                                                                                                                        15/15 
      Verifying  : libtirpc-0.2.4-0.16.el7.x86_64                                                                                                                                                                                                                            1/15 
      Verifying  : libverto-libevent-0.2.5-4.el7.x86_64                                                                                                                                                                                                                      2/15 
      Verifying  : 1:quota-4.01-19.el7.x86_64                                                                                                                                                                                                                                3/15 
      Verifying  : gssproxy-0.7.0-26.el7.x86_64                                                                                                                                                                                                                              4/15 
      Verifying  : libpath_utils-0.2.1-32.el7.x86_64                                                                                                                                                                                                                         5/15 
      Verifying  : libnfsidmap-0.25-19.el7.x86_64                                                                                                                                                                                                                            6/15 
      Verifying  : keyutils-1.5.8-3.el7.x86_64                                                                                                                                                                                                                               7/15 
      Verifying  : 1:nfs-utils-1.3.0-0.65.el7.x86_64                                                                                                                                                                                                                         8/15 
      Verifying  : tcp_wrappers-7.6-77.el7.x86_64                                                                                                                                                                                                                            9/15 
      Verifying  : libcollection-0.7.0-32.el7.x86_64                                                                                                                                                                                                                        10/15 
      Verifying  : libref_array-0.1.5-32.el7.x86_64                                                                                                                                                                                                                         11/15 
      Verifying  : libbasicobjects-0.1.1-32.el7.x86_64                                                                                                                                                                                                                      12/15 
      Verifying  : rpcbind-0.2.0-48.el7.x86_64                                                                                                                                                                                                                              13/15 
      Verifying  : libini_config-1.3.1-32.el7.x86_64                                                                                                                                                                                                                        14/15 
      Verifying  : 1:quota-nls-4.01-19.el7.noarch                                                                                                                                                                                                                           15/15 
    
    Installed:
      nfs-utils.x86_64 1:1.3.0-0.65.el7                                                                                                                                                                                                                                           
    
    Dependency Installed:
      gssproxy.x86_64 0:0.7.0-26.el7        keyutils.x86_64 0:1.5.8-3.el7       libbasicobjects.x86_64 0:0.1.1-32.el7     libcollection.x86_64 0:0.7.0-32.el7    libini_config.x86_64 0:1.3.1-32.el7    libnfsidmap.x86_64 0:0.25-19.el7    libpath_utils.x86_64 0:0.2.1-32.el7   
      libref_array.x86_64 0:0.1.5-32.el7    libtirpc.x86_64 0:0.2.4-0.16.el7    libverto-libevent.x86_64 0:0.2.5-4.el7    quota.x86_64 1:4.01-19.el7             quota-nls.noarch 1:4.01-19.el7         rpcbind.x86_64 0:0.2.0-48.el7       tcp_wrappers.x86_64 0:7.6-77.el7      
    
    Complete!
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# yum -y install nfs-utils
    [root@node201.yinzhengjie.org.cn ~]#  yum -y install nfs-utils
    Loaded plugins: fastestmirror
    Loading mirror speeds from cached hostfile
     * base: mirrors.aliyun.com
     * extras: mirrors.aliyun.com
     * updates: mirror.bit.edu.cn
    base                                                                                                                                                                                                                                                   | 3.6 kB  00:00:00     
    docker-ce-stable                                                                                                                                                                                                                                       | 3.5 kB  00:00:00     
    extras                                                                                                                                                                                                                                                 | 2.9 kB  00:00:00     
    kubernetes                                                                                                                                                                                                                                             | 1.4 kB  00:00:00     
    updates                                                                                                                                                                                                                                                | 2.9 kB  00:00:00     
    updates/7/x86_64/primary_db                                                                                                                                                                                                                            | 6.7 MB  00:00:00     
    Resolving Dependencies
    --> Running transaction check
    ---> Package nfs-utils.x86_64 1:1.3.0-0.65.el7 will be installed
    --> Processing Dependency: libtirpc >= 0.2.4-0.7 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: gssproxy >= 0.7.0-3 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: rpcbind for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: quota for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: libnfsidmap for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: keyutils for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: libtirpc.so.1()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: libnfsidmap.so.0()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Running transaction check
    ---> Package gssproxy.x86_64 0:0.7.0-26.el7 will be installed
    --> Processing Dependency: libini_config >= 1.3.1-31 for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libverto-module-base for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libref_array.so.1(REF_ARRAY_0.1.1)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libini_config.so.3(INI_CONFIG_1.2.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libini_config.so.3(INI_CONFIG_1.1.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libref_array.so.1()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libini_config.so.3()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libcollection.so.2()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libbasicobjects.so.0()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    ---> Package keyutils.x86_64 0:1.5.8-3.el7 will be installed
    ---> Package libnfsidmap.x86_64 0:0.25-19.el7 will be installed
    ---> Package libtirpc.x86_64 0:0.2.4-0.16.el7 will be installed
    ---> Package quota.x86_64 1:4.01-19.el7 will be installed
    --> Processing Dependency: quota-nls = 1:4.01-19.el7 for package: 1:quota-4.01-19.el7.x86_64
    --> Processing Dependency: tcp_wrappers for package: 1:quota-4.01-19.el7.x86_64
    ---> Package rpcbind.x86_64 0:0.2.0-48.el7 will be installed
    --> Running transaction check
    ---> Package libbasicobjects.x86_64 0:0.1.1-32.el7 will be installed
    ---> Package libcollection.x86_64 0:0.7.0-32.el7 will be installed
    ---> Package libini_config.x86_64 0:1.3.1-32.el7 will be installed
    --> Processing Dependency: libpath_utils.so.1(PATH_UTILS_0.2.1)(64bit) for package: libini_config-1.3.1-32.el7.x86_64
    --> Processing Dependency: libpath_utils.so.1()(64bit) for package: libini_config-1.3.1-32.el7.x86_64
    ---> Package libref_array.x86_64 0:0.1.5-32.el7 will be installed
    ---> Package libverto-libevent.x86_64 0:0.2.5-4.el7 will be installed
    ---> Package quota-nls.noarch 1:4.01-19.el7 will be installed
    ---> Package tcp_wrappers.x86_64 0:7.6-77.el7 will be installed
    --> Running transaction check
    ---> Package libpath_utils.x86_64 0:0.2.1-32.el7 will be installed
    --> Finished Dependency Resolution
    
    Dependencies Resolved
    
    ==============================================================================================================================================================================================================================================================================
     Package                                                                 Arch                                                         Version                                                                Repository                                                  Size
    ==============================================================================================================================================================================================================================================================================
    Installing:
     nfs-utils                                                               x86_64                                                       1:1.3.0-0.65.el7                                                       base                                                       412 k
    Installing for dependencies:
     gssproxy                                                                x86_64                                                       0.7.0-26.el7                                                           base                                                       110 k
     keyutils                                                                x86_64                                                       1.5.8-3.el7                                                            base                                                        54 k
     libbasicobjects                                                         x86_64                                                       0.1.1-32.el7                                                           base                                                        26 k
     libcollection                                                           x86_64                                                       0.7.0-32.el7                                                           base                                                        42 k
     libini_config                                                           x86_64                                                       1.3.1-32.el7                                                           base                                                        64 k
     libnfsidmap                                                             x86_64                                                       0.25-19.el7                                                            base                                                        50 k
     libpath_utils                                                           x86_64                                                       0.2.1-32.el7                                                           base                                                        28 k
     libref_array                                                            x86_64                                                       0.1.5-32.el7                                                           base                                                        27 k
     libtirpc                                                                x86_64                                                       0.2.4-0.16.el7                                                         base                                                        89 k
     libverto-libevent                                                       x86_64                                                       0.2.5-4.el7                                                            base                                                       8.9 k
     quota                                                                   x86_64                                                       1:4.01-19.el7                                                          base                                                       179 k
     quota-nls                                                               noarch                                                       1:4.01-19.el7                                                          base                                                        90 k
     rpcbind                                                                 x86_64                                                       0.2.0-48.el7                                                           base                                                        60 k
     tcp_wrappers                                                            x86_64                                                       7.6-77.el7                                                             base                                                        78 k
    
    Transaction Summary
    ==============================================================================================================================================================================================================================================================================
    Install  1 Package (+14 Dependent packages)
    
    Total download size: 1.3 M
    Installed size: 3.6 M
    Downloading packages:
    (1/15): gssproxy-0.7.0-26.el7.x86_64.rpm                                                                                                                                                                                                               | 110 kB  00:00:00     
    (2/15): keyutils-1.5.8-3.el7.x86_64.rpm                                                                                                                                                                                                                |  54 kB  00:00:00     
    (3/15): libcollection-0.7.0-32.el7.x86_64.rpm                                                                                                                                                                                                          |  42 kB  00:00:00     
    (4/15): libnfsidmap-0.25-19.el7.x86_64.rpm                                                                                                                                                                                                             |  50 kB  00:00:00     
    (5/15): libbasicobjects-0.1.1-32.el7.x86_64.rpm                                                                                                                                                                                                        |  26 kB  00:00:00     
    (6/15): libini_config-1.3.1-32.el7.x86_64.rpm                                                                                                                                                                                                          |  64 kB  00:00:00     
    (7/15): libtirpc-0.2.4-0.16.el7.x86_64.rpm                                                                                                                                                                                                             |  89 kB  00:00:00     
    (8/15): libref_array-0.1.5-32.el7.x86_64.rpm                                                                                                                                                                                                           |  27 kB  00:00:00     
    (9/15): libverto-libevent-0.2.5-4.el7.x86_64.rpm                                                                                                                                                                                                       | 8.9 kB  00:00:00     
    (10/15): libpath_utils-0.2.1-32.el7.x86_64.rpm                                                                                                                                                                                                         |  28 kB  00:00:00     
    (11/15): nfs-utils-1.3.0-0.65.el7.x86_64.rpm                                                                                                                                                                                                           | 412 kB  00:00:00     
    (12/15): quota-4.01-19.el7.x86_64.rpm                                                                                                                                                                                                                  | 179 kB  00:00:00     
    (13/15): quota-nls-4.01-19.el7.noarch.rpm                                                                                                                                                                                                              |  90 kB  00:00:00     
    (14/15): tcp_wrappers-7.6-77.el7.x86_64.rpm                                                                                                                                                                                                            |  78 kB  00:00:00     
    (15/15): rpcbind-0.2.0-48.el7.x86_64.rpm                                                                                                                                                                                                               |  60 kB  00:00:00     
    ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    Total                                                                                                                                                                                                                                         2.2 MB/s | 1.3 MB  00:00:00     
    Running transaction check
    Running transaction test
    Transaction test succeeded
    Running transaction
      Installing : libbasicobjects-0.1.1-32.el7.x86_64                                                                                                                                                                                                                       1/15 
      Installing : libref_array-0.1.5-32.el7.x86_64                                                                                                                                                                                                                          2/15 
      Installing : libcollection-0.7.0-32.el7.x86_64                                                                                                                                                                                                                         3/15 
      Installing : libtirpc-0.2.4-0.16.el7.x86_64                                                                                                                                                                                                                            4/15 
      Installing : rpcbind-0.2.0-48.el7.x86_64                                                                                                                                                                                                                               5/15 
      Installing : 1:quota-nls-4.01-19.el7.noarch                                                                                                                                                                                                                            6/15 
      Installing : tcp_wrappers-7.6-77.el7.x86_64                                                                                                                                                                                                                            7/15 
      Installing : 1:quota-4.01-19.el7.x86_64                                                                                                                                                                                                                                8/15 
      Installing : keyutils-1.5.8-3.el7.x86_64                                                                                                                                                                                                                               9/15 
      Installing : libnfsidmap-0.25-19.el7.x86_64                                                                                                                                                                                                                           10/15 
      Installing : libpath_utils-0.2.1-32.el7.x86_64                                                                                                                                                                                                                        11/15 
      Installing : libini_config-1.3.1-32.el7.x86_64                                                                                                                                                                                                                        12/15 
      Installing : libverto-libevent-0.2.5-4.el7.x86_64                                                                                                                                                                                                                     13/15 
      Installing : gssproxy-0.7.0-26.el7.x86_64                                                                                                                                                                                                                             14/15 
      Installing : 1:nfs-utils-1.3.0-0.65.el7.x86_64                                                                                                                                                                                                                        15/15 
      Verifying  : libtirpc-0.2.4-0.16.el7.x86_64                                                                                                                                                                                                                            1/15 
      Verifying  : libverto-libevent-0.2.5-4.el7.x86_64                                                                                                                                                                                                                      2/15 
      Verifying  : 1:quota-4.01-19.el7.x86_64                                                                                                                                                                                                                                3/15 
      Verifying  : gssproxy-0.7.0-26.el7.x86_64                                                                                                                                                                                                                              4/15 
      Verifying  : libpath_utils-0.2.1-32.el7.x86_64                                                                                                                                                                                                                         5/15 
      Verifying  : libnfsidmap-0.25-19.el7.x86_64                                                                                                                                                                                                                            6/15 
      Verifying  : keyutils-1.5.8-3.el7.x86_64                                                                                                                                                                                                                               7/15 
      Verifying  : 1:nfs-utils-1.3.0-0.65.el7.x86_64                                                                                                                                                                                                                         8/15 
      Verifying  : tcp_wrappers-7.6-77.el7.x86_64                                                                                                                                                                                                                            9/15 
      Verifying  : libcollection-0.7.0-32.el7.x86_64                                                                                                                                                                                                                        10/15 
      Verifying  : libref_array-0.1.5-32.el7.x86_64                                                                                                                                                                                                                         11/15 
      Verifying  : libbasicobjects-0.1.1-32.el7.x86_64                                                                                                                                                                                                                      12/15 
      Verifying  : rpcbind-0.2.0-48.el7.x86_64                                                                                                                                                                                                                              13/15 
      Verifying  : libini_config-1.3.1-32.el7.x86_64                                                                                                                                                                                                                        14/15 
      Verifying  : 1:quota-nls-4.01-19.el7.noarch                                                                                                                                                                                                                           15/15 
    
    Installed:
      nfs-utils.x86_64 1:1.3.0-0.65.el7                                                                                                                                                                                                                                           
    
    Dependency Installed:
      gssproxy.x86_64 0:0.7.0-26.el7        keyutils.x86_64 0:1.5.8-3.el7       libbasicobjects.x86_64 0:0.1.1-32.el7     libcollection.x86_64 0:0.7.0-32.el7    libini_config.x86_64 0:1.3.1-32.el7    libnfsidmap.x86_64 0:0.25-19.el7    libpath_utils.x86_64 0:0.2.1-32.el7   
      libref_array.x86_64 0:0.1.5-32.el7    libtirpc.x86_64 0:0.2.4-0.16.el7    libverto-libevent.x86_64 0:0.2.5-4.el7    quota.x86_64 1:4.01-19.el7             quota-nls.noarch 1:4.01-19.el7         rpcbind.x86_64 0:0.2.0-48.el7       tcp_wrappers.x86_64 0:7.6-77.el7      
    
    Complete!
    [root@node201.yinzhengjie.org.cn ~]# 
    [root@node201.yinzhengjie.org.cn ~]# yum -y install nfs-utils
    [root@node202.yinzhengjie.org.cn ~]# yum -y install nfs-utils
    Loaded plugins: fastestmirror
    Loading mirror speeds from cached hostfile
     * base: mirror.bit.edu.cn
     * extras: mirror.bit.edu.cn
     * updates: mirror.bit.edu.cn
    base                                                                                                                                                                                                                                                   | 3.6 kB  00:00:00     
    docker-ce-stable                                                                                                                                                                                                                                       | 3.5 kB  00:00:00     
    extras                                                                                                                                                                                                                                                 | 2.9 kB  00:00:00     
    kubernetes                                                                                                                                                                                                                                             | 1.4 kB  00:00:00     
    updates                                                                                                                                                                                                                                                | 2.9 kB  00:00:00     
    updates/7/x86_64/primary_db                                                                                                                                                                                                                            | 6.7 MB  00:00:01     
    Resolving Dependencies
    --> Running transaction check
    ---> Package nfs-utils.x86_64 1:1.3.0-0.65.el7 will be installed
    --> Processing Dependency: libtirpc >= 0.2.4-0.7 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: gssproxy >= 0.7.0-3 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: rpcbind for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: quota for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: libnfsidmap for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: keyutils for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: libtirpc.so.1()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: libnfsidmap.so.0()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Running transaction check
    ---> Package gssproxy.x86_64 0:0.7.0-26.el7 will be installed
    --> Processing Dependency: libini_config >= 1.3.1-31 for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libverto-module-base for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libref_array.so.1(REF_ARRAY_0.1.1)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libini_config.so.3(INI_CONFIG_1.2.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libini_config.so.3(INI_CONFIG_1.1.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libref_array.so.1()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libini_config.so.3()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libcollection.so.2()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libbasicobjects.so.0()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    ---> Package keyutils.x86_64 0:1.5.8-3.el7 will be installed
    ---> Package libnfsidmap.x86_64 0:0.25-19.el7 will be installed
    ---> Package libtirpc.x86_64 0:0.2.4-0.16.el7 will be installed
    ---> Package quota.x86_64 1:4.01-19.el7 will be installed
    --> Processing Dependency: quota-nls = 1:4.01-19.el7 for package: 1:quota-4.01-19.el7.x86_64
    --> Processing Dependency: tcp_wrappers for package: 1:quota-4.01-19.el7.x86_64
    ---> Package rpcbind.x86_64 0:0.2.0-48.el7 will be installed
    --> Running transaction check
    ---> Package libbasicobjects.x86_64 0:0.1.1-32.el7 will be installed
    ---> Package libcollection.x86_64 0:0.7.0-32.el7 will be installed
    ---> Package libini_config.x86_64 0:1.3.1-32.el7 will be installed
    --> Processing Dependency: libpath_utils.so.1(PATH_UTILS_0.2.1)(64bit) for package: libini_config-1.3.1-32.el7.x86_64
    --> Processing Dependency: libpath_utils.so.1()(64bit) for package: libini_config-1.3.1-32.el7.x86_64
    ---> Package libref_array.x86_64 0:0.1.5-32.el7 will be installed
    ---> Package libverto-libevent.x86_64 0:0.2.5-4.el7 will be installed
    ---> Package quota-nls.noarch 1:4.01-19.el7 will be installed
    ---> Package tcp_wrappers.x86_64 0:7.6-77.el7 will be installed
    --> Running transaction check
    ---> Package libpath_utils.x86_64 0:0.2.1-32.el7 will be installed
    --> Finished Dependency Resolution
    
    Dependencies Resolved
    
    ==============================================================================================================================================================================================================================================================================
     Package                                                                 Arch                                                         Version                                                                Repository                                                  Size
    ==============================================================================================================================================================================================================================================================================
    Installing:
     nfs-utils                                                               x86_64                                                       1:1.3.0-0.65.el7                                                       base                                                       412 k
    Installing for dependencies:
     gssproxy                                                                x86_64                                                       0.7.0-26.el7                                                           base                                                       110 k
     keyutils                                                                x86_64                                                       1.5.8-3.el7                                                            base                                                        54 k
     libbasicobjects                                                         x86_64                                                       0.1.1-32.el7                                                           base                                                        26 k
     libcollection                                                           x86_64                                                       0.7.0-32.el7                                                           base                                                        42 k
     libini_config                                                           x86_64                                                       1.3.1-32.el7                                                           base                                                        64 k
     libnfsidmap                                                             x86_64                                                       0.25-19.el7                                                            base                                                        50 k
     libpath_utils                                                           x86_64                                                       0.2.1-32.el7                                                           base                                                        28 k
     libref_array                                                            x86_64                                                       0.1.5-32.el7                                                           base                                                        27 k
     libtirpc                                                                x86_64                                                       0.2.4-0.16.el7                                                         base                                                        89 k
     libverto-libevent                                                       x86_64                                                       0.2.5-4.el7                                                            base                                                       8.9 k
     quota                                                                   x86_64                                                       1:4.01-19.el7                                                          base                                                       179 k
     quota-nls                                                               noarch                                                       1:4.01-19.el7                                                          base                                                        90 k
     rpcbind                                                                 x86_64                                                       0.2.0-48.el7                                                           base                                                        60 k
     tcp_wrappers                                                            x86_64                                                       7.6-77.el7                                                             base                                                        78 k
    
    Transaction Summary
    ==============================================================================================================================================================================================================================================================================
    Install  1 Package (+14 Dependent packages)
    
    Total download size: 1.3 M
    Installed size: 3.6 M
    Downloading packages:
    (1/15): keyutils-1.5.8-3.el7.x86_64.rpm                                                                                                                                                                                                                |  54 kB  00:00:00     
    (2/15): libbasicobjects-0.1.1-32.el7.x86_64.rpm                                                                                                                                                                                                        |  26 kB  00:00:00     
    (3/15): libnfsidmap-0.25-19.el7.x86_64.rpm                                                                                                                                                                                                             |  50 kB  00:00:00     
    (4/15): libini_config-1.3.1-32.el7.x86_64.rpm                                                                                                                                                                                                          |  64 kB  00:00:00     
    (5/15): libref_array-0.1.5-32.el7.x86_64.rpm                                                                                                                                                                                                           |  27 kB  00:00:00     
    (6/15): libpath_utils-0.2.1-32.el7.x86_64.rpm                                                                                                                                                                                                          |  28 kB  00:00:00     
    (7/15): libverto-libevent-0.2.5-4.el7.x86_64.rpm                                                                                                                                                                                                       | 8.9 kB  00:00:00     
    (8/15): gssproxy-0.7.0-26.el7.x86_64.rpm                                                                                                                                                                                                               | 110 kB  00:00:00     
    (9/15): libcollection-0.7.0-32.el7.x86_64.rpm                                                                                                                                                                                                          |  42 kB  00:00:00     
    (10/15): nfs-utils-1.3.0-0.65.el7.x86_64.rpm                                                                                                                                                                                                           | 412 kB  00:00:00     
    (11/15): libtirpc-0.2.4-0.16.el7.x86_64.rpm                                                                                                                                                                                                            |  89 kB  00:00:00     
    (12/15): quota-4.01-19.el7.x86_64.rpm                                                                                                                                                                                                                  | 179 kB  00:00:00     
    (13/15): tcp_wrappers-7.6-77.el7.x86_64.rpm                                                                                                                                                                                                            |  78 kB  00:00:00     
    (14/15): quota-nls-4.01-19.el7.noarch.rpm                                                                                                                                                                                                              |  90 kB  00:00:00     
    (15/15): rpcbind-0.2.0-48.el7.x86_64.rpm                                                                                                                                                                                                               |  60 kB  00:00:00     
    ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    Total                                                                                                                                                                                                                                         1.9 MB/s | 1.3 MB  00:00:00     
    Running transaction check
    Running transaction test
    Transaction test succeeded
    Running transaction
      Installing : libbasicobjects-0.1.1-32.el7.x86_64                                                                                                                                                                                                                       1/15 
      Installing : libref_array-0.1.5-32.el7.x86_64                                                                                                                                                                                                                          2/15 
      Installing : libcollection-0.7.0-32.el7.x86_64                                                                                                                                                                                                                         3/15 
      Installing : libtirpc-0.2.4-0.16.el7.x86_64                                                                                                                                                                                                                            4/15 
      Installing : rpcbind-0.2.0-48.el7.x86_64                                                                                                                                                                                                                               5/15 
      Installing : 1:quota-nls-4.01-19.el7.noarch                                                                                                                                                                                                                            6/15 
      Installing : tcp_wrappers-7.6-77.el7.x86_64                                                                                                                                                                                                                            7/15 
      Installing : 1:quota-4.01-19.el7.x86_64                                                                                                                                                                                                                                8/15 
      Installing : keyutils-1.5.8-3.el7.x86_64                                                                                                                                                                                                                               9/15 
      Installing : libnfsidmap-0.25-19.el7.x86_64                                                                                                                                                                                                                           10/15 
      Installing : libpath_utils-0.2.1-32.el7.x86_64                                                                                                                                                                                                                        11/15 
      Installing : libini_config-1.3.1-32.el7.x86_64                                                                                                                                                                                                                        12/15 
      Installing : libverto-libevent-0.2.5-4.el7.x86_64                                                                                                                                                                                                                     13/15 
      Installing : gssproxy-0.7.0-26.el7.x86_64                                                                                                                                                                                                                             14/15 
      Installing : 1:nfs-utils-1.3.0-0.65.el7.x86_64                                                                                                                                                                                                                        15/15 
      Verifying  : libtirpc-0.2.4-0.16.el7.x86_64                                                                                                                                                                                                                            1/15 
      Verifying  : libverto-libevent-0.2.5-4.el7.x86_64                                                                                                                                                                                                                      2/15 
      Verifying  : 1:quota-4.01-19.el7.x86_64                                                                                                                                                                                                                                3/15 
      Verifying  : gssproxy-0.7.0-26.el7.x86_64                                                                                                                                                                                                                              4/15 
      Verifying  : libpath_utils-0.2.1-32.el7.x86_64                                                                                                                                                                                                                         5/15 
      Verifying  : libnfsidmap-0.25-19.el7.x86_64                                                                                                                                                                                                                            6/15 
      Verifying  : keyutils-1.5.8-3.el7.x86_64                                                                                                                                                                                                                               7/15 
      Verifying  : 1:nfs-utils-1.3.0-0.65.el7.x86_64                                                                                                                                                                                                                         8/15 
      Verifying  : tcp_wrappers-7.6-77.el7.x86_64                                                                                                                                                                                                                            9/15 
      Verifying  : libcollection-0.7.0-32.el7.x86_64                                                                                                                                                                                                                        10/15 
      Verifying  : libref_array-0.1.5-32.el7.x86_64                                                                                                                                                                                                                         11/15 
      Verifying  : libbasicobjects-0.1.1-32.el7.x86_64                                                                                                                                                                                                                      12/15 
      Verifying  : rpcbind-0.2.0-48.el7.x86_64                                                                                                                                                                                                                              13/15 
      Verifying  : libini_config-1.3.1-32.el7.x86_64                                                                                                                                                                                                                        14/15 
      Verifying  : 1:quota-nls-4.01-19.el7.noarch                                                                                                                                                                                                                           15/15 
    
    Installed:
      nfs-utils.x86_64 1:1.3.0-0.65.el7                                                                                                                                                                                                                                           
    
    Dependency Installed:
      gssproxy.x86_64 0:0.7.0-26.el7        keyutils.x86_64 0:1.5.8-3.el7       libbasicobjects.x86_64 0:0.1.1-32.el7     libcollection.x86_64 0:0.7.0-32.el7    libini_config.x86_64 0:1.3.1-32.el7    libnfsidmap.x86_64 0:0.25-19.el7    libpath_utils.x86_64 0:0.2.1-32.el7   
      libref_array.x86_64 0:0.1.5-32.el7    libtirpc.x86_64 0:0.2.4-0.16.el7    libverto-libevent.x86_64 0:0.2.5-4.el7    quota.x86_64 1:4.01-19.el7             quota-nls.noarch 1:4.01-19.el7         rpcbind.x86_64 0:0.2.0-48.el7       tcp_wrappers.x86_64 0:7.6-77.el7      
    
    Complete!
    [root@node202.yinzhengjie.org.cn ~]# 
    [root@node202.yinzhengjie.org.cn ~]# yum -y install nfs-utils
    [root@node203.yinzhengjie.org.cn ~]# yum -y install nfs-utils
    Loaded plugins: fastestmirror
    Loading mirror speeds from cached hostfile
     * base: mirrors.aliyun.com
     * extras: mirrors.aliyun.com
     * updates: mirror.bit.edu.cn
    base                                                                                                                                                                                                                                                   | 3.6 kB  00:00:00     
    docker-ce-stable                                                                                                                                                                                                                                       | 3.5 kB  00:00:00     
    extras                                                                                                                                                                                                                                                 | 2.9 kB  00:00:00     
    kubernetes                                                                                                                                                                                                                                             | 1.4 kB  00:00:00     
    updates                                                                                                                                                                                                                                                | 2.9 kB  00:00:00     
    updates/7/x86_64/primary_db                                                                                                                                                                                                                            | 6.7 MB  00:00:00     
    Resolving Dependencies
    --> Running transaction check
    ---> Package nfs-utils.x86_64 1:1.3.0-0.65.el7 will be installed
    --> Processing Dependency: libtirpc >= 0.2.4-0.7 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: gssproxy >= 0.7.0-3 for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: rpcbind for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: quota for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: libnfsidmap for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: keyutils for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: libtirpc.so.1()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Processing Dependency: libnfsidmap.so.0()(64bit) for package: 1:nfs-utils-1.3.0-0.65.el7.x86_64
    --> Running transaction check
    ---> Package gssproxy.x86_64 0:0.7.0-26.el7 will be installed
    --> Processing Dependency: libini_config >= 1.3.1-31 for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libverto-module-base for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libref_array.so.1(REF_ARRAY_0.1.1)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libini_config.so.3(INI_CONFIG_1.2.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libini_config.so.3(INI_CONFIG_1.1.0)(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libref_array.so.1()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libini_config.so.3()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libcollection.so.2()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    --> Processing Dependency: libbasicobjects.so.0()(64bit) for package: gssproxy-0.7.0-26.el7.x86_64
    ---> Package keyutils.x86_64 0:1.5.8-3.el7 will be installed
    ---> Package libnfsidmap.x86_64 0:0.25-19.el7 will be installed
    ---> Package libtirpc.x86_64 0:0.2.4-0.16.el7 will be installed
    ---> Package quota.x86_64 1:4.01-19.el7 will be installed
    --> Processing Dependency: quota-nls = 1:4.01-19.el7 for package: 1:quota-4.01-19.el7.x86_64
    --> Processing Dependency: tcp_wrappers for package: 1:quota-4.01-19.el7.x86_64
    ---> Package rpcbind.x86_64 0:0.2.0-48.el7 will be installed
    --> Running transaction check
    ---> Package libbasicobjects.x86_64 0:0.1.1-32.el7 will be installed
    ---> Package libcollection.x86_64 0:0.7.0-32.el7 will be installed
    ---> Package libini_config.x86_64 0:1.3.1-32.el7 will be installed
    --> Processing Dependency: libpath_utils.so.1(PATH_UTILS_0.2.1)(64bit) for package: libini_config-1.3.1-32.el7.x86_64
    --> Processing Dependency: libpath_utils.so.1()(64bit) for package: libini_config-1.3.1-32.el7.x86_64
    ---> Package libref_array.x86_64 0:0.1.5-32.el7 will be installed
    ---> Package libverto-libevent.x86_64 0:0.2.5-4.el7 will be installed
    ---> Package quota-nls.noarch 1:4.01-19.el7 will be installed
    ---> Package tcp_wrappers.x86_64 0:7.6-77.el7 will be installed
    --> Running transaction check
    ---> Package libpath_utils.x86_64 0:0.2.1-32.el7 will be installed
    --> Finished Dependency Resolution
    
    Dependencies Resolved
    
    ==============================================================================================================================================================================================================================================================================
     Package                                                                 Arch                                                         Version                                                                Repository                                                  Size
    ==============================================================================================================================================================================================================================================================================
    Installing:
     nfs-utils                                                               x86_64                                                       1:1.3.0-0.65.el7                                                       base                                                       412 k
    Installing for dependencies:
     gssproxy                                                                x86_64                                                       0.7.0-26.el7                                                           base                                                       110 k
     keyutils                                                                x86_64                                                       1.5.8-3.el7                                                            base                                                        54 k
     libbasicobjects                                                         x86_64                                                       0.1.1-32.el7                                                           base                                                        26 k
     libcollection                                                           x86_64                                                       0.7.0-32.el7                                                           base                                                        42 k
     libini_config                                                           x86_64                                                       1.3.1-32.el7                                                           base                                                        64 k
     libnfsidmap                                                             x86_64                                                       0.25-19.el7                                                            base                                                        50 k
     libpath_utils                                                           x86_64                                                       0.2.1-32.el7                                                           base                                                        28 k
     libref_array                                                            x86_64                                                       0.1.5-32.el7                                                           base                                                        27 k
     libtirpc                                                                x86_64                                                       0.2.4-0.16.el7                                                         base                                                        89 k
     libverto-libevent                                                       x86_64                                                       0.2.5-4.el7                                                            base                                                       8.9 k
     quota                                                                   x86_64                                                       1:4.01-19.el7                                                          base                                                       179 k
     quota-nls                                                               noarch                                                       1:4.01-19.el7                                                          base                                                        90 k
     rpcbind                                                                 x86_64                                                       0.2.0-48.el7                                                           base                                                        60 k
     tcp_wrappers                                                            x86_64                                                       7.6-77.el7                                                             base                                                        78 k
    
    Transaction Summary
    ==============================================================================================================================================================================================================================================================================
    Install  1 Package (+14 Dependent packages)
    
    Total download size: 1.3 M
    Installed size: 3.6 M
    Downloading packages:
    (1/15): libcollection-0.7.0-32.el7.x86_64.rpm                                                                                                                                                                                                          |  42 kB  00:00:00     
    (2/15): gssproxy-0.7.0-26.el7.x86_64.rpm                                                                                                                                                                                                               | 110 kB  00:00:00     
    (3/15): libpath_utils-0.2.1-32.el7.x86_64.rpm                                                                                                                                                                                                          |  28 kB  00:00:00     
    (4/15): libref_array-0.1.5-32.el7.x86_64.rpm                                                                                                                                                                                                           |  27 kB  00:00:00     
    (5/15): keyutils-1.5.8-3.el7.x86_64.rpm                                                                                                                                                                                                                |  54 kB  00:00:00     
    (6/15): libnfsidmap-0.25-19.el7.x86_64.rpm                                                                                                                                                                                                             |  50 kB  00:00:00     
    (7/15): libini_config-1.3.1-32.el7.x86_64.rpm                                                                                                                                                                                                          |  64 kB  00:00:00     
    (8/15): libtirpc-0.2.4-0.16.el7.x86_64.rpm                                                                                                                                                                                                             |  89 kB  00:00:00     
    (9/15): quota-nls-4.01-19.el7.noarch.rpm                                                                                                                                                                                                               |  90 kB  00:00:00     
    (10/15): quota-4.01-19.el7.x86_64.rpm                                                                                                                                                                                                                  | 179 kB  00:00:00     
    (11/15): libbasicobjects-0.1.1-32.el7.x86_64.rpm                                                                                                                                                                                                       |  26 kB  00:00:00     
    (12/15): libverto-libevent-0.2.5-4.el7.x86_64.rpm                                                                                                                                                                                                      | 8.9 kB  00:00:00     
    (13/15): rpcbind-0.2.0-48.el7.x86_64.rpm                                                                                                                                                                                                               |  60 kB  00:00:00     
    (14/15): tcp_wrappers-7.6-77.el7.x86_64.rpm                                                                                                                                                                                                            |  78 kB  00:00:00     
    (15/15): nfs-utils-1.3.0-0.65.el7.x86_64.rpm                                                                                                                                                                                                           | 412 kB  00:00:00     
    ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    Total                                                                                                                                                                                                                                         2.6 MB/s | 1.3 MB  00:00:00     
    Running transaction check
    Running transaction test
    Transaction test succeeded
    Running transaction
      Installing : libbasicobjects-0.1.1-32.el7.x86_64                                                                                                                                                                                                                       1/15 
      Installing : libref_array-0.1.5-32.el7.x86_64                                                                                                                                                                                                                          2/15 
      Installing : libcollection-0.7.0-32.el7.x86_64                                                                                                                                                                                                                         3/15 
      Installing : libtirpc-0.2.4-0.16.el7.x86_64                                                                                                                                                                                                                            4/15 
      Installing : rpcbind-0.2.0-48.el7.x86_64                                                                                                                                                                                                                               5/15 
      Installing : 1:quota-nls-4.01-19.el7.noarch                                                                                                                                                                                                                            6/15 
      Installing : tcp_wrappers-7.6-77.el7.x86_64                                                                                                                                                                                                                            7/15 
      Installing : 1:quota-4.01-19.el7.x86_64                                                                                                                                                                                                                                8/15 
      Installing : keyutils-1.5.8-3.el7.x86_64                                                                                                                                                                                                                               9/15 
      Installing : libnfsidmap-0.25-19.el7.x86_64                                                                                                                                                                                                                           10/15 
      Installing : libpath_utils-0.2.1-32.el7.x86_64                                                                                                                                                                                                                        11/15 
      Installing : libini_config-1.3.1-32.el7.x86_64                                                                                                                                                                                                                        12/15 
      Installing : libverto-libevent-0.2.5-4.el7.x86_64                                                                                                                                                                                                                     13/15 
      Installing : gssproxy-0.7.0-26.el7.x86_64                                                                                                                                                                                                                             14/15 
      Installing : 1:nfs-utils-1.3.0-0.65.el7.x86_64                                                                                                                                                                                                                        15/15 
      Verifying  : libtirpc-0.2.4-0.16.el7.x86_64                                                                                                                                                                                                                            1/15 
      Verifying  : libverto-libevent-0.2.5-4.el7.x86_64                                                                                                                                                                                                                      2/15 
      Verifying  : 1:quota-4.01-19.el7.x86_64                                                                                                                                                                                                                                3/15 
      Verifying  : gssproxy-0.7.0-26.el7.x86_64                                                                                                                                                                                                                              4/15 
      Verifying  : libpath_utils-0.2.1-32.el7.x86_64                                                                                                                                                                                                                         5/15 
      Verifying  : libnfsidmap-0.25-19.el7.x86_64                                                                                                                                                                                                                            6/15 
      Verifying  : keyutils-1.5.8-3.el7.x86_64                                                                                                                                                                                                                               7/15 
      Verifying  : 1:nfs-utils-1.3.0-0.65.el7.x86_64                                                                                                                                                                                                                         8/15 
      Verifying  : tcp_wrappers-7.6-77.el7.x86_64                                                                                                                                                                                                                            9/15 
      Verifying  : libcollection-0.7.0-32.el7.x86_64                                                                                                                                                                                                                        10/15 
      Verifying  : libref_array-0.1.5-32.el7.x86_64                                                                                                                                                                                                                         11/15 
      Verifying  : libbasicobjects-0.1.1-32.el7.x86_64                                                                                                                                                                                                                      12/15 
      Verifying  : rpcbind-0.2.0-48.el7.x86_64                                                                                                                                                                                                                              13/15 
      Verifying  : libini_config-1.3.1-32.el7.x86_64                                                                                                                                                                                                                        14/15 
      Verifying  : 1:quota-nls-4.01-19.el7.noarch                                                                                                                                                                                                                           15/15 
    
    Installed:
      nfs-utils.x86_64 1:1.3.0-0.65.el7                                                                                                                                                                                                                                           
    
    Dependency Installed:
      gssproxy.x86_64 0:0.7.0-26.el7        keyutils.x86_64 0:1.5.8-3.el7       libbasicobjects.x86_64 0:0.1.1-32.el7     libcollection.x86_64 0:0.7.0-32.el7    libini_config.x86_64 0:1.3.1-32.el7    libnfsidmap.x86_64 0:0.25-19.el7    libpath_utils.x86_64 0:0.2.1-32.el7   
      libref_array.x86_64 0:0.1.5-32.el7    libtirpc.x86_64 0:0.2.4-0.16.el7    libverto-libevent.x86_64 0:0.2.5-4.el7    quota.x86_64 1:4.01-19.el7             quota-nls.noarch 1:4.01-19.el7         rpcbind.x86_64 0:0.2.0-48.el7       tcp_wrappers.x86_64 0:7.6-77.el7      
    
    Complete!
    [root@node203.yinzhengjie.org.cn ~]# 
    [root@node203.yinzhengjie.org.cn ~]# yum -y install nfs-utils
    [root@master200.yinzhengjie.org.cn ~]# mount -t nfs 172.200.1.110:/yinzhengjie/data/volume1 /mnt      #在K8S node节点测试NFS服务是否可以成功挂载
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# mount | grep mnt
    172.200.1.110:/yinzhengjie/data/volume1 on /mnt type nfs4 (rw,relatime,vers=4.1,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=172.200.1.200,local_lock=none,addr=172.200.1.110)
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# df -h | grep mnt
    172.200.1.110:/yinzhengjie/data/volume1  1.6T  416M  1.6T   1% /mnt
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# mount -t nfs 172.200.1.110:/yinzhengjie/data/volume1 /mnt      #在K8S node节点测试NFS服务是否可以成功挂载,即验证可用性。

    3>.创建nfs对应的pv

    [root@master200.yinzhengjie.org.cn ~]# vim /yinzhengjie/data/k8s/manifests/basic/statefulset/pv-nfs-demo.yaml
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# cat /yinzhengjie/data/k8s/manifests/basic/statefulset/pv-nfs-demo.yaml
    apiVersion: v1
    kind: PersistentVolume
    metadata:
      name: pv-nfs-v3
      labels:
        storsys: nfs
    spec:
      accessModes: ["ReadWriteOnce","ReadOnlyMany","ReadWriteMany"]
      capacity:
        storage: 5Gi
      volumeMode: Filesystem
      persistentVolumeReclaimPolicy: Retain
      nfs:
        server: 172.200.1.110
        path: /yinzhengjie/data/volume3
    ---
    apiVersion: v1
    kind: PersistentVolume
    metadata:
      name: pv-nfs-v4
      labels:
        storsys: nfs
    spec:
      accessModes: ["ReadWriteOnce","ReadOnlyMany","ReadWriteMany"]
      capacity:
        storage: 5Gi
      volumeMode: Filesystem
      persistentVolumeReclaimPolicy: Retain
      nfs:
        server: 172.200.1.110
        path: /yinzhengjie/data/volume4
    ---
    apiVersion: v1
    kind: PersistentVolume
    metadata:
      name: pv-nfs-v5
      labels:
        storsys: nfs
    spec:
      accessModes: ["ReadWriteOnce","ReadOnlyMany","ReadWriteMany"]
      capacity:
        storage: 5Gi
      volumeMode: Filesystem
      persistentVolumeReclaimPolicy: Retain
      nfs:
        server: 172.200.1.110
        path: /yinzhengjie/data/volume5
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# vim /yinzhengjie/data/k8s/manifests/basic/statefulset/pv-nfs-demo.yaml
    [root@master200.yinzhengjie.org.cn ~]# kubectl apply -f /yinzhengjie/data/k8s/manifests/basic/statefulset/pv-nfs-demo.yaml
    persistentvolume/pv-nfs-v3 created
    persistentvolume/pv-nfs-v4 created
    persistentvolume/pv-nfs-v5 created
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl  get pv
    NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
    pv-nfs-v3   5Gi        RWO,ROX,RWX    Retain           Available                                   12s
    pv-nfs-v4   5Gi        RWO,ROX,RWX    Retain           Available                                   11s
    pv-nfs-v5   5Gi        RWO,ROX,RWX    Retain           Available                                   11s
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl apply -f /yinzhengjie/data/k8s/manifests/basic/statefulset/pv-nfs-demo.yaml
    [root@master200.yinzhengjie.org.cn ~]# kubectl  get pv -o wide
    NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE   VOLUMEMODE
    pv-nfs-v3   5Gi        RWO,ROX,RWX    Retain           Available                                   16s   Filesystem
    pv-nfs-v4   5Gi        RWO,ROX,RWX    Retain           Available                                   15s   Filesystem
    pv-nfs-v5   5Gi        RWO,ROX,RWX    Retain           Available                                   15s   Filesystem
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl get pv -o wide

    4>.基于statefulset部署Pod

    [root@master200.yinzhengjie.org.cn ~]# kubectl  get pv -o wide
    NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE   VOLUMEMODE
    pv-nfs-v3   5Gi        RWO,ROX,RWX    Retain           Available                                   16s   Filesystem
    pv-nfs-v4   5Gi        RWO,ROX,RWX    Retain           Available                                   15s   Filesystem
    pv-nfs-v5   5Gi        RWO,ROX,RWX    Retain           Available                                   15s   Filesystem
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl get pv -o wide
    [root@master200.yinzhengjie.org.cn ~]# vim /yinzhengjie/data/k8s/manifests/basic/statefulset/statefulset-demo.yaml 
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# cat /yinzhengjie/data/k8s/manifests/basic/statefulset/statefulset-demo.yaml 
    apiVersion: v1
    kind: Namespace
    metadata:
      name: yinzhengjie-sts
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: myapp-sts-svc
      namespace: yinzhengjie-sts
      labels:
        app: myapp
        controller: mystatefulset
    spec:
      ports:
      - port: 80
        name: web
      clusterIP: None
      selector:
        app: myapp-pod
    ---
    apiVersion: apps/v1
    kind: StatefulSet
    metadata:
      name: statefulset-demo
      namespace: yinzhengjie-sts
    spec:
      selector:
        matchLabels:
          app: myapp-pod
          controller: mystatefulset
      serviceName: "myapp-sts-svc"
      replicas: 2
      template:
        metadata:
          namespace: yinzhengjie-sts
          labels:
            app: myapp-pod 
            controller: mystatefulset
        spec:
          terminationGracePeriodSeconds: 10
          containers:
          - name: myapp
            image: ikubernetes/myapp:v1
            ports:
            - containerPort: 80
              name: web
            volumeMounts:
            - name: myapp-pvc
              mountPath: /usr/share/nginx/html
      volumeClaimTemplates:
      - metadata:
          name: myapp-pvc
          namespace: yinzhengjie-sts
        spec:
          accessModes: [ "ReadWriteOnce" ]
          resources:
            requests:
              storage: 2Gi
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# vim /yinzhengjie/data/k8s/manifests/basic/statefulset/statefulset-demo.yaml
    [root@master200.yinzhengjie.org.cn ~]# kubectl apply -f /yinzhengjie/data/k8s/manifests/basic/statefulset/statefulset-demo.yaml 
    namespace/yinzhengjie-sts created
    service/myapp-sts-svc created
    statefulset.apps/statefulset-demo created
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl apply -f /yinzhengjie/data/k8s/manifests/basic/statefulset/statefulset-demo.yaml
    [root@master200.yinzhengjie.org.cn ~]# kubectl get all -n yinzhengjie-sts
    NAME                     READY   STATUS    RESTARTS   AGE
    pod/statefulset-demo-0   1/1     Running   0          56s
    pod/statefulset-demo-1   1/1     Running   0          55s
    
    NAME                    TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
    service/myapp-sts-svc   ClusterIP   None         <none>        80/TCP    56s
    
    NAME                                READY   AGE
    statefulset.apps/statefulset-demo   2/2     56s
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl get all -n yinzhengjie-sts
    [root@master200.yinzhengjie.org.cn ~]# kubectl get pvc -n yinzhengjie-sts
    NAME                           STATUS   VOLUME      CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    myapp-pvc-statefulset-demo-0   Bound    pv-nfs-v5   5Gi        RWO,ROX,RWX                   73s
    myapp-pvc-statefulset-demo-1   Bound    pv-nfs-v3   5Gi        RWO,ROX,RWX                   72s
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl get pvc -n yinzhengjie-sts
    [root@master200.yinzhengjie.org.cn ~]# kubectl get pv -o wide
    NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                                          STORAGECLASS   REASON   AGE     VOLUMEMODE
    pv-nfs-v3   5Gi        RWO,ROX,RWX    Retain           Bound       yinzhengjie-sts/myapp-pvc-statefulset-demo-1                           6m34s   Filesystem
    pv-nfs-v4   5Gi        RWO,ROX,RWX    Retain           Available                                                                          6m33s   Filesystem
    pv-nfs-v5   5Gi        RWO,ROX,RWX    Retain           Bound       yinzhengjie-sts/myapp-pvc-statefulset-demo-0                           6m33s   Filesystem
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl get pv -o wide
    [root@master200.yinzhengjie.org.cn ~]# kubectl get svc -n yinzhengjie-sts
    NAME            TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
    myapp-sts-svc   ClusterIP   None         <none>        80/TCP    4m40s
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl get svc -n yinzhengjie-sts
    [root@master200.yinzhengjie.org.cn ~]# kubectl get pods -n yinzhengjie-sts -o wide
    NAME                 READY   STATUS    RESTARTS   AGE     IP            NODE                         NOMINATED NODE   READINESS GATES
    statefulset-demo-0   1/1     Running   0          5m21s   10.244.1.35   node201.yinzhengjie.org.cn   <none>           <none>
    statefulset-demo-1   1/1     Running   0          5m20s   10.244.3.25   node203.yinzhengjie.org.cn   <none>           <none>
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl get pods -n yinzhengjie-sts -o wide

    5>.扩容Pod的副本数

    [root@master200.yinzhengjie.org.cn ~]# kubectl get pv
    NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                                          STORAGECLASS   REASON   AGE
    pv-nfs-v3   5Gi        RWO,ROX,RWX    Retain           Bound       yinzhengjie-sts/myapp-pvc-statefulset-demo-1                           18m
    pv-nfs-v4   5Gi        RWO,ROX,RWX    Retain           Available                                                                          18m
    pv-nfs-v5   5Gi        RWO,ROX,RWX    Retain           Bound       yinzhengjie-sts/myapp-pvc-statefulset-demo-0                           18m
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl get all -o wide -n yinzhengjie-sts 
    NAME                     READY   STATUS    RESTARTS   AGE   IP            NODE                         NOMINATED NODE   READINESS GATES
    pod/statefulset-demo-0   1/1     Running   0          13m   10.244.1.35   node201.yinzhengjie.org.cn   <none>           <none>
    pod/statefulset-demo-1   1/1     Running   0          13m   10.244.3.25   node203.yinzhengjie.org.cn   <none>           <none>
    
    NAME                    TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE   SELECTOR
    service/myapp-sts-svc   ClusterIP   None         <none>        80/TCP    13m   app=myapp-pod
    
    NAME                                READY   AGE   CONTAINERS   IMAGES
    statefulset.apps/statefulset-demo   2/2     13m   myapp        ikubernetes/myapp:v1
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl scale sts statefulset-demo --replicas=3 -n yinzhengjie-sts
    statefulset.apps/statefulset-demo scaled
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl get pv
    NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                          STORAGECLASS   REASON   AGE
    pv-nfs-v3   5Gi        RWO,ROX,RWX    Retain           Bound    yinzhengjie-sts/myapp-pvc-statefulset-demo-1                           19m
    pv-nfs-v4   5Gi        RWO,ROX,RWX    Retain           Bound    yinzhengjie-sts/myapp-pvc-statefulset-demo-2                           19m
    pv-nfs-v5   5Gi        RWO,ROX,RWX    Retain           Bound    yinzhengjie-sts/myapp-pvc-statefulset-demo-0                           19m
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl get all -o wide -n yinzhengjie-sts 
    NAME                     READY   STATUS    RESTARTS   AGE   IP            NODE                         NOMINATED NODE   READINESS GATES
    pod/statefulset-demo-0   1/1     Running   0          15m   10.244.1.35   node201.yinzhengjie.org.cn   <none>           <none>
    pod/statefulset-demo-1   1/1     Running   0          14m   10.244.3.25   node203.yinzhengjie.org.cn   <none>           <none>
    pod/statefulset-demo-2   1/1     Running   0          14s   10.244.2.30   node202.yinzhengjie.org.cn   <none>           <none>
    
    NAME                    TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE   SELECTOR
    service/myapp-sts-svc   ClusterIP   None         <none>        80/TCP    15m   app=myapp-pod
    
    NAME                                READY   AGE   CONTAINERS   IMAGES
    statefulset.apps/statefulset-demo   3/3     15m   myapp        ikubernetes/myapp:v1
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl scale sts statefulset-demo --replicas=3 -n yinzhengjie-sts

    6>.升级Pod版本

    [root@master200.yinzhengjie.org.cn ~]# kubectl get all -o wide -n yinzhengjie-sts 
    NAME                     READY   STATUS    RESTARTS   AGE     IP            NODE                         NOMINATED NODE   READINESS GATES
    pod/statefulset-demo-0   1/1     Running   0          19m     10.244.1.35   node201.yinzhengjie.org.cn   <none>           <none>
    pod/statefulset-demo-1   1/1     Running   0          19m     10.244.3.25   node203.yinzhengjie.org.cn   <none>           <none>
    pod/statefulset-demo-2   1/1     Running   0          4m29s   10.244.2.30   node202.yinzhengjie.org.cn   <none>           <none>
    
    NAME                    TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE   SELECTOR
    service/myapp-sts-svc   ClusterIP   None         <none>        80/TCP    19m   app=myapp-pod
    
    NAME                                READY   AGE   CONTAINERS   IMAGES
    statefulset.apps/statefulset-demo   3/3     19m   myapp        ikubernetes/myapp:v1
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl set image sts statefulset-demo myapp=ikubernetes/myapp:v2 -n yinzhengjie-sts
    statefulset.apps/statefulset-demo image updated
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl get all -o wide -n yinzhengjie-sts 
    NAME                     READY   STATUS    RESTARTS   AGE    IP            NODE                         NOMINATED NODE   READINESS GATES
    pod/statefulset-demo-0   1/1     Running   0          69s    10.244.1.36   node201.yinzhengjie.org.cn   <none>           <none>
    pod/statefulset-demo-1   1/1     Running   0          90s    10.244.3.26   node203.yinzhengjie.org.cn   <none>           <none>
    pod/statefulset-demo-2   1/1     Running   0          103s   10.244.2.31   node202.yinzhengjie.org.cn   <none>           <none>
    
    NAME                    TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE   SELECTOR
    service/myapp-sts-svc   ClusterIP   None         <none>        80/TCP    22m   app=myapp-pod
    
    NAME                                READY   AGE   CONTAINERS   IMAGES
    statefulset.apps/statefulset-demo   3/3     22m   myapp        ikubernetes/myapp:v2
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl set image sts statefulset-demo myapp=ikubernetes/myapp:v2 -n yinzhengjie-sts
    [root@master200.yinzhengjie.org.cn ~]# kubectl describe sts statefulset-demo -n yinzhengjie-sts
    Name:               statefulset-demo
    Namespace:          yinzhengjie-sts
    CreationTimestamp:  Wed, 12 Feb 2020 10:40:46 +0800
    Selector:           app=myapp-pod,controller=mystatefulset
    Labels:             <none>
    Annotations:        kubectl.kubernetes.io/last-applied-configuration:
                          {"apiVersion":"apps/v1","kind":"StatefulSet","metadata":{"annotations":{},"name":"statefulset-demo","namespace":"yinzhengjie-sts"},"spec":...
    Replicas:           3 desired | 3 total
    Update Strategy:    RollingUpdate
      Partition:        824644603528
    Pods Status:        3 Running / 0 Waiting / 0 Succeeded / 0 Failed
    Pod Template:
      Labels:  app=myapp-pod
               controller=mystatefulset
      Containers:
       myapp:
        Image:        ikubernetes/myapp:v2
        Port:         80/TCP
        Host Port:    0/TCP
        Environment:  <none>
        Mounts:
          /usr/share/nginx/html from myapp-pvc (rw)
      Volumes:  <none>
    Volume Claims:
      Name:          myapp-pvc
      StorageClass:  
      Labels:        <none>
      Annotations:   <none>
      Capacity:      2Gi
      Access Modes:  [ReadWriteOnce]
    Events:
      Type    Reason            Age                  From                    Message
      ----    ------            ----                 ----                    -------
      Normal  SuccessfulCreate  25m                  statefulset-controller  create Claim myapp-pvc-statefulset-demo-0 Pod statefulset-demo-0 in StatefulSet statefulset-demo success
      Normal  SuccessfulCreate  25m                  statefulset-controller  create Claim myapp-pvc-statefulset-demo-1 Pod statefulset-demo-1 in StatefulSet statefulset-demo success
      Normal  SuccessfulCreate  10m                  statefulset-controller  create Claim myapp-pvc-statefulset-demo-2 Pod statefulset-demo-2 in StatefulSet statefulset-demo success
      Normal  SuccessfulDelete  4m45s                statefulset-controller  delete Pod statefulset-demo-2 in StatefulSet statefulset-demo successful
      Normal  SuccessfulCreate  4m33s (x2 over 10m)  statefulset-controller  create Pod statefulset-demo-2 in StatefulSet statefulset-demo successful
      Normal  SuccessfulDelete  4m23s                statefulset-controller  delete Pod statefulset-demo-1 in StatefulSet statefulset-demo successful
      Normal  SuccessfulCreate  4m20s (x2 over 25m)  statefulset-controller  create Pod statefulset-demo-1 in StatefulSet statefulset-demo successful
      Normal  SuccessfulDelete  4m10s                statefulset-controller  delete Pod statefulset-demo-0 in StatefulSet statefulset-demo successful
      Normal  SuccessfulCreate  3m59s (x2 over 25m)  statefulset-controller  create Pod statefulset-demo-0 in StatefulSet statefulset-demo successful
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl describe sts statefulset-demo -n yinzhengjie-sts

    7>.缩容Pod的副本数(缩容时pvc对应pv数据并不会被删除)

    [root@master200.yinzhengjie.org.cn ~]# kubectl get pods -n yinzhengjie-sts
    NAME                 READY   STATUS    RESTARTS   AGE
    statefulset-demo-0   1/1     Running   0          19m
    statefulset-demo-1   1/1     Running   0          19m
    statefulset-demo-2   1/1     Running   0          5s
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl exec -it -n yinzhengjie-sts statefulset-demo-2 -- sh
    / # 
    / # cd /usr/share/nginx/html/        #进入到存储卷挂载路径
    /usr/share/nginx/html # 
    /usr/share/nginx/html # ls
    /usr/share/nginx/html # 
    /usr/share/nginx/html # echo "https://www.cnblogs.com/yinzhengjie/" > blog.txt      #在存储卷路径写入测试数据。
    /usr/share/nginx/html # 
    /usr/share/nginx/html # ls -l
    total 4
    -rw-r--r--    1 root     root            37 Feb 12  2020 blog.txt
    /usr/share/nginx/html # 
    /usr/share/nginx/html # exit
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# 
    /usr/share/nginx/html # echo "https://www.cnblogs.com/yinzhengjie/" > blog.txt      #在存储卷路径写入测试数据。
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl get pods -n yinzhengjie-sts
    NAME                 READY   STATUS    RESTARTS   AGE
    statefulset-demo-0   1/1     Running   0          19m
    statefulset-demo-1   1/1     Running   0          19m
    statefulset-demo-2   1/1     Running   0          5s
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl exec -it -n yinzhengjie-sts statefulset-demo-2 -- sh
    / # 
    / # cd /usr/share/nginx/html/
    /usr/share/nginx/html # 
    /usr/share/nginx/html # ls
    /usr/share/nginx/html # 
    /usr/share/nginx/html # echo "https://www.cnblogs.com/yinzhengjie/" > blog.txt
    /usr/share/nginx/html # 
    /usr/share/nginx/html # ls -l
    total 4
    -rw-r--r--    1 root     root            37 Feb 12  2020 blog.txt
    /usr/share/nginx/html # 
    /usr/share/nginx/html # exit
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl scale sts statefulset-demo --replicas=2 -n yinzhengjie-sts
    statefulset.apps/statefulset-demo scaled
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl get pods -n yinzhengjie-sts
    NAME                 READY   STATUS    RESTARTS   AGE
    statefulset-demo-0   1/1     Running   0          24m
    statefulset-demo-1   1/1     Running   0          24m
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl scale sts statefulset-demo --replicas=2 -n yinzhengjie-sts      #对Pod的副本进行缩容

    8>.删除statefulset

    [root@master200.yinzhengjie.org.cn ~]# kubectl get sts -n yinzhengjie-sts
    NAME               READY   AGE
    statefulset-demo   3/3     55m
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl delete sts statefulset-demo  -n yinzhengjie-sts
    statefulset.apps "statefulset-demo" deleted
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl get sts -n yinzhengjie-sts
    No resources found in yinzhengjie-sts namespace.
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl delete sts statefulset-demo -n yinzhengjie-sts

    三.使用StatefulSet部署etcd服务

      StatefulSet(简称sts)可以做到简单的部署,扩容等功能,但是要涉及到比较复杂的功能比如数据备份等功能建议大家使用Operator。

    1>.创建etcd的service资源

    [root@master200.yinzhengjie.org.cn ~]# vim /yinzhengjie/data/k8s/manifests/basic/statefulset/etcd-services.yaml 
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# cat /yinzhengjie/data/k8s/manifests/basic/statefulset/etcd-services.yaml 
    apiVersion: v1
    kind: Service
    metadata:
      name: etcd
      labels:
        app: etcd
      annotations:
        # Create endpoints also if the related pod isn't ready
        service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
    spec:
      ports:
      - port: 2379
        name: client
      - port: 2380
        name: peer
      clusterIP: None
      selector:
        app: etcd-member
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: etcd-client
      labels:
        app: etcd
    spec:
      ports:
      - name: etcd-client
        port: 2379
        protocol: TCP
        targetPort: 2379
      selector:
        app: etcd-member
      type: NodePort
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# vim /yinzhengjie/data/k8s/manifests/basic/statefulset/etcd-services.yaml
    [root@master200.yinzhengjie.org.cn ~]# kubectl apply -f /yinzhengjie/data/k8s/manifests/basic/statefulset/etcd-services.yaml 
    service/etcd created
    service/etcd-client created
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl get svc 
    NAME          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
    etcd          ClusterIP   None             <none>        2379/TCP,2380/TCP   14s
    etcd-client   NodePort    10.107.187.160   <none>        2379:32360/TCP      14s
    kubernetes    ClusterIP   10.96.0.1        <none>        443/TCP             7d16h
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl apply -f /yinzhengjie/data/k8s/manifests/basic/statefulset/etcd-services.yaml
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl get svc -o wide
    NAME          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE     SELECTOR
    etcd          ClusterIP   None             <none>        2379/TCP,2380/TCP   31s     app=etcd-member
    etcd-client   NodePort    10.107.187.160   <none>        2379:32360/TCP      31s     app=etcd-member
    kubernetes    ClusterIP   10.96.0.1        <none>        443/TCP             7d16h   <none>
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl get svc -o wide

    2>.确认nfs有足够的空间及挂载目录并创建pv

    [root@test110.yinzhengjie.org.cn ~]# vim /etc/exports
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# cat /etc/exports
    /yinzhengjie/data/volume1    172.200.0.0/21(rw,sync,no_root_squash)
    /yinzhengjie/data/volume2    172.200.0.0/21(rw,sync,no_root_squash)
    /yinzhengjie/data/volume3    172.200.0.0/21(rw,sync,no_root_squash)
    /yinzhengjie/data/volume4    172.200.0.0/21(rw,sync,no_root_squash)
    /yinzhengjie/data/volume5    172.200.0.0/21(rw,sync,no_root_squash)
    /yinzhengjie/data/volume6    172.200.0.0/21(rw,sync,no_root_squash)
    /yinzhengjie/data/volume7    172.200.0.0/21(rw,sync,no_root_squash)
    /yinzhengjie/data/volume8    172.200.0.0/21(rw,sync,no_root_squash)
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# vim /etc/exports
    [root@test110.yinzhengjie.org.cn ~]# mkdir /yinzhengjie/data/volume{6,7,8}
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# ll /yinzhengjie/data/
    total 0
    drwxr-xr-x 2 polkitd root 22 Feb 10 06:40 volume1
    drwxr-xr-x 2 polkitd root  6 Feb 10 20:04 volume2
    drwxr-xr-x 2 root    root  6 Feb 10 06:04 volume3
    drwxr-xr-x 2 root    root 22 Feb 13 03:11 volume4
    drwxr-xr-x 2 root    root  6 Feb 10 06:04 volume5
    drwxr-xr-x 2 root    root  6 Feb 13 03:42 volume6
    drwxr-xr-x 2 root    root  6 Feb 13 03:42 volume7
    drwxr-xr-x 2 root    root  6 Feb 13 03:42 volume8
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# exportfs -rav
    exporting 172.200.0.0/21:/yinzhengjie/data/volume8
    exporting 172.200.0.0/21:/yinzhengjie/data/volume7
    exporting 172.200.0.0/21:/yinzhengjie/data/volume6
    exporting 172.200.0.0/21:/yinzhengjie/data/volume5
    exporting 172.200.0.0/21:/yinzhengjie/data/volume4
    exporting 172.200.0.0/21:/yinzhengjie/data/volume3
    exporting 172.200.0.0/21:/yinzhengjie/data/volume2
    exporting 172.200.0.0/21:/yinzhengjie/data/volume1
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# 
    [root@test110.yinzhengjie.org.cn ~]# exportfs -rav
    [root@master200.yinzhengjie.org.cn ~]# vim /yinzhengjie/data/k8s/manifests/basic/statefulset/pv-nfs-demo.yaml 
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# cat /yinzhengjie/data/k8s/manifests/basic/statefulset/pv-nfs-demo.yaml 
    apiVersion: v1
    kind: PersistentVolume
    metadata:
      name: pv-nfs-v3
      labels:
        storsys: nfs
    spec:
      accessModes: ["ReadWriteOnce","ReadOnlyMany","ReadWriteMany"]
      capacity:
        storage: 5Gi
      volumeMode: Filesystem
      persistentVolumeReclaimPolicy: Retain
      nfs:
        server: 172.200.1.110
        path: /yinzhengjie/data/volume3
    ---
    apiVersion: v1
    kind: PersistentVolume
    metadata:
      name: pv-nfs-v4
      labels:
        storsys: nfs
    spec:
      accessModes: ["ReadWriteOnce","ReadOnlyMany","ReadWriteMany"]
      capacity:
        storage: 5Gi
      volumeMode: Filesystem
      persistentVolumeReclaimPolicy: Retain
      nfs:
        server: 172.200.1.110
        path: /yinzhengjie/data/volume4
    ---
    apiVersion: v1
    kind: PersistentVolume
    metadata:
      name: pv-nfs-v5
      labels:
        storsys: nfs
    spec:
      accessModes: ["ReadWriteOnce","ReadOnlyMany","ReadWriteMany"]
      capacity:
        storage: 5Gi
      volumeMode: Filesystem
      persistentVolumeReclaimPolicy: Retain
      nfs:
        server: 172.200.1.110
        path: /yinzhengjie/data/volume5
    ---
    apiVersion: v1
    kind: PersistentVolume
    metadata:
      name: pv-nfs-v6
      labels:
        storsys: nfs
    spec:
      accessModes: ["ReadWriteOnce","ReadOnlyMany","ReadWriteMany"]
      capacity:
        storage: 5Gi
      volumeMode: Filesystem
      persistentVolumeReclaimPolicy: Retain
      nfs:
        server: 172.200.1.110
        path: /yinzhengjie/data/volume6
    ---
    apiVersion: v1
    kind: PersistentVolume
    metadata:
      name: pv-nfs-v7
      labels:
        storsys: nfs
    spec:
      accessModes: ["ReadWriteOnce","ReadOnlyMany","ReadWriteMany"]
      capacity:
        storage: 5Gi
      volumeMode: Filesystem
      persistentVolumeReclaimPolicy: Retain
      nfs:
        server: 172.200.1.110
        path: /yinzhengjie/data/volume7
    ---
    apiVersion: v1
    kind: PersistentVolume
    metadata:
      name: pv-nfs-v8
      labels:
        storsys: nfs
    spec:
      accessModes: ["ReadWriteOnce","ReadOnlyMany","ReadWriteMany"]
      capacity:
        storage: 5Gi
      volumeMode: Filesystem
      persistentVolumeReclaimPolicy: Retain
      nfs:
        server: 172.200.1.110
        path: /yinzhengjie/data/volume8
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# vim /yinzhengjie/data/k8s/manifests/basic/statefulset/pv-nfs-demo.yaml
    [root@master200.yinzhengjie.org.cn ~]# kubectl get pv
    NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                          STORAGECLASS   REASON   AGE
    pv-nfs-v3   5Gi        RWO,ROX,RWX    Retain           Bound    yinzhengjie-sts/myapp-pvc-statefulset-demo-1                           23m
    pv-nfs-v4   5Gi        RWO,ROX,RWX    Retain           Bound    yinzhengjie-sts/myapp-pvc-statefulset-demo-2                           23m
    pv-nfs-v5   5Gi        RWO,ROX,RWX    Retain           Bound    yinzhengjie-sts/myapp-pvc-statefulset-demo-0                           23m
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl apply -f /yinzhengjie/data/k8s/manifests/basic/statefulset/pv-nfs-demo.yaml 
    persistentvolume/pv-nfs-v3 unchanged
    persistentvolume/pv-nfs-v4 unchanged
    persistentvolume/pv-nfs-v5 unchanged
    persistentvolume/pv-nfs-v6 created
    persistentvolume/pv-nfs-v7 created
    persistentvolume/pv-nfs-v8 created
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl get pv
    NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                                          STORAGECLASS   REASON   AGE
    pv-nfs-v3   5Gi        RWO,ROX,RWX    Retain           Bound       yinzhengjie-sts/myapp-pvc-statefulset-demo-1                           3h52m
    pv-nfs-v4   5Gi        RWO,ROX,RWX    Retain           Bound       yinzhengjie-sts/myapp-pvc-statefulset-demo-2                           3h52m
    pv-nfs-v5   5Gi        RWO,ROX,RWX    Retain           Bound       yinzhengjie-sts/myapp-pvc-statefulset-demo-0                           3h52m
    pv-nfs-v6   5Gi        RWO,ROX,RWX    Retain           Available                                                                          11s
    pv-nfs-v7   5Gi        RWO,ROX,RWX    Retain           Available                                                                          11s
    pv-nfs-v8   5Gi        RWO,ROX,RWX    Retain           Available                                                                          11s
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# kubectl apply -f /yinzhengjie/data/k8s/manifests/basic/statefulset/pv-nfs-demo.yaml

    3>.创建etcd的实例StatefulSet配置文件

    [root@master200.yinzhengjie.org.cn ~]# cat /yinzhengjie/data/k8s/manifests/basic/statefulset/etcd-statefulset.yaml 
    apiVersion: apps/v1
    kind: StatefulSet
    metadata:
      name: etcd
      labels:
        app: etcd
    spec:
      serviceName: etcd
      # changing replicas value will require a manual etcdctl member remove/add
      #   # command (remove before decreasing and add after increasing)
      replicas: 3
      selector:
        matchLabels:
          app: etcd-member
      template:
        metadata:
          name: etcd
          labels:
            app: etcd-member
        spec:
          containers:
          - name: etcd
            image: "quay.io/coreos/etcd:v3.2.16"
            ports:
            - containerPort: 2379
              name: client
            - containerPort: 2380
              name: peer
            env:
            - name: CLUSTER_SIZE
              value: "3"
            - name: SET_NAME
              value: "etcd"
            volumeMounts:
            - name: data
              mountPath: /var/run/etcd
            command:
              - "/bin/sh"
              - "-ecx"
              - |
                IP=$(hostname -i)
                PEERS=""
                for i in $(seq 0 $((${CLUSTER_SIZE} - 1))); do
                    PEERS="${PEERS}${PEERS:+,}${SET_NAME}-${i}=http://${SET_NAME}-${i}.${SET_NAME}:2380"
                done
                # start etcd. If cluster is already initialized the `--initial-*` options will be ignored.
                exec etcd --name ${HOSTNAME} 
                  --listen-peer-urls http://${IP}:2380 
                  --listen-client-urls http://${IP}:2379,http://127.0.0.1:2379 
                  --advertise-client-urls http://${HOSTNAME}.${SET_NAME}:2379 
                  --initial-advertise-peer-urls http://${HOSTNAME}.${SET_NAME}:2380 
                  --initial-cluster-token etcd-cluster-1 
                  --initial-cluster ${PEERS} 
                  --initial-cluster-state new 
                  --data-dir /var/run/etcd/default.etcd
      volumeClaimTemplates:
      - metadata:
          name: data
        spec:
          accessModes:
            - "ReadWriteOnce"
          resources:
            requests:
              storage: 1Gi
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# cat /yinzhengjie/data/k8s/manifests/basic/statefulset/etcd-statefulset.yaml

    四.使用StatefulSet部署zookeeper服务参考配置文件

    [root@master200.yinzhengjie.org.cn ~]# cat /yinzhengjie/data/k8s/manifests/basic/statefulset/zk-sts.yaml 
    apiVersion: v1
    kind: Service
    metadata:
      name: zk-hs
      labels:
        app: zk
    spec:
      ports:
      - port: 2888
        name: server
      - port: 3888
        name: leader-election
      clusterIP: None
      selector:
        app: zk
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: zk-cs
      labels:
        app: zk
    spec:
      ports:
      - port: 2181
        name: client
      selector:
        app: zk
    ---
    apiVersion: policy/v1beta1
    kind: PodDisruptionBudget
    metadata:
      name: zk-pdb
    spec:
      selector:
        matchLabels:
          app: zk
      maxUnavailable: 1
    ---
    apiVersion: apps/v1
    kind: StatefulSet
    metadata:
      name: zk
    spec:
      selector:
        matchLabels:
          app: zk
      serviceName: zk-hs
      replicas: 3
      updateStrategy:
        type: RollingUpdate
      podManagementPolicy: Parallel
      template:
        metadata:
          labels:
            app: zk
        spec:
          affinity:
            podAntiAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                - labelSelector:
                    matchExpressions:
                      - key: "app"
                        operator: In
                        values:
                        - zk-hs
                  topologyKey: "kubernetes.io/hostname"
          containers:
          - name: kubernetes-zookeeper
            image: gcr.io/google-containers/kubernetes-zookeeper:1.0-3.4.10
            resources:
              requests:
                memory: "1Gi"
                cpu: "0.5"
            ports:
            - containerPort: 2181
              name: client
            - containerPort: 2888
              name: server
            - containerPort: 3888
              name: leader-election
            command:
            - sh
            - -c
            - "start-zookeeper 
              --servers=3 
              --data_dir=/var/lib/zookeeper/data 
              --data_log_dir=/var/lib/zookeeper/data/log 
              --conf_dir=/opt/zookeeper/conf 
              --client_port=2181 
              --election_port=3888 
              --server_port=2888 
              --tick_time=2000 
              --init_limit=10 
              --sync_limit=5 
              --heap=512M 
              --max_client_cnxns=60 
              --snap_retain_count=3 
              --purge_interval=12 
              --max_session_timeout=40000 
              --min_session_timeout=4000 
              --log_level=INFO"
            readinessProbe:
              exec:
                command:
                - sh
                - -c
                - "zookeeper-ready 2181"
              initialDelaySeconds: 10
              timeoutSeconds: 5
            livenessProbe:
              exec:
                command:
                - sh
                - -c
                - "zookeeper-ready 2181"
              initialDelaySeconds: 10
              timeoutSeconds: 5
            volumeMounts:
            - name: data
              mountPath: /var/lib/zookeeper
          securityContext:
            runAsUser: 1000
            fsGroup: 1000
      volumeClaimTemplates:
      - metadata:
          name: data
        spec:
          accessModes: [ "ReadWriteOnce" ]
          storageClassName: gluster-dynamic
          resources:
            requests:
              storage: 5Gi
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# 
    [root@master200.yinzhengjie.org.cn ~]# cat /yinzhengjie/data/k8s/manifests/basic/statefulset/zk-sts.yaml

    五.基于Operator部署etcd

      博主推荐阅读:
        https://github.com/coreos/etcd-operator
        https://github.com/coreos/etcd-operator/blob/master/example/deployment.yaml
        https://github.com/coreos/etcd-operator/blob/master/example/example-etcd-cluster.yaml

  • 相关阅读:
    数组迭代方法
    promise
    Gulp执行预处理
    第一个gulp 项目
    vue 单元素过渡
    webpack 入门
    webpack初始化
    v-for 指令
    ajax 工作原理
    面试小问题
  • 原文地址:https://www.cnblogs.com/yinzhengjie/p/12297707.html
Copyright © 2011-2022 走看看