CentOS 7.6使用kubeadm部署k8s 1.17.2测试集群实战篇
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
kubernetes技术已经称为原生云技术的事实标准，它是目前基础软件领域最为热门的分布式调度和管理平台。于是,kubernetes也几乎成了时下开发工程师和运维工程师必备的技能之一。今天我们就来一起搭建一个Mini版本的kubernetes集群来简单体验一下吧。
一.主机基础环境准备
1>.测试环境说明
测试使用的kubernetes集群可由一个master主机及一个以上(建议至少两个)node主机组成，这些主机可以是物理服务器,也可以是vmware,virtualbox或kvm等虚拟化平台上的虚拟机，甚至是公有云上的VPS主机。 本测试环境将由master200,node201,node202,node203四个独立的主机组成，它们分别拥有4核心的CPU及4G的内存资源，操作系统均为"CentOS Linux release 7.6.1810 (Core)",域名为"yinzhengjie.org.cn",具体配置如下图所示。
此外,各主机需要预设的系统环境如下:
(1)借助NTP服务设置节点时间精确同步;
(2)通过DNS完成各节点的主机名解析，测试环境主机数量较少时也可以使用hosts文件进行;
(3)关闭各节点的iptables或firewalld服务,并确保它们被禁止随系统引导过程启动;
(4)各节点禁用Selinux(否则在运行容器时可以会遇到各种奇葩报错);
(5)各节点禁用所有的swap设备(生产环境中强烈建议禁用,虽说K8S也支持配置参数来启用swap但这样会降低集群性能，使用"swapoff -a"只是临时关闭交换分区使用,永久关闭需要编辑"/etc/fstab"文件将挂载swap哪一行前面加一个"#"进行注释);
(6)若要使用ipvs模型的proxy，各节点还需要载入ipvs相关的各模块;
2>.搭建内网的时间服务器(设定集群各节点时钟同步)
[root@master200.yinzhengjie.org.cn ~]# yum -y install chrony Loaded plugins: fastestmirror Determining fastest mirrors * base: mirror.bit.edu.cn * extras: mirrors.tuna.tsinghua.edu.cn * updates: mirrors.tuna.tsinghua.edu.cn base | 3.6 kB 00:00:00 extras | 2.9 kB 00:00:00 updates | 2.9 kB 00:00:00 (1/2): extras/7/x86_64/primary_db | 159 kB 00:00:00 (2/2): updates/7/x86_64/primary_db | 5.9 MB 00:00:00 Resolving Dependencies --> Running transaction check ---> Package chrony.x86_64 0:3.4-1.el7 will be installed --> Processing Dependency: libseccomp.so.2()(64bit) for package: chrony-3.4-1.el7.x86_64 --> Running transaction check ---> Package libseccomp.x86_64 0:2.3.1-3.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================================================================================================================================================================== Package Arch Version Repository Size ============================================================================================================================================================================================================================================================================== Installing: chrony x86_64 3.4-1.el7 base 251 k Installing for dependencies: libseccomp x86_64 2.3.1-3.el7 base 56 k Transaction Summary ============================================================================================================================================================================================================================================================================== Install 1 Package (+1 Dependent package) Total download size: 306 k Installed size: 788 k Downloading packages: (1/2): chrony-3.4-1.el7.x86_64.rpm | 251 kB 00:00:00 (2/2): libseccomp-2.3.1-3.el7.x86_64.rpm | 56 kB 00:00:00 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Total 1.7 MB/s | 306 kB 00:00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : libseccomp-2.3.1-3.el7.x86_64 1/2 Installing : chrony-3.4-1.el7.x86_64 2/2 Verifying : libseccomp-2.3.1-3.el7.x86_64 1/2 Verifying : chrony-3.4-1.el7.x86_64 2/2 Installed: chrony.x86_64 0:3.4-1.el7 Dependency Installed: libseccomp.x86_64 0:2.3.1-3.el7 Complete! [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# cp /etc/chrony.conf /etc/chrony.conf-`date +%F` [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# sed -r -i 's@(^server)@#1@g' /etc/chrony.conf [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# sed -r -i 's@#(allow) 192.168.0.0/16@1 172.200.0.0/21@' /etc/chrony.conf [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# sed -r -i 's@#(local)@1@' /etc/chrony.conf [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# vim /etc/chrony.conf [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# egrep -v "^#|^$" /etc/chrony.conf server master200.yinzhengjie.org.cn iburst driftfile /var/lib/chrony/drift makestep 1.0 3 rtcsync allow 172.200.0.0/21 allow 127.0.0.0/8 local stratum 10 logdir /var/log/chrony [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# systemctl restart chronyd.service [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# systemctl status chronyd.service ● chronyd.service - NTP client/server Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2020-02-04 17:25:31 CST; 4s ago Docs: man:chronyd(8) man:chrony.conf(5) Process: 5658 ExecStartPost=/usr/libexec/chrony-helper update-daemon (code=exited, status=0/SUCCESS) Process: 5654 ExecStart=/usr/sbin/chronyd $OPTIONS (code=exited, status=0/SUCCESS) Main PID: 5656 (chronyd) CGroup: /system.slice/chronyd.service └─5656 /usr/sbin/chronyd Feb 04 17:25:31 master200.yinzhengjie.org.cn systemd[1]: Starting NTP client/server... Feb 04 17:25:31 master200.yinzhengjie.org.cn chronyd[5656]: chronyd version 3.4 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +SECHASH +IPV6 +DEBUG) Feb 04 17:25:31 master200.yinzhengjie.org.cn systemd[1]: Started NTP client/server. Feb 04 17:25:35 master200.yinzhengjie.org.cn chronyd[5656]: Selected source 172.200.1.200 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# systemctl enable chronyd.service [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# systemctl list-unit-files | grep chronyd chronyd.service enabled [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
[root@node201.yinzhengjie.org.cn ~]# vim /etc/chrony.conf [root@node201.yinzhengjie.org.cn ~]# [root@node201.yinzhengjie.org.cn ~]# egrep -v "^#|^$" /etc/chrony.conf server master200.yinzhengjie.org.cn iburst driftfile /var/lib/chrony/drift makestep 1.0 3 rtcsync logdir /var/log/chrony [root@node201.yinzhengjie.org.cn ~]# [root@node201.yinzhengjie.org.cn ~]# systemctl restart chronyd.service [root@node201.yinzhengjie.org.cn ~]# [root@node201.yinzhengjie.org.cn ~]# systemctl enable chronyd.service [root@node201.yinzhengjie.org.cn ~]# [root@node201.yinzhengjie.org.cn ~]# systemctl list-unit-files | grep chronyd chronyd.service enabled [root@node201.yinzhengjie.org.cn ~]# [root@node201.yinzhengjie.org.cn ~]#
[root@node201.yinzhengjie.org.cn ~]# chronyc sources 210 Number of sources = 1 MS Name/IP address Stratum Poll Reach LastRx Last sample =============================================================================== ^* master200.yinzhengjie.or> 11 6 37 48 +75ns[+2097ns] +/- 87ms [root@node201.yinzhengjie.org.cn ~]#
[root@node201.yinzhengjie.org.cn ~]# chronyc sourcestats -v 210 Number of sources = 1 .- Number of sample points in measurement set. / .- Number of residual runs with same sign. | / .- Length of measurement set (time). | | / .- Est. clock freq error (ppm). | | | / .- Est. error in freq. | | | | / .- Est. offset. | | | | | | On the -. | | | | | | samples. | | | | | | | Name/IP Address NP NR Span Frequency Freq Skew Offset Std Dev ============================================================================== master200.yinzhengjie.or> 6 6 136 -0.002 0.471 -23ns 4950ns [root@node201.yinzhengjie.org.cn ~]#
3>.启用ipvs内核模块
[root@master200.yinzhengjie.org.cn ~]# vim /etc/sysconfig/modules/ipvs.modules [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# cat /etc/sysconfig/modules/ipvs.modules #!/bin/bash # #******************************************************************** #Author: yinzhengjie #QQ: 1053419035 #Date: 2019-11-30 #URL: http://www.cnblogs.com/yinzhengjie #Description： enable ipvs script #Copyright notice: original works, no reprint! Otherwise, legal liability will be investigated. #******************************************************************** ipvs_mods_dir="/usr/lib/modules/$(uname -r)/kernel/net/netfilter/ipvs" for mod in $(ls $ipvs_mods_dir | grep -o "^[^.]*");do /usr/sbin/modinfo -F filename $mod &> /dev/null if [ $? -eq 0 ];then /sbin/modprobe $mod fi done [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# lsmod | grep ip_vs [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# chmod +x /etc/sysconfig/modules/ipvs.modules [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# /etc/sysconfig/modules/ipvs.modules [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# lsmod | grep ip_vs ip_vs_wrr 12697 0 ip_vs_wlc 12519 0 ip_vs_sh 12688 0 ip_vs_sed 12519 0 ip_vs_rr 12600 0 ip_vs_pe_sip 12740 0 nf_conntrack_sip 33860 1 ip_vs_pe_sip ip_vs_nq 12516 0 ip_vs_lc 12516 0 ip_vs_lblcr 12922 0 ip_vs_lblc 12819 0 ip_vs_ftp 13079 0 nf_nat 26787 1 ip_vs_ftp ip_vs_dh 12688 0 ip_vs 145497 24 ip_vs_dh,ip_vs_lc,ip_vs_nq,ip_vs_rr,ip_vs_sh,ip_vs_ftp,ip_vs_sed,ip_vs_wlc,ip_vs_wrr,ip_vs_pe_sip,ip_vs_lblcr,ip_vs_lblc nf_conntrack 133095 3 ip_vs,nf_nat,nf_conntrack_sip libcrc32c 12644 4 xfs,ip_vs,nf_nat,nf_conntrack [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# ll /etc/sysconfig/modules/ipvs.modules -rwxr-xr-x 1 root root 680 Feb 4 10:08 /etc/sysconfig/modules/ipvs.modules [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# scp -p /etc/sysconfig/modules/ipvs.modules node201.yinzhengjie.org.cn:/etc/sysconfig/modules/ root@node201.yinzhengjie.org.cn's password: ipvs.modules 100% 680 521.4KB/s 00:00 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# ll /etc/sysconfig/modules/ipvs.modules -rwxr-xr-x 1 root root 680 Feb 4 10:08 /etc/sysconfig/modules/ipvs.modules [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# scp -p /etc/sysconfig/modules/ipvs.modules node202.yinzhengjie.org.cn:/etc/sysconfig/modules/ root@node202.yinzhengjie.org.cn's password: ipvs.modules 100% 680 521.4KB/s 00:00 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# ll /etc/sysconfig/modules/ipvs.modules -rwxr-xr-x 1 root root 680 Feb 4 10:08 /etc/sysconfig/modules/ipvs.modules [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# scp -p /etc/sysconfig/modules/ipvs.modules node203.yinzhengjie.org.cn:/etc/sysconfig/modules/ root@node203.yinzhengjie.org.cn's password: ipvs.modules 100% 680 521.4KB/s 00:00 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
二.安装docker并启动(每个节点都需要安装docker环境)
1>.下载docker阿里云的软件源文件
[root@master200.yinzhengjie.org.cn ~]# cd /etc/yum.repos.d/ [root@master200.yinzhengjie.org.cn /etc/yum.repos.d]# [root@master200.yinzhengjie.org.cn /etc/yum.repos.d]# ll total 32 -rw-r--r--. 1 root root 1664 Nov 23 2018 CentOS-Base.repo -rw-r--r--. 1 root root 1309 Nov 23 2018 CentOS-CR.repo -rw-r--r--. 1 root root 649 Nov 23 2018 CentOS-Debuginfo.repo -rw-r--r--. 1 root root 314 Nov 23 2018 CentOS-fasttrack.repo -rw-r--r--. 1 root root 630 Nov 23 2018 CentOS-Media.repo -rw-r--r--. 1 root root 1331 Nov 23 2018 CentOS-Sources.repo -rw-r--r--. 1 root root 5701 Nov 23 2018 CentOS-Vault.repo [root@master200.yinzhengjie.org.cn /etc/yum.repos.d]# [root@master200.yinzhengjie.org.cn /etc/yum.repos.d]# wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo --2020-02-04 10:44:27-- https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo Resolving mirrors.aliyun.com (mirrors.aliyun.com)... 61.240.147.118, 27.221.92.111, 61.240.147.114, ... Connecting to mirrors.aliyun.com (mirrors.aliyun.com)|61.240.147.118|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 2640 (2.6K) [application/octet-stream] Saving to: ‘docker-ce.repo’ 100%[===================================================================================================================================================================================================================>] 2,640 --.-K/s in 0s 2020-02-04 10:44:27 (69.9 MB/s) - ‘docker-ce.repo’ saved [2640/2640] [root@master200.yinzhengjie.org.cn /etc/yum.repos.d]# [root@master200.yinzhengjie.org.cn /etc/yum.repos.d]# ll total 36 -rw-r--r--. 1 root root 1664 Nov 23 2018 CentOS-Base.repo -rw-r--r--. 1 root root 1309 Nov 23 2018 CentOS-CR.repo -rw-r--r--. 1 root root 649 Nov 23 2018 CentOS-Debuginfo.repo -rw-r--r--. 1 root root 314 Nov 23 2018 CentOS-fasttrack.repo -rw-r--r--. 1 root root 630 Nov 23 2018 CentOS-Media.repo -rw-r--r--. 1 root root 1331 Nov 23 2018 CentOS-Sources.repo -rw-r--r--. 1 root root 5701 Nov 23 2018 CentOS-Vault.repo -rw-r--r-- 1 root root 2640 Feb 3 16:23 docker-ce.repo [root@master200.yinzhengjie.org.cn /etc/yum.repos.d]# [root@master200.yinzhengjie.org.cn /etc/yum.repos.d]#
2>.安装docker
[root@master200.yinzhengjie.org.cn ~]# yum -y install docker-ce Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirror.bit.edu.cn * extras: mirrors.tuna.tsinghua.edu.cn * updates: mirrors.tuna.tsinghua.edu.cn docker-ce-stable | 3.5 kB 00:00:00 (1/2): docker-ce-stable/x86_64/primary_db | 37 kB 00:00:00 (2/2): docker-ce-stable/x86_64/updateinfo | 55 B 00:00:00 Resolving Dependencies --> Running transaction check ---> Package docker-ce.x86_64 3:19.03.5-3.el7 will be installed --> Processing Dependency: container-selinux >= 2:2.74 for package: 3:docker-ce-19.03.5-3.el7.x86_64 --> Processing Dependency: containerd.io >= 1.2.2-3 for package: 3:docker-ce-19.03.5-3.el7.x86_64 --> Processing Dependency: docker-ce-cli for package: 3:docker-ce-19.03.5-3.el7.x86_64 --> Processing Dependency: libcgroup for package: 3:docker-ce-19.03.5-3.el7.x86_64 --> Running transaction check ---> Package container-selinux.noarch 2:2.107-3.el7 will be installed --> Processing Dependency: policycoreutils-python for package: 2:container-selinux-2.107-3.el7.noarch ---> Package containerd.io.x86_64 0:1.2.10-3.2.el7 will be installed ---> Package docker-ce-cli.x86_64 1:19.03.5-3.el7 will be installed ---> Package libcgroup.x86_64 0:0.41-21.el7 will be installed --> Running transaction check ---> Package policycoreutils-python.x86_64 0:2.5-33.el7 will be installed --> Processing Dependency: policycoreutils = 2.5-33.el7 for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: setools-libs >= 3.3.8-4 for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libsemanage-python >= 2.5-14 for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: audit-libs-python >= 2.1.3-4 for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: python-IPy for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libqpol.so.1(VERS_1.4)(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libqpol.so.1(VERS_1.2)(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libapol.so.4(VERS_4.0)(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: checkpolicy for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libqpol.so.1()(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libapol.so.4()(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Running transaction check ---> Package audit-libs-python.x86_64 0:2.8.5-4.el7 will be installed --> Processing Dependency: audit-libs(x86-64) = 2.8.5-4.el7 for package: audit-libs-python-2.8.5-4.el7.x86_64 ---> Package checkpolicy.x86_64 0:2.5-8.el7 will be installed ---> Package libsemanage-python.x86_64 0:2.5-14.el7 will be installed ---> Package policycoreutils.x86_64 0:2.5-29.el7 will be updated ---> Package policycoreutils.x86_64 0:2.5-33.el7 will be an update ---> Package python-IPy.noarch 0:0.75-6.el7 will be installed ---> Package setools-libs.x86_64 0:3.3.8-4.el7 will be installed --> Running transaction check ---> Package audit-libs.x86_64 0:2.8.4-4.el7 will be updated --> Processing Dependency: audit-libs(x86-64) = 2.8.4-4.el7 for package: audit-2.8.4-4.el7.x86_64 ---> Package audit-libs.x86_64 0:2.8.5-4.el7 will be an update --> Running transaction check ---> Package audit.x86_64 0:2.8.4-4.el7 will be updated ---> Package audit.x86_64 0:2.8.5-4.el7 will be an update --> Finished Dependency Resolution Dependencies Resolved ========================================================================================================================================================================================================================================================= Package Arch Version Repository Size ========================================================================================================================================================================================================================================================= Installing: docker-ce x86_64 3:19.03.5-3.el7 docker-ce-stable 24 M Installing for dependencies: audit-libs-python x86_64 2.8.5-4.el7 base 76 k checkpolicy x86_64 2.5-8.el7 base 295 k container-selinux noarch 2:2.107-3.el7 extras 39 k containerd.io x86_64 1.2.10-3.2.el7 docker-ce-stable 23 M docker-ce-cli x86_64 1:19.03.5-3.el7 docker-ce-stable 39 M libcgroup x86_64 0.41-21.el7 base 66 k libsemanage-python x86_64 2.5-14.el7 base 113 k policycoreutils-python x86_64 2.5-33.el7 base 457 k python-IPy noarch 0.75-6.el7 base 32 k setools-libs x86_64 3.3.8-4.el7 base 620 k Updating for dependencies: audit x86_64 2.8.5-4.el7 base 256 k audit-libs x86_64 2.8.5-4.el7 base 102 k policycoreutils x86_64 2.5-33.el7 base 916 k Transaction Summary ========================================================================================================================================================================================================================================================= Install 1 Package (+10 Dependent packages) Upgrade ( 3 Dependent packages) Total download size: 90 M Downloading packages: Delta RPMs disabled because /usr/bin/applydeltarpm not installed. (1/14): audit-libs-2.8.5-4.el7.x86_64.rpm | 102 kB 00:00:00 (2/14): audit-2.8.5-4.el7.x86_64.rpm | 256 kB 00:00:00 (3/14): audit-libs-python-2.8.5-4.el7.x86_64.rpm | 76 kB 00:00:00 (4/14): checkpolicy-2.5-8.el7.x86_64.rpm | 295 kB 00:00:00 (5/14): container-selinux-2.107-3.el7.noarch.rpm | 39 kB 00:00:00 warning: /var/cache/yum/x86_64/7/docker-ce-stable/packages/containerd.io-1.2.10-3.2.el7.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 621e9f35: NOKEY ] 3.6 MB/s | 23 MB 00:00:18 ETA Public key for containerd.io-1.2.10-3.2.el7.x86_64.rpm is not installed (6/14): containerd.io-1.2.10-3.2.el7.x86_64.rpm | 23 MB 00:00:02 (7/14): libcgroup-0.41-21.el7.x86_64.rpm | 66 kB 00:00:01 (8/14): libsemanage-python-2.5-14.el7.x86_64.rpm | 113 kB 00:00:03 (9/14): python-IPy-0.75-6.el7.noarch.rpm | 32 kB 00:00:00 (10/14): docker-ce-cli-19.03.5-3.el7.x86_64.rpm | 39 MB 00:00:05 (11/14): setools-libs-3.3.8-4.el7.x86_64.rpm | 620 kB 00:00:00 (12/14): policycoreutils-2.5-33.el7.x86_64.rpm | 916 kB 00:00:03 (13/14): policycoreutils-python-2.5-33.el7.x86_64.rpm | 457 kB 00:00:03 (14/14): docker-ce-19.03.5-3.el7.x86_64.rpm | 24 MB 00:00:09 --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 9.4 MB/s | 90 MB 00:00:09 Retrieving key from https://mirrors.aliyun.com/docker-ce/linux/centos/gpg Importing GPG key 0x621E9F35: Userid : "Docker Release (CE rpm) <docker@docker.com>" Fingerprint: 060a 61c5 1b55 8a7f 742b 77aa c52f eb6b 621e 9f35 From : https://mirrors.aliyun.com/docker-ce/linux/centos/gpg Running transaction check Running transaction test Transaction test succeeded Running transaction Updating : audit-libs-2.8.5-4.el7.x86_64 1/17 Updating : policycoreutils-2.5-33.el7.x86_64 2/17 Installing : libcgroup-0.41-21.el7.x86_64 3/17 Installing : audit-libs-python-2.8.5-4.el7.x86_64 4/17 Installing : setools-libs-3.3.8-4.el7.x86_64 5/17 Installing : 1:docker-ce-cli-19.03.5-3.el7.x86_64 6/17 Installing : checkpolicy-2.5-8.el7.x86_64 7/17 Installing : python-IPy-0.75-6.el7.noarch 8/17 Installing : libsemanage-python-2.5-14.el7.x86_64 9/17 Installing : policycoreutils-python-2.5-33.el7.x86_64 10/17 Installing : 2:container-selinux-2.107-3.el7.noarch 11/17 setsebool: SELinux is disabled. Installing : containerd.io-1.2.10-3.2.el7.x86_64 12/17 Installing : 3:docker-ce-19.03.5-3.el7.x86_64 13/17 Updating : audit-2.8.5-4.el7.x86_64 14/17 Cleanup : policycoreutils-2.5-29.el7.x86_64 15/17 Cleanup : audit-2.8.4-4.el7.x86_64 16/17 Cleanup : audit-libs-2.8.4-4.el7.x86_64 17/17 Verifying : audit-libs-2.8.5-4.el7.x86_64 1/17 Verifying : policycoreutils-python-2.5-33.el7.x86_64 2/17 Verifying : audit-2.8.5-4.el7.x86_64 3/17 Verifying : 3:docker-ce-19.03.5-3.el7.x86_64 4/17 Verifying : audit-libs-python-2.8.5-4.el7.x86_64 5/17 Verifying : libsemanage-python-2.5-14.el7.x86_64 6/17 Verifying : 2:container-selinux-2.107-3.el7.noarch 7/17 Verifying : python-IPy-0.75-6.el7.noarch 8/17 Verifying : checkpolicy-2.5-8.el7.x86_64 9/17 Verifying : policycoreutils-2.5-33.el7.x86_64 10/17 Verifying : containerd.io-1.2.10-3.2.el7.x86_64 11/17 Verifying : 1:docker-ce-cli-19.03.5-3.el7.x86_64 12/17 Verifying : setools-libs-3.3.8-4.el7.x86_64 13/17 Verifying : libcgroup-0.41-21.el7.x86_64 14/17 Verifying : policycoreutils-2.5-29.el7.x86_64 15/17 Verifying : audit-2.8.4-4.el7.x86_64 16/17 Verifying : audit-libs-2.8.4-4.el7.x86_64 17/17 Installed: docker-ce.x86_64 3:19.03.5-3.el7 Dependency Installed: audit-libs-python.x86_64 0:2.8.5-4.el7 checkpolicy.x86_64 0:2.5-8.el7 container-selinux.noarch 2:2.107-3.el7 containerd.io.x86_64 0:1.2.10-3.2.el7 docker-ce-cli.x86_64 1:19.03.5-3.el7 libcgroup.x86_64 0:0.41-21.el7 libsemanage-python.x86_64 0:2.5-14.el7 policycoreutils-python.x86_64 0:2.5-33.el7 python-IPy.noarch 0:0.75-6.el7 setools-libs.x86_64 0:3.3.8-4.el7 Dependency Updated: audit.x86_64 0:2.8.5-4.el7 audit-libs.x86_64 0:2.8.5-4.el7 policycoreutils.x86_64 0:2.5-33.el7 Complete! [root@master200.yinzhengjie.org.cn ~]#
[root@node201.yinzhengjie.org.cn ~]# yum -y install docker-ce Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.aliyun.com * extras: mirrors.aliyun.com * updates: mirrors.aliyun.com docker-ce-stable | 3.5 kB 00:00:00 (1/2): docker-ce-stable/x86_64/primary_db | 37 kB 00:00:00 (2/2): docker-ce-stable/x86_64/updateinfo | 55 B 00:00:00 Resolving Dependencies --> Running transaction check ---> Package docker-ce.x86_64 3:19.03.5-3.el7 will be installed --> Processing Dependency: container-selinux >= 2:2.74 for package: 3:docker-ce-19.03.5-3.el7.x86_64 --> Processing Dependency: containerd.io >= 1.2.2-3 for package: 3:docker-ce-19.03.5-3.el7.x86_64 --> Processing Dependency: docker-ce-cli for package: 3:docker-ce-19.03.5-3.el7.x86_64 --> Processing Dependency: libcgroup for package: 3:docker-ce-19.03.5-3.el7.x86_64 --> Running transaction check ---> Package container-selinux.noarch 2:2.107-3.el7 will be installed --> Processing Dependency: policycoreutils-python for package: 2:container-selinux-2.107-3.el7.noarch ---> Package containerd.io.x86_64 0:1.2.10-3.2.el7 will be installed ---> Package docker-ce-cli.x86_64 1:19.03.5-3.el7 will be installed ---> Package libcgroup.x86_64 0:0.41-21.el7 will be installed --> Running transaction check ---> Package policycoreutils-python.x86_64 0:2.5-33.el7 will be installed --> Processing Dependency: policycoreutils = 2.5-33.el7 for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: setools-libs >= 3.3.8-4 for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libsemanage-python >= 2.5-14 for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: audit-libs-python >= 2.1.3-4 for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: python-IPy for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libqpol.so.1(VERS_1.4)(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libqpol.so.1(VERS_1.2)(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libapol.so.4(VERS_4.0)(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: checkpolicy for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libqpol.so.1()(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libapol.so.4()(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Running transaction check ---> Package audit-libs-python.x86_64 0:2.8.5-4.el7 will be installed --> Processing Dependency: audit-libs(x86-64) = 2.8.5-4.el7 for package: audit-libs-python-2.8.5-4.el7.x86_64 ---> Package checkpolicy.x86_64 0:2.5-8.el7 will be installed ---> Package libsemanage-python.x86_64 0:2.5-14.el7 will be installed ---> Package policycoreutils.x86_64 0:2.5-29.el7 will be updated ---> Package policycoreutils.x86_64 0:2.5-33.el7 will be an update ---> Package python-IPy.noarch 0:0.75-6.el7 will be installed ---> Package setools-libs.x86_64 0:3.3.8-4.el7 will be installed --> Running transaction check ---> Package audit-libs.x86_64 0:2.8.4-4.el7 will be updated --> Processing Dependency: audit-libs(x86-64) = 2.8.4-4.el7 for package: audit-2.8.4-4.el7.x86_64 ---> Package audit-libs.x86_64 0:2.8.5-4.el7 will be an update --> Running transaction check ---> Package audit.x86_64 0:2.8.4-4.el7 will be updated ---> Package audit.x86_64 0:2.8.5-4.el7 will be an update --> Finished Dependency Resolution Dependencies Resolved ========================================================================================================================================================================================================================================================= Package Arch Version Repository Size ========================================================================================================================================================================================================================================================= Installing: docker-ce x86_64 3:19.03.5-3.el7 docker-ce-stable 24 M Installing for dependencies: audit-libs-python x86_64 2.8.5-4.el7 base 76 k checkpolicy x86_64 2.5-8.el7 base 295 k container-selinux noarch 2:2.107-3.el7 extras 39 k containerd.io x86_64 1.2.10-3.2.el7 docker-ce-stable 23 M docker-ce-cli x86_64 1:19.03.5-3.el7 docker-ce-stable 39 M libcgroup x86_64 0.41-21.el7 base 66 k libsemanage-python x86_64 2.5-14.el7 base 113 k policycoreutils-python x86_64 2.5-33.el7 base 457 k python-IPy noarch 0.75-6.el7 base 32 k setools-libs x86_64 3.3.8-4.el7 base 620 k Updating for dependencies: audit x86_64 2.8.5-4.el7 base 256 k audit-libs x86_64 2.8.5-4.el7 base 102 k policycoreutils x86_64 2.5-33.el7 base 916 k Transaction Summary ========================================================================================================================================================================================================================================================= Install 1 Package (+10 Dependent packages) Upgrade ( 3 Dependent packages) Total download size: 90 M Downloading packages: Delta RPMs disabled because /usr/bin/applydeltarpm not installed. (1/14): audit-libs-2.8.5-4.el7.x86_64.rpm | 102 kB 00:00:00 (2/14): audit-libs-python-2.8.5-4.el7.x86_64.rpm | 76 kB 00:00:00 (3/14): audit-2.8.5-4.el7.x86_64.rpm | 256 kB 00:00:00 (4/14): checkpolicy-2.5-8.el7.x86_64.rpm | 295 kB 00:00:00 (5/14): container-selinux-2.107-3.el7.noarch.rpm | 39 kB 00:00:00 warning: /var/cache/yum/x86_64/7/docker-ce-stable/packages/containerd.io-1.2.10-3.2.el7.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 621e9f35: NOKEY ] 4.7 MB/s | 22 MB 00:00:14 ETA Public key for containerd.io-1.2.10-3.2.el7.x86_64.rpm is not installed (6/14): containerd.io-1.2.10-3.2.el7.x86_64.rpm | 23 MB 00:00:03 (7/14): libcgroup-0.41-21.el7.x86_64.rpm | 66 kB 00:00:00 (8/14): libsemanage-python-2.5-14.el7.x86_64.rpm | 113 kB 00:00:00 (9/14): python-IPy-0.75-6.el7.noarch.rpm | 32 kB 00:00:00 (10/14): docker-ce-cli-19.03.5-3.el7.x86_64.rpm | 39 MB 00:00:04 (11/14): setools-libs-3.3.8-4.el7.x86_64.rpm | 620 kB 00:00:03 (12/14): policycoreutils-2.5-33.el7.x86_64.rpm | 916 kB 00:00:06 (13/14): policycoreutils-python-2.5-33.el7.x86_64.rpm | 457 kB 00:00:08 (14/14): docker-ce-19.03.5-3.el7.x86_64.rpm | 24 MB 00:00:20 --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 4.3 MB/s | 90 MB 00:00:20 Retrieving key from https://mirrors.aliyun.com/docker-ce/linux/centos/gpg Importing GPG key 0x621E9F35: Userid : "Docker Release (CE rpm) <docker@docker.com>" Fingerprint: 060a 61c5 1b55 8a7f 742b 77aa c52f eb6b 621e 9f35 From : https://mirrors.aliyun.com/docker-ce/linux/centos/gpg Running transaction check Running transaction test Transaction test succeeded Running transaction Updating : audit-libs-2.8.5-4.el7.x86_64 1/17 Updating : policycoreutils-2.5-33.el7.x86_64 2/17 Installing : libcgroup-0.41-21.el7.x86_64 3/17 Installing : audit-libs-python-2.8.5-4.el7.x86_64 4/17 Installing : setools-libs-3.3.8-4.el7.x86_64 5/17 Installing : 1:docker-ce-cli-19.03.5-3.el7.x86_64 6/17 Installing : checkpolicy-2.5-8.el7.x86_64 7/17 Installing : python-IPy-0.75-6.el7.noarch 8/17 Installing : libsemanage-python-2.5-14.el7.x86_64 9/17 Installing : policycoreutils-python-2.5-33.el7.x86_64 10/17 Installing : 2:container-selinux-2.107-3.el7.noarch 11/17 setsebool: SELinux is disabled. Installing : containerd.io-1.2.10-3.2.el7.x86_64 12/17 Installing : 3:docker-ce-19.03.5-3.el7.x86_64 13/17 Updating : audit-2.8.5-4.el7.x86_64 14/17 Cleanup : policycoreutils-2.5-29.el7.x86_64 15/17 Cleanup : audit-2.8.4-4.el7.x86_64 16/17 Cleanup : audit-libs-2.8.4-4.el7.x86_64 17/17 Verifying : audit-libs-2.8.5-4.el7.x86_64 1/17 Verifying : policycoreutils-python-2.5-33.el7.x86_64 2/17 Verifying : audit-2.8.5-4.el7.x86_64 3/17 Verifying : 3:docker-ce-19.03.5-3.el7.x86_64 4/17 Verifying : audit-libs-python-2.8.5-4.el7.x86_64 5/17 Verifying : libsemanage-python-2.5-14.el7.x86_64 6/17 Verifying : 2:container-selinux-2.107-3.el7.noarch 7/17 Verifying : python-IPy-0.75-6.el7.noarch 8/17 Verifying : checkpolicy-2.5-8.el7.x86_64 9/17 Verifying : policycoreutils-2.5-33.el7.x86_64 10/17 Verifying : containerd.io-1.2.10-3.2.el7.x86_64 11/17 Verifying : 1:docker-ce-cli-19.03.5-3.el7.x86_64 12/17 Verifying : setools-libs-3.3.8-4.el7.x86_64 13/17 Verifying : libcgroup-0.41-21.el7.x86_64 14/17 Verifying : policycoreutils-2.5-29.el7.x86_64 15/17 Verifying : audit-2.8.4-4.el7.x86_64 16/17 Verifying : audit-libs-2.8.4-4.el7.x86_64 17/17 Installed: docker-ce.x86_64 3:19.03.5-3.el7 Dependency Installed: audit-libs-python.x86_64 0:2.8.5-4.el7 checkpolicy.x86_64 0:2.5-8.el7 container-selinux.noarch 2:2.107-3.el7 containerd.io.x86_64 0:1.2.10-3.2.el7 docker-ce-cli.x86_64 1:19.03.5-3.el7 libcgroup.x86_64 0:0.41-21.el7 libsemanage-python.x86_64 0:2.5-14.el7 policycoreutils-python.x86_64 0:2.5-33.el7 python-IPy.noarch 0:0.75-6.el7 setools-libs.x86_64 0:3.3.8-4.el7 Dependency Updated: audit.x86_64 0:2.8.5-4.el7 audit-libs.x86_64 0:2.8.5-4.el7 policycoreutils.x86_64 0:2.5-33.el7 Complete! [root@node201.yinzhengjie.org.cn ~]#
[root@node202.yinzhengjie.org.cn ~]# yum -y install docker-ce Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirror.bit.edu.cn * extras: mirrors.aliyun.com * updates: mirrors.aliyun.com docker-ce-stable | 3.5 kB 00:00:00 (1/2): docker-ce-stable/x86_64/updateinfo | 55 B 00:00:00 (2/2): docker-ce-stable/x86_64/primary_db | 37 kB 00:00:00 Resolving Dependencies --> Running transaction check ---> Package docker-ce.x86_64 3:19.03.5-3.el7 will be installed --> Processing Dependency: container-selinux >= 2:2.74 for package: 3:docker-ce-19.03.5-3.el7.x86_64 --> Processing Dependency: containerd.io >= 1.2.2-3 for package: 3:docker-ce-19.03.5-3.el7.x86_64 --> Processing Dependency: docker-ce-cli for package: 3:docker-ce-19.03.5-3.el7.x86_64 --> Processing Dependency: libcgroup for package: 3:docker-ce-19.03.5-3.el7.x86_64 --> Running transaction check ---> Package container-selinux.noarch 2:2.107-3.el7 will be installed --> Processing Dependency: policycoreutils-python for package: 2:container-selinux-2.107-3.el7.noarch ---> Package containerd.io.x86_64 0:1.2.10-3.2.el7 will be installed ---> Package docker-ce-cli.x86_64 1:19.03.5-3.el7 will be installed ---> Package libcgroup.x86_64 0:0.41-21.el7 will be installed --> Running transaction check ---> Package policycoreutils-python.x86_64 0:2.5-33.el7 will be installed --> Processing Dependency: policycoreutils = 2.5-33.el7 for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: setools-libs >= 3.3.8-4 for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libsemanage-python >= 2.5-14 for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: audit-libs-python >= 2.1.3-4 for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: python-IPy for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libqpol.so.1(VERS_1.4)(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libqpol.so.1(VERS_1.2)(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libapol.so.4(VERS_4.0)(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: checkpolicy for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libqpol.so.1()(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libapol.so.4()(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Running transaction check ---> Package audit-libs-python.x86_64 0:2.8.5-4.el7 will be installed --> Processing Dependency: audit-libs(x86-64) = 2.8.5-4.el7 for package: audit-libs-python-2.8.5-4.el7.x86_64 ---> Package checkpolicy.x86_64 0:2.5-8.el7 will be installed ---> Package libsemanage-python.x86_64 0:2.5-14.el7 will be installed ---> Package policycoreutils.x86_64 0:2.5-29.el7 will be updated ---> Package policycoreutils.x86_64 0:2.5-33.el7 will be an update ---> Package python-IPy.noarch 0:0.75-6.el7 will be installed ---> Package setools-libs.x86_64 0:3.3.8-4.el7 will be installed --> Running transaction check ---> Package audit-libs.x86_64 0:2.8.4-4.el7 will be updated --> Processing Dependency: audit-libs(x86-64) = 2.8.4-4.el7 for package: audit-2.8.4-4.el7.x86_64 ---> Package audit-libs.x86_64 0:2.8.5-4.el7 will be an update --> Running transaction check ---> Package audit.x86_64 0:2.8.4-4.el7 will be updated ---> Package audit.x86_64 0:2.8.5-4.el7 will be an update --> Finished Dependency Resolution Dependencies Resolved ========================================================================================================================================================================================================================================================= Package Arch Version Repository Size ========================================================================================================================================================================================================================================================= Installing: docker-ce x86_64 3:19.03.5-3.el7 docker-ce-stable 24 M Installing for dependencies: audit-libs-python x86_64 2.8.5-4.el7 base 76 k checkpolicy x86_64 2.5-8.el7 base 295 k container-selinux noarch 2:2.107-3.el7 extras 39 k containerd.io x86_64 1.2.10-3.2.el7 docker-ce-stable 23 M docker-ce-cli x86_64 1:19.03.5-3.el7 docker-ce-stable 39 M libcgroup x86_64 0.41-21.el7 base 66 k libsemanage-python x86_64 2.5-14.el7 base 113 k policycoreutils-python x86_64 2.5-33.el7 base 457 k python-IPy noarch 0.75-6.el7 base 32 k setools-libs x86_64 3.3.8-4.el7 base 620 k Updating for dependencies: audit x86_64 2.8.5-4.el7 base 256 k audit-libs x86_64 2.8.5-4.el7 base 102 k policycoreutils x86_64 2.5-33.el7 base 916 k Transaction Summary ========================================================================================================================================================================================================================================================= Install 1 Package (+10 Dependent packages) Upgrade ( 3 Dependent packages) Total download size: 90 M Downloading packages: Delta RPMs disabled because /usr/bin/applydeltarpm not installed. (1/14): audit-libs-2.8.5-4.el7.x86_64.rpm | 102 kB 00:00:01 (2/14): audit-2.8.5-4.el7.x86_64.rpm | 256 kB 00:00:01 (3/14): container-selinux-2.107-3.el7.noarch.rpm | 39 kB 00:00:00 (4/14): audit-libs-python-2.8.5-4.el7.x86_64.rpm | 76 kB 00:00:01 (5/14): checkpolicy-2.5-8.el7.x86_64.rpm | 295 kB 00:00:03 warning: /var/cache/yum/x86_64/7/docker-ce-stable/packages/containerd.io-1.2.10-3.2.el7.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 621e9f35: NOKEY ] 3.8 MB/s | 34 MB 00:00:14 ETA Public key for containerd.io-1.2.10-3.2.el7.x86_64.rpm is not installed (6/14): containerd.io-1.2.10-3.2.el7.x86_64.rpm | 23 MB 00:00:15 (7/14): libcgroup-0.41-21.el7.x86_64.rpm | 66 kB 00:00:00 (8/14): libsemanage-python-2.5-14.el7.x86_64.rpm | 113 kB 00:00:00 (9/14): python-IPy-0.75-6.el7.noarch.rpm | 32 kB 00:00:00 (10/14): policycoreutils-python-2.5-33.el7.x86_64.rpm | 457 kB 00:00:00 (11/14): setools-libs-3.3.8-4.el7.x86_64.rpm | 620 kB 00:00:00 (12/14): policycoreutils-2.5-33.el7.x86_64.rpm | 916 kB 00:00:02 (13/14): docker-ce-19.03.5-3.el7.x86_64.rpm | 24 MB 00:00:21 (14/14): docker-ce-cli-19.03.5-3.el7.x86_64.rpm | 39 MB 00:00:07 --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 3.7 MB/s | 90 MB 00:00:24 Retrieving key from https://mirrors.aliyun.com/docker-ce/linux/centos/gpg Importing GPG key 0x621E9F35: Userid : "Docker Release (CE rpm) <docker@docker.com>" Fingerprint: 060a 61c5 1b55 8a7f 742b 77aa c52f eb6b 621e 9f35 From : https://mirrors.aliyun.com/docker-ce/linux/centos/gpg Running transaction check Running transaction test Transaction test succeeded Running transaction Updating : audit-libs-2.8.5-4.el7.x86_64 1/17 Updating : policycoreutils-2.5-33.el7.x86_64 2/17 Installing : libcgroup-0.41-21.el7.x86_64 3/17 Installing : audit-libs-python-2.8.5-4.el7.x86_64 4/17 Installing : setools-libs-3.3.8-4.el7.x86_64 5/17 Installing : 1:docker-ce-cli-19.03.5-3.el7.x86_64 6/17 Installing : checkpolicy-2.5-8.el7.x86_64 7/17 Installing : python-IPy-0.75-6.el7.noarch 8/17 Installing : libsemanage-python-2.5-14.el7.x86_64 9/17 Installing : policycoreutils-python-2.5-33.el7.x86_64 10/17 Installing : 2:container-selinux-2.107-3.el7.noarch 11/17 setsebool: SELinux is disabled. Installing : containerd.io-1.2.10-3.2.el7.x86_64 12/17 Installing : 3:docker-ce-19.03.5-3.el7.x86_64 13/17 Updating : audit-2.8.5-4.el7.x86_64 14/17 Cleanup : policycoreutils-2.5-29.el7.x86_64 15/17 Cleanup : audit-2.8.4-4.el7.x86_64 16/17 Cleanup : audit-libs-2.8.4-4.el7.x86_64 17/17 Verifying : audit-libs-2.8.5-4.el7.x86_64 1/17 Verifying : policycoreutils-python-2.5-33.el7.x86_64 2/17 Verifying : audit-2.8.5-4.el7.x86_64 3/17 Verifying : 3:docker-ce-19.03.5-3.el7.x86_64 4/17 Verifying : audit-libs-python-2.8.5-4.el7.x86_64 5/17 Verifying : libsemanage-python-2.5-14.el7.x86_64 6/17 Verifying : 2:container-selinux-2.107-3.el7.noarch 7/17 Verifying : python-IPy-0.75-6.el7.noarch 8/17 Verifying : checkpolicy-2.5-8.el7.x86_64 9/17 Verifying : policycoreutils-2.5-33.el7.x86_64 10/17 Verifying : containerd.io-1.2.10-3.2.el7.x86_64 11/17 Verifying : 1:docker-ce-cli-19.03.5-3.el7.x86_64 12/17 Verifying : setools-libs-3.3.8-4.el7.x86_64 13/17 Verifying : libcgroup-0.41-21.el7.x86_64 14/17 Verifying : policycoreutils-2.5-29.el7.x86_64 15/17 Verifying : audit-2.8.4-4.el7.x86_64 16/17 Verifying : audit-libs-2.8.4-4.el7.x86_64 17/17 Installed: docker-ce.x86_64 3:19.03.5-3.el7 Dependency Installed: audit-libs-python.x86_64 0:2.8.5-4.el7 checkpolicy.x86_64 0:2.5-8.el7 container-selinux.noarch 2:2.107-3.el7 containerd.io.x86_64 0:1.2.10-3.2.el7 docker-ce-cli.x86_64 1:19.03.5-3.el7 libcgroup.x86_64 0:0.41-21.el7 libsemanage-python.x86_64 0:2.5-14.el7 policycoreutils-python.x86_64 0:2.5-33.el7 python-IPy.noarch 0:0.75-6.el7 setools-libs.x86_64 0:3.3.8-4.el7 Dependency Updated: audit.x86_64 0:2.8.5-4.el7 audit-libs.x86_64 0:2.8.5-4.el7 policycoreutils.x86_64 0:2.5-33.el7 Complete! [root@node202.yinzhengjie.org.cn ~]#
[root@node203.yinzhengjie.org.cn ~]# yum -y install docker-ce Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.aliyun.com * extras: mirrors.aliyun.com * updates: mirrors.aliyun.com docker-ce-stable | 3.5 kB 00:00:00 (1/2): docker-ce-stable/x86_64/updateinfo | 55 B 00:00:00 (2/2): docker-ce-stable/x86_64/primary_db | 37 kB 00:00:01 Resolving Dependencies --> Running transaction check ---> Package docker-ce.x86_64 3:19.03.5-3.el7 will be installed --> Processing Dependency: container-selinux >= 2:2.74 for package: 3:docker-ce-19.03.5-3.el7.x86_64 --> Processing Dependency: containerd.io >= 1.2.2-3 for package: 3:docker-ce-19.03.5-3.el7.x86_64 --> Processing Dependency: docker-ce-cli for package: 3:docker-ce-19.03.5-3.el7.x86_64 --> Processing Dependency: libcgroup for package: 3:docker-ce-19.03.5-3.el7.x86_64 --> Running transaction check ---> Package container-selinux.noarch 2:2.107-3.el7 will be installed --> Processing Dependency: policycoreutils-python for package: 2:container-selinux-2.107-3.el7.noarch ---> Package containerd.io.x86_64 0:1.2.10-3.2.el7 will be installed ---> Package docker-ce-cli.x86_64 1:19.03.5-3.el7 will be installed ---> Package libcgroup.x86_64 0:0.41-21.el7 will be installed --> Running transaction check ---> Package policycoreutils-python.x86_64 0:2.5-33.el7 will be installed --> Processing Dependency: policycoreutils = 2.5-33.el7 for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: setools-libs >= 3.3.8-4 for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libsemanage-python >= 2.5-14 for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: audit-libs-python >= 2.1.3-4 for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: python-IPy for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libqpol.so.1(VERS_1.4)(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libqpol.so.1(VERS_1.2)(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libapol.so.4(VERS_4.0)(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: checkpolicy for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libqpol.so.1()(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Processing Dependency: libapol.so.4()(64bit) for package: policycoreutils-python-2.5-33.el7.x86_64 --> Running transaction check ---> Package audit-libs-python.x86_64 0:2.8.5-4.el7 will be installed --> Processing Dependency: audit-libs(x86-64) = 2.8.5-4.el7 for package: audit-libs-python-2.8.5-4.el7.x86_64 ---> Package checkpolicy.x86_64 0:2.5-8.el7 will be installed ---> Package libsemanage-python.x86_64 0:2.5-14.el7 will be installed ---> Package policycoreutils.x86_64 0:2.5-29.el7 will be updated ---> Package policycoreutils.x86_64 0:2.5-33.el7 will be an update ---> Package python-IPy.noarch 0:0.75-6.el7 will be installed ---> Package setools-libs.x86_64 0:3.3.8-4.el7 will be installed --> Running transaction check ---> Package audit-libs.x86_64 0:2.8.4-4.el7 will be updated --> Processing Dependency: audit-libs(x86-64) = 2.8.4-4.el7 for package: audit-2.8.4-4.el7.x86_64 ---> Package audit-libs.x86_64 0:2.8.5-4.el7 will be an update --> Running transaction check ---> Package audit.x86_64 0:2.8.4-4.el7 will be updated ---> Package audit.x86_64 0:2.8.5-4.el7 will be an update --> Finished Dependency Resolution Dependencies Resolved ========================================================================================================================================================================================================================================================= Package Arch Version Repository Size ========================================================================================================================================================================================================================================================= Installing: docker-ce x86_64 3:19.03.5-3.el7 docker-ce-stable 24 M Installing for dependencies: audit-libs-python x86_64 2.8.5-4.el7 base 76 k checkpolicy x86_64 2.5-8.el7 base 295 k container-selinux noarch 2:2.107-3.el7 extras 39 k containerd.io x86_64 1.2.10-3.2.el7 docker-ce-stable 23 M docker-ce-cli x86_64 1:19.03.5-3.el7 docker-ce-stable 39 M libcgroup x86_64 0.41-21.el7 base 66 k libsemanage-python x86_64 2.5-14.el7 base 113 k policycoreutils-python x86_64 2.5-33.el7 base 457 k python-IPy noarch 0.75-6.el7 base 32 k setools-libs x86_64 3.3.8-4.el7 base 620 k Updating for dependencies: audit x86_64 2.8.5-4.el7 base 256 k audit-libs x86_64 2.8.5-4.el7 base 102 k policycoreutils x86_64 2.5-33.el7 base 916 k Transaction Summary ========================================================================================================================================================================================================================================================= Install 1 Package (+10 Dependent packages) Upgrade ( 3 Dependent packages) Total download size: 90 M Downloading packages: Delta RPMs disabled because /usr/bin/applydeltarpm not installed. (1/14): audit-libs-2.8.5-4.el7.x86_64.rpm | 102 kB 00:00:00 (2/14): audit-libs-python-2.8.5-4.el7.x86_64.rpm | 76 kB 00:00:00 (3/14): container-selinux-2.107-3.el7.noarch.rpm | 39 kB 00:00:00 (4/14): audit-2.8.5-4.el7.x86_64.rpm | 256 kB 00:00:01 (5/14): checkpolicy-2.5-8.el7.x86_64.rpm | 295 kB 00:00:00 warning: /var/cache/yum/x86_64/7/docker-ce-stable/packages/docker-ce-19.03.5-3.el7.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 621e9f35: NOKEY ] 3.8 MB/s | 24 MB 00:00:17 ETA Public key for docker-ce-19.03.5-3.el7.x86_64.rpm is not installed (6/14): docker-ce-19.03.5-3.el7.x86_64.rpm | 24 MB 00:00:03 (7/14): libcgroup-0.41-21.el7.x86_64.rpm | 66 kB 00:00:00 (8/14): libsemanage-python-2.5-14.el7.x86_64.rpm | 113 kB 00:00:00 (9/14): python-IPy-0.75-6.el7.noarch.rpm | 32 kB 00:00:00 (10/14): policycoreutils-python-2.5-33.el7.x86_64.rpm | 457 kB 00:00:01 (11/14): policycoreutils-2.5-33.el7.x86_64.rpm | 916 kB 00:00:01 (12/14): setools-libs-3.3.8-4.el7.x86_64.rpm | 620 kB 00:00:01 (13/14): docker-ce-cli-19.03.5-3.el7.x86_64.rpm | 39 MB 00:00:08 (14/14): containerd.io-1.2.10-3.2.el7.x86_64.rpm | 23 MB 00:00:19 --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 4.5 MB/s | 90 MB 00:00:20 Retrieving key from https://mirrors.aliyun.com/docker-ce/linux/centos/gpg Importing GPG key 0x621E9F35: Userid : "Docker Release (CE rpm) <docker@docker.com>" Fingerprint: 060a 61c5 1b55 8a7f 742b 77aa c52f eb6b 621e 9f35 From : https://mirrors.aliyun.com/docker-ce/linux/centos/gpg Running transaction check Running transaction test Transaction test succeeded Running transaction Updating : audit-libs-2.8.5-4.el7.x86_64 1/17 Updating : policycoreutils-2.5-33.el7.x86_64 2/17 Installing : libcgroup-0.41-21.el7.x86_64 3/17 Installing : audit-libs-python-2.8.5-4.el7.x86_64 4/17 Installing : setools-libs-3.3.8-4.el7.x86_64 5/17 Installing : 1:docker-ce-cli-19.03.5-3.el7.x86_64 6/17 Installing : checkpolicy-2.5-8.el7.x86_64 7/17 Installing : python-IPy-0.75-6.el7.noarch 8/17 Installing : libsemanage-python-2.5-14.el7.x86_64 9/17 Installing : policycoreutils-python-2.5-33.el7.x86_64 10/17 Installing : 2:container-selinux-2.107-3.el7.noarch 11/17 setsebool: SELinux is disabled. Installing : containerd.io-1.2.10-3.2.el7.x86_64 12/17 Installing : 3:docker-ce-19.03.5-3.el7.x86_64 13/17 Updating : audit-2.8.5-4.el7.x86_64 14/17 Cleanup : policycoreutils-2.5-29.el7.x86_64 15/17 Cleanup : audit-2.8.4-4.el7.x86_64 16/17 Cleanup : audit-libs-2.8.4-4.el7.x86_64 17/17 Verifying : audit-libs-2.8.5-4.el7.x86_64 1/17 Verifying : policycoreutils-python-2.5-33.el7.x86_64 2/17 Verifying : audit-2.8.5-4.el7.x86_64 3/17 Verifying : 3:docker-ce-19.03.5-3.el7.x86_64 4/17 Verifying : audit-libs-python-2.8.5-4.el7.x86_64 5/17 Verifying : libsemanage-python-2.5-14.el7.x86_64 6/17 Verifying : 2:container-selinux-2.107-3.el7.noarch 7/17 Verifying : python-IPy-0.75-6.el7.noarch 8/17 Verifying : checkpolicy-2.5-8.el7.x86_64 9/17 Verifying : policycoreutils-2.5-33.el7.x86_64 10/17 Verifying : containerd.io-1.2.10-3.2.el7.x86_64 11/17 Verifying : 1:docker-ce-cli-19.03.5-3.el7.x86_64 12/17 Verifying : setools-libs-3.3.8-4.el7.x86_64 13/17 Verifying : libcgroup-0.41-21.el7.x86_64 14/17 Verifying : policycoreutils-2.5-29.el7.x86_64 15/17 Verifying : audit-2.8.4-4.el7.x86_64 16/17 Verifying : audit-libs-2.8.4-4.el7.x86_64 17/17 Installed: docker-ce.x86_64 3:19.03.5-3.el7 Dependency Installed: audit-libs-python.x86_64 0:2.8.5-4.el7 checkpolicy.x86_64 0:2.5-8.el7 container-selinux.noarch 2:2.107-3.el7 containerd.io.x86_64 0:1.2.10-3.2.el7 docker-ce-cli.x86_64 1:19.03.5-3.el7 libcgroup.x86_64 0:0.41-21.el7 libsemanage-python.x86_64 0:2.5-14.el7 policycoreutils-python.x86_64 0:2.5-33.el7 python-IPy.noarch 0:0.75-6.el7 setools-libs.x86_64 0:3.3.8-4.el7 Dependency Updated: audit.x86_64 0:2.8.5-4.el7 audit-libs.x86_64 0:2.8.5-4.el7 policycoreutils.x86_64 0:2.5-33.el7 Complete! [root@node203.yinzhengjie.org.cn ~]#
3>.修改docker的默认策略,重新将FORWARD链的默认策略设置为ACCEPT
docker自1.13版起会自动设置iptable的FORWARD默认策略为DROP，这可能会影响kubernetes集群依赖的报文转发功能,因此，需要在docker服务启动后，重新将FORWARD链的默认策略设置为ACCEPT。 如下图所示,修改"/usr/lib/systemd/system/docker.service"文件，在"ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock"一行之后新增一行"ExecStartPost=/usr/sbin/iptables -P FORWARD ACCEPT"(意思是docker服务器启动成功后会执行执行该命令)
4>.为docker设置代理(一旦我们为docker服务器设置代理后再去配置阿里云加速就没有多大意义了,因此我这里并没有配置镜像加速，因为默认会走代理服务器)
若要通过默认的k8s.gcr.io镜像仓库(实际上国内的网络是无法直接访问Google的资源)获取kubernetes系统组件的相关镜像,需要配置docker Unit File(/usr/lib/systemd/system/docker.service)中的Environment变量,为其定义合理的HTTPS_PROXY(说白了就是你自己的VPN服务器，如果没有VPN的话还是乖乖的使用阿里云镜像吧),当然我们访问跟本地网络或者node的IP时则无需代理，使用NO_PROXY指定不代理的网段，格式如下(具体配置参考下图): Environment="HTTPS_PROXY=PROTOCOL://HOST:PORT" Environment="NO_PROXY=127.0.0.0/8,172.200.0.0/21" 温馨提示: 如果你不想通过k8s.gcr.io镜像仓库下载那就可以直接忽略此步骤，因为阿里云已经帮咱们提前下载好k8s镜像啦，我们直接去阿里云下载即可，使用阿里云下载需要配置阿里云加速，具体步骤可参考:"https://www.cnblogs.com/yinzhengjie/p/12182645.html"
5>.启动docker
[root@master200.yinzhengjie.org.cn ~]# systemctl daemon-reload [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# systemctl start docker.service [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# systemctl enable docker.service Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service. [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# systemctl list-unit-files | grep docker docker.service enabled docker.socket disabled [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# systemctl status docker.service ● docker.service - Docker Application Container Engine Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2020-02-04 11:30:27 CST; 1min 4s ago Docs: https://docs.docker.com Main PID: 6182 (dockerd) CGroup: /system.slice/docker.service └─6182 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock Feb 04 11:30:27 master200.yinzhengjie.org.cn dockerd[6182]: time="2020-02-04T11:30:27.706191791+08:00" level=info msg="scheme "unix" not registered, fallback to default scheme" module=grpc Feb 04 11:30:27 master200.yinzhengjie.org.cn dockerd[6182]: time="2020-02-04T11:30:27.706203607+08:00" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock 0 <nil>}] <nil>}" module=grpc Feb 04 11:30:27 master200.yinzhengjie.org.cn dockerd[6182]: time="2020-02-04T11:30:27.706210087+08:00" level=info msg="ClientConn switching balancer to "pick_first"" module=grpc Feb 04 11:30:27 master200.yinzhengjie.org.cn dockerd[6182]: time="2020-02-04T11:30:27.732793779+08:00" level=info msg="Loading containers: start." Feb 04 11:30:27 master200.yinzhengjie.org.cn dockerd[6182]: time="2020-02-04T11:30:27.819420862+08:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a...rred IP address" Feb 04 11:30:27 master200.yinzhengjie.org.cn dockerd[6182]: time="2020-02-04T11:30:27.858229542+08:00" level=info msg="Loading containers: done." Feb 04 11:30:27 master200.yinzhengjie.org.cn dockerd[6182]: time="2020-02-04T11:30:27.871357290+08:00" level=info msg="Docker daemon" commit=633a0ea graphdriver(s)=overlay2 version=19.03.5 Feb 04 11:30:27 master200.yinzhengjie.org.cn dockerd[6182]: time="2020-02-04T11:30:27.871433828+08:00" level=info msg="Daemon has completed initialization" Feb 04 11:30:27 master200.yinzhengjie.org.cn dockerd[6182]: time="2020-02-04T11:30:27.893618790+08:00" level=info msg="API listen on /var/run/docker.sock" Feb 04 11:30:27 master200.yinzhengjie.org.cn systemd[1]: Started Docker Application Container Engine. Hint: Some lines were ellipsized, use -l to show in full. [root@master200.yinzhengjie.org.cn ~]#
6>.查看docker的代理信息
[root@master200.yinzhengjie.org.cn ~]# docker info Client: Debug Mode: false Server: Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 0 Server Version: 19.03.5 Storage Driver: overlay2 Backing Filesystem: xfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657 init version: fec3683 Security Options: seccomp Profile: default Kernel Version: 3.10.0-957.el7.x86_64 Operating System: CentOS Linux 7 (Core) OSType: linux Architecture: x86_64 CPUs: 2 Total Memory: 3.84GiB Name: master200.yinzhengjie.org.cn ID: CSHT:CVB3:JFYB:GQ77:FYDH:X3UJ:B2SH:5WEX:5QNE:NABW:MJZM:K2T2 Docker Root Dir: /var/lib/docker Debug Mode: false HTTPS Proxy: http://www.yinzhengjie.org.cn:10086 No Proxy: 127.0.0.0/8,172.200.0.0/21 Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false [root@master200.yinzhengjie.org.cn ~]#
7>.查看防火墙FORWARD链的默认策略
[root@master200.yinzhengjie.org.cn ~]# iptables -vnL Chain INPUT (policy ACCEPT 247 packets, 17304 bytes) pkts bytes target prot opt in out source destination Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 0 0 DOCKER-USER all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 DOCKER-ISOLATION-STAGE-1 all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT all -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED 0 0 DOCKER all -- * docker0 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT all -- docker0 docker0 0.0.0.0/0 0.0.0.0/0 Chain OUTPUT (policy ACCEPT 147 packets, 18292 bytes) pkts bytes target prot opt in out source destination Chain DOCKER (1 references) pkts bytes target prot opt in out source destination Chain DOCKER-ISOLATION-STAGE-1 (1 references) pkts bytes target prot opt in out source destination 0 0 DOCKER-ISOLATION-STAGE-2 all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0 0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 Chain DOCKER-ISOLATION-STAGE-2 (1 references) pkts bytes target prot opt in out source destination 0 0 DROP all -- * docker0 0.0.0.0/0 0.0.0.0/0 0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 Chain DOCKER-USER (1 references) pkts bytes target prot opt in out source destination 0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
8>.将master200的docker文件拷贝到其它节点启动docker并设置为开机自启动
[root@master200.yinzhengjie.org.cn ~]# scp /usr/lib/systemd/system/docker.service node201.yinzhengjie.org.cn:/usr/lib/systemd/system/docker.service root@node201.yinzhengjie.org.cn's password: docker.service 100% 1846 1.2MB/s 00:00 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# scp /usr/lib/systemd/system/docker.service node202.yinzhengjie.org.cn:/usr/lib/systemd/system/docker.service root@node202.yinzhengjie.org.cn's password: docker.service 100% 1846 1.5MB/s 00:00 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# scp /usr/lib/systemd/system/docker.service node203.yinzhengjie.org.cn:/usr/lib/systemd/system/docker.service root@node203.yinzhengjie.org.cn's password: docker.service 100% 1846 790.6KB/s 00:00 [root@master200.yinzhengjie.org.cn ~]#
9>.启动docker后验证的网络相关的内核参数(了解一下即可，如果你遇到了类似问题安装下面的方法解决即可)
[root@master200.yinzhengjie.org.cn ~]# sysctl -a | grep bridge net.bridge.bridge-nf-call-arptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-filter-pppoe-tagged = 0 net.bridge.bridge-nf-filter-vlan-tagged = 0 net.bridge.bridge-nf-pass-vlan-input-dev = 0 sysctl: reading key "net.ipv6.conf.all.stable_secret" sysctl: reading key "net.ipv6.conf.bond0.stable_secret" sysctl: reading key "net.ipv6.conf.bond1.stable_secret" sysctl: reading key "net.ipv6.conf.default.stable_secret" sysctl: reading key "net.ipv6.conf.docker0.stable_secret" sysctl: reading key "net.ipv6.conf.eth0.stable_secret" sysctl: reading key "net.ipv6.conf.eth1.stable_secret" sysctl: reading key "net.ipv6.conf.eth2.stable_secret" sysctl: reading key "net.ipv6.conf.eth3.stable_secret" sysctl: reading key "net.ipv6.conf.lo.stable_secret" [root@master200.yinzhengjie.org.cn ~]# [root@node203.yinzhengjie.org.cn ~]# cat /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 [root@node203.yinzhengjie.org.cn ~]# [root@node203.yinzhengjie.org.cn ~]# sysctl -p /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 [root@node203.yinzhengjie.org.cn ~]#
三.安装kubernetes相关的软件包并初始化集群
1>.编写k8s的软件源仓库的配置文件
[root@master200.yinzhengjie.org.cn ~]# vim /etc/yum.repos.d/kubernetes.repo [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# cat /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Yinzhengjie's Kubernetes Repository baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ gpgcheck=1 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# yum repolist Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirror.bit.edu.cn * extras: mirrors.tuna.tsinghua.edu.cn * updates: mirrors.tuna.tsinghua.edu.cn kubernetes | 1.4 kB 00:00:00 kubernetes/primary | 63 kB 00:00:00 kubernetes 460/460 repo id repo name status base/7/x86_64 CentOS-7 - Base 10,097 docker-ce-stable/x86_64 Docker CE Stable - x86_64 63 extras/7/x86_64 CentOS-7 - Extras 323 kubernetes Yinzhengjie's Kubernetes Repository 460 updates/7/x86_64 CentOS-7 - Updates 1,117 repolist: 12,060 [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# yum list all | grep "^kube" kubeadm.x86_64 1.17.2-0 kubernetes kubectl.x86_64 1.17.2-0 kubernetes kubelet.x86_64 1.17.2-0 kubernetes kubernetes.x86_64 1.5.2-0.7.git269f928.el7 extras kubernetes-client.x86_64 1.5.2-0.7.git269f928.el7 extras kubernetes-cni.x86_64 0.7.5-0 kubernetes kubernetes-master.x86_64 1.5.2-0.7.git269f928.el7 extras kubernetes-node.x86_64 1.5.2-0.7.git269f928.el7 extras [root@master200.yinzhengjie.org.cn ~]#
2>.安装软件包并查看安装的版本
[root@master200.yinzhengjie.org.cn ~]# yum -y install kubeadm kubectl kubelet Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirror.bit.edu.cn * extras: mirrors.tuna.tsinghua.edu.cn * updates: mirrors.tuna.tsinghua.edu.cn Resolving Dependencies --> Running transaction check ---> Package kubeadm.x86_64 0:1.17.2-0 will be installed --> Processing Dependency: kubernetes-cni >= 0.7.5 for package: kubeadm-1.17.2-0.x86_64 --> Processing Dependency: cri-tools >= 1.13.0 for package: kubeadm-1.17.2-0.x86_64 ---> Package kubectl.x86_64 0:1.17.2-0 will be installed ---> Package kubelet.x86_64 0:1.17.2-0 will be installed --> Processing Dependency: socat for package: kubelet-1.17.2-0.x86_64 --> Processing Dependency: conntrack for package: kubelet-1.17.2-0.x86_64 --> Running transaction check ---> Package conntrack-tools.x86_64 0:1.4.4-5.el7_7.2 will be installed --> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.1)(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.0)(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_cthelper.so.0(LIBNETFILTER_CTHELPER_1.0)(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_queue.so.1()(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_cttimeout.so.1()(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_cthelper.so.0()(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 ---> Package cri-tools.x86_64 0:1.13.0-0 will be installed ---> Package kubernetes-cni.x86_64 0:0.7.5-0 will be installed ---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed --> Running transaction check ---> Package libnetfilter_cthelper.x86_64 0:1.0.0-10.el7_7.1 will be installed ---> Package libnetfilter_cttimeout.x86_64 0:1.0.0-6.el7_7.1 will be installed ---> Package libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 will be installed --> Finished Dependency Resolution Dependencies Resolved ========================================================================================================================================================================================================================================================== Package Arch Version Repository Size ========================================================================================================================================================================================================================================================== Installing: kubeadm x86_64 1.17.2-0 kubernetes 8.7 M kubectl x86_64 1.17.2-0 kubernetes 9.4 M kubelet x86_64 1.17.2-0 kubernetes 20 M Installing for dependencies: conntrack-tools x86_64 1.4.4-5.el7_7.2 updates 187 k cri-tools x86_64 1.13.0-0 kubernetes 5.1 M kubernetes-cni x86_64 0.7.5-0 kubernetes 10 M libnetfilter_cthelper x86_64 1.0.0-10.el7_7.1 updates 18 k libnetfilter_cttimeout x86_64 1.0.0-6.el7_7.1 updates 18 k libnetfilter_queue x86_64 1.0.2-2.el7_2 base 23 k socat x86_64 1.7.3.2-2.el7 base 290 k Transaction Summary ========================================================================================================================================================================================================================================================== Install 3 Packages (+7 Dependent packages) Total download size: 54 M Installed size: 243 M Downloading packages: (1/10): conntrack-tools-1.4.4-5.el7_7.2.x86_64.rpm | 187 kB 00:00:00 warning: /var/cache/yum/x86_64/7/kubernetes/packages/14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 3e1ba8d5: NOKEY ] 2.0 MB/s | 1.7 MB 00:00:26 ETA Public key for 14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm is not installed (2/10): 14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm | 5.1 MB 00:00:00 (3/10): b44630896c69cd411db53be1d5cb5ae899a40aba7c0766317ea904390fcfc45b-kubectl-1.17.2-0.x86_64.rpm | 9.4 MB 00:00:01 (4/10): 105d89f0607c7baf91305ba352e78000bd20aad5cdf706bffff3b31cd546dbf3-kubeadm-1.17.2-0.x86_64.rpm | 8.7 MB 00:00:02 (5/10): libnetfilter_cthelper-1.0.0-10.el7_7.1.x86_64.rpm | 18 kB 00:00:00 (6/10): libnetfilter_queue-1.0.2-2.el7_2.x86_64.rpm | 23 kB 00:00:00 (7/10): libnetfilter_cttimeout-1.0.0-6.el7_7.1.x86_64.rpm | 18 kB 00:00:00 (8/10): socat-1.7.3.2-2.el7.x86_64.rpm | 290 kB 00:00:01 (9/10): 3ee7f2dff78e6fbb3ac3af8acb1a907f4bec1b1ef4cf627cbe02fa553707f2e9-kubelet-1.17.2-0.x86_64.rpm | 20 MB 00:00:02 (10/10): 548a0dcd865c16a50980420ddfa5fbccb8b59621179798e6dc905c9bf8af3b34-kubernetes-cni-0.7.5-0.x86_64.rpm | 10 MB 00:00:03 ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 9.1 MB/s | 54 MB 00:00:05 Retrieving key from https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg Importing GPG key 0x3E1BA8D5: Userid : "Google Cloud Packages RPM Signing Key <gc-team@google.com>" Fingerprint: 3749 e1ba 95a8 6ce0 5454 6ed2 f09c 394c 3e1b a8d5 From : https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg Retrieving key from https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg Importing GPG key 0xA7317B0F: Userid : "Google Cloud Packages Automatic Signing Key <gc-team@google.com>" Fingerprint: d0bc 747f d8ca f711 7500 d6fa 3746 c208 a731 7b0f From : https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : libnetfilter_cttimeout-1.0.0-6.el7_7.1.x86_64 1/10 Installing : socat-1.7.3.2-2.el7.x86_64 2/10 Installing : cri-tools-1.13.0-0.x86_64 3/10 Installing : kubectl-1.17.2-0.x86_64 4/10 Installing : libnetfilter_queue-1.0.2-2.el7_2.x86_64 5/10 Installing : libnetfilter_cthelper-1.0.0-10.el7_7.1.x86_64 6/10 Installing : conntrack-tools-1.4.4-5.el7_7.2.x86_64 7/10 Installing : kubernetes-cni-0.7.5-0.x86_64 8/10 Installing : kubelet-1.17.2-0.x86_64 9/10 Installing : kubeadm-1.17.2-0.x86_64 10/10 Verifying : libnetfilter_cthelper-1.0.0-10.el7_7.1.x86_64 1/10 Verifying : conntrack-tools-1.4.4-5.el7_7.2.x86_64 2/10 Verifying : libnetfilter_queue-1.0.2-2.el7_2.x86_64 3/10 Verifying : kubectl-1.17.2-0.x86_64 4/10 Verifying : cri-tools-1.13.0-0.x86_64 5/10 Verifying : kubeadm-1.17.2-0.x86_64 6/10 Verifying : kubernetes-cni-0.7.5-0.x86_64 7/10 Verifying : socat-1.7.3.2-2.el7.x86_64 8/10 Verifying : kubelet-1.17.2-0.x86_64 9/10 Verifying : libnetfilter_cttimeout-1.0.0-6.el7_7.1.x86_64 10/10 Installed: kubeadm.x86_64 0:1.17.2-0 kubectl.x86_64 0:1.17.2-0 kubelet.x86_64 0:1.17.2-0 Dependency Installed: conntrack-tools.x86_64 0:1.4.4-5.el7_7.2 cri-tools.x86_64 0:1.13.0-0 kubernetes-cni.x86_64 0:0.7.5-0 libnetfilter_cthelper.x86_64 0:1.0.0-10.el7_7.1 libnetfilter_cttimeout.x86_64 0:1.0.0-6.el7_7.1 libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 socat.x86_64 0:1.7.3.2-2.el7 Complete! [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# rpm -ql kubelet /etc/kubernetes/manifests /etc/sysconfig/kubelet /usr/bin/kubelet /usr/lib/systemd/system/kubelet.service [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# rpm -ql kubeadm /usr/bin/kubeadm /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# rpm -ql kubectl /usr/bin/kubectl [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# rpm -q kubeadm kubeadm-1.17.2-0.x86_64 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# rpm -q kubectl kubectl-1.17.2-0.x86_64 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# rpm -q kubelet kubelet-1.17.2-0.x86_64 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# systemctl list-unit-files | grep kubelet kubelet.service disabled [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# systemctl status kubelet ● kubelet.service - kubelet: The Kubernetes Node Agent Loaded: loaded (/usr/lib/systemd/system/kubelet.service; disabled; vendor preset: disabled) Drop-In: /usr/lib/systemd/system/kubelet.service.d └─10-kubeadm.conf Active: inactive (dead) Docs: https://kubernetes.io/docs/ [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# systemctl enable kubelet Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service. [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# systemctl list-unit-files | grep kubelet kubelet.service enabled [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# systemctl start kubelet [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# systemctl status kubelet ● kubelet.service - kubelet: The Kubernetes Node Agent Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled) Drop-In: /usr/lib/systemd/system/kubelet.service.d └─10-kubeadm.conf Active: active (running) since Wed 2020-02-05 05:12:34 CST; 2s ago Docs: https://kubernetes.io/docs/ Main PID: 5675 (kubelet) Tasks: 11 Memory: 117.3M CGroup: /system.slice/kubelet.service └─5675 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-im... Feb 05 05:12:35 master200.yinzhengjie.org.cn kubelet[5675]: I0205 05:12:35.006504 5675 remote_image.go:50] parsed scheme: "" Feb 05 05:12:35 master200.yinzhengjie.org.cn kubelet[5675]: I0205 05:12:35.006511 5675 remote_image.go:50] scheme "" not registered, fallback to default scheme Feb 05 05:12:35 master200.yinzhengjie.org.cn kubelet[5675]: I0205 05:12:35.006519 5675 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/var/run/dockershim.sock 0 <nil>}] <nil>} Feb 05 05:12:35 master200.yinzhengjie.org.cn kubelet[5675]: I0205 05:12:35.006523 5675 clientconn.go:577] ClientConn switching balancer to "pick_first" Feb 05 05:12:35 master200.yinzhengjie.org.cn kubelet[5675]: E0205 05:12:35.933510 5675 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/kubelet.go:458: Failed to list *v1.Node: Get https://172.200.1.200:6443/api/v1/nodes?fieldSelector=metadata.n... Feb 05 05:12:35 master200.yinzhengjie.org.cn kubelet[5675]: E0205 05:12:35.934154 5675 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/kubelet.go:449: Failed to list *v1.Service: Get https://172.200.1.200:6443/api/v1/services?li...nnection refused Feb 05 05:12:35 master200.yinzhengjie.org.cn kubelet[5675]: E0205 05:12:35.935345 5675 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://172.200.1.200:6443/api/v1/pods?fieldSelector=spec... Feb 05 05:12:36 master200.yinzhengjie.org.cn kubelet[5675]: E0205 05:12:36.934349 5675 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/kubelet.go:458: Failed to list *v1.Node: Get https://172.200.1.200:6443/api/v1/nodes?fieldSelector=metadata.n... Feb 05 05:12:36 master200.yinzhengjie.org.cn kubelet[5675]: E0205 05:12:36.935197 5675 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/kubelet.go:449: Failed to list *v1.Service: Get https://172.200.1.200:6443/api/v1/services?li...nnection refused Feb 05 05:12:36 master200.yinzhengjie.org.cn kubelet[5675]: E0205 05:12:36.937286 5675 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://172.200.1.200:6443/api/v1/pods?fieldSelector=spec... Hint: Some lines were ellipsized, use -l to show in full. [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
3>.设置忽略swap启用的状态错误(如果你未禁用swap设备则需要编辑kubelet的配置文件，我们在安装操作系统时就压根没有分配swap分区,因此该步骤直接忽略即可)
[root@master200.yinzhengjie.org.cn ~]# vim /etc/sysconfig/kubelet [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# cat /etc/sysconfig/kubelet KUBELET_EXTRA_ARGS="--fail-swap-on=false" #该参数表示尽管swap分区是启用的也不报错，如果不设置默认情况下如果你的服务器启用了swap分区那么初始化时会报错的 [root@master200.yinzhengjie.org.cn ~]#
4>.先将镜像文件下载到本地
[root@master200.yinzhengjie.org.cn ~]# kubeadm config print init-defaults W0204 12:49:25.072012 10194 validation.go:28] Cannot validate kubelet config - no validator is available W0204 12:49:25.072058 10194 validation.go:28] Cannot validate kube-proxy config - no validator is available apiVersion: kubeadm.k8s.io/v1beta2 bootstrapTokens: - groups: - system:bootstrappers:kubeadm:default-node-token token: abcdef.0123456789abcdef ttl: 24h0m0s usages: - signing - authentication kind: InitConfiguration localAPIEndpoint: advertiseAddress: 1.2.3.4 bindPort: 6443 nodeRegistration: criSocket: /var/run/dockershim.sock name: master200.yinzhengjie.org.cn taints: - effect: NoSchedule key: node-role.kubernetes.io/master --- apiServer: timeoutForControlPlane: 4m0s apiVersion: kubeadm.k8s.io/v1beta2 certificatesDir: /etc/kubernetes/pki clusterName: kubernetes controllerManager: {} dns: type: CoreDNS etcd: local: dataDir: /var/lib/etcd imageRepository: k8s.gcr.io kind: ClusterConfiguration kubernetesVersion: v1.17.0 networking: dnsDomain: cluster.local serviceSubnet: 10.96.0.0/12 scheduler: {} [root@master200.yinzhengjie.org.cn ~]#
如果你有对docker配置了代理(即指定了VPN服务器)那么就可以使用Google官方提供的镜像仓库进行下载,执行如下命令即可: [root@master200.yinzhengjie.org.cn ~]# kubeadm config images pull 如上图所示,默认情况下是无法直接访问Google的镜像仓库"k8s.gcr.io"，此时咱们有三种解决方案: (1)在互联网上找一些免费的VPN进行翻墙操作或者在网上购买收费的VPN，推荐使用收费版本的，因为免费版本的VPN并不太稳定而且下载速度也并不是很理想，付费的VPN相对来说稳定性比较强,但是这种情况存在一个风险，因为你的所有操作都会被提供VPN的服务商在后台记录; (2)在互联网上购买VPS服务器，自己搭建VPN，前提是你购买的VPS可以访问国外的各种网站，直接购买国内的阿里云服务器可以你搭建出来VPN也是白搭,这种方案也需要掏钱，但相比于上面的那种方式安全性较强，因为你访问的记录在你的VPS中有记录，你想干啥就干啥; (3)我们的目的就是下载Google的镜像而已,可以在其它国内站点下载即可，而且很多国人已经很热心的提供了下载连接,如果你没有经常翻墙的需求其实没有必要采取上面两种方案(而且有很多公司也会为员工购买翻墙的工具，自己搭建VPN服务器的一般就是一些运维或开发人员喜欢干这事)。 本篇博客采取第三种方案，直接在国内的其它网站(比如阿里云的镜像网站)下载需要的镜像，下载后给镜像重新更名即可，具体操作请参考我笔记: K8S镜像下载报错解决方案(使用阿里云镜像去下载kubeadm需要的镜像文件):https://www.cnblogs.com/yinzhengjie/p/12258215.html 镜像成功下载到本地后,可以直接使用docker命令查看本地镜像，如下图所示。
5>.初始化集群(初始化过程会自动去下载镜像，但上一步骤我们已经把镜像提前下载到本地啦，因此不会去官网下载镜像啦,这一步骤也会快很多)
[root@master200.yinzhengjie.org.cn ~]# kubeadm init --kubernetes-version="v1.17.2" --pod-network-cidr="10.244.0.0/16" --dry-run W0204 19:37:26.784651 31963 validation.go:28] Cannot validate kube-proxy config - no validator is available W0204 19:37:26.784688 31963 validation.go:28] Cannot validate kubelet config - no validator is available [init] Using Kubernetes version: v1.17.2 [preflight] Running pre-flight checks [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/ [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service' [preflight] Would pull the required images (like 'kubeadm config images pull') [kubelet-start] Writing kubelet environment file with flags to file "/etc/kubernetes/tmp/kubeadm-init-dryrun988930826/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/etc/kubernetes/tmp/kubeadm-init-dryrun988930826/config.yaml" [certs] Using certificateDir folder "/etc/kubernetes/tmp/kubeadm-init-dryrun988930826" [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [master200.yinzhengjie.org.cn kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.200.1.200] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [master200.yinzhengjie.org.cn localhost] and IPs [172.200.1.200 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [master200.yinzhengjie.org.cn localhost] and IPs [172.200.1.200 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes/tmp/kubeadm-init-dryrun988930826" [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "kubelet.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file [control-plane] Using manifest folder "/etc/kubernetes/tmp/kubeadm-init-dryrun988930826" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" W0204 19:37:28.996203 31963 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC" [control-plane] Creating static Pod manifest for "kube-scheduler" W0204 19:37:28.997161 31963 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC" [dryrun] Would ensure that "/var/lib/etcd" directory is present [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/tmp/kubeadm-init-dryrun988930826" [dryrun] Wrote certificates, kubeconfig files and control plane manifests to the "/etc/kubernetes/tmp/kubeadm-init-dryrun988930826" directory [dryrun] The certificates or kubeconfig files would not be printed due to their sensitive nature [dryrun] Please examine the "/etc/kubernetes/tmp/kubeadm-init-dryrun988930826" directory for details about what would be written [dryrun] Would write file "/etc/kubernetes/manifests/kube-apiserver.yaml" with content: apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: component: kube-apiserver tier: control-plane name: kube-apiserver namespace: kube-system spec: containers: - command: - kube-apiserver - --advertise-address=172.200.1.200 - --allow-privileged=true - --authorization-mode=Node,RBAC - --client-ca-file=/etc/kubernetes/pki/ca.crt - --enable-admission-plugins=NodeRestriction - --enable-bootstrap-token-auth=true - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key - --etcd-servers=https://127.0.0.1:2379 - --insecure-port=0 - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key - --requestheader-allowed-names=front-proxy-client - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt - --requestheader-extra-headers-prefix=X-Remote-Extra- - --requestheader-group-headers=X-Remote-Group - --requestheader-username-headers=X-Remote-User - --secure-port=6443 - --service-account-key-file=/etc/kubernetes/pki/sa.pub - --service-cluster-ip-range=10.96.0.0/12 - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key image: k8s.gcr.io/kube-apiserver:v1.17.2 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 8 httpGet: host: 172.200.1.200 path: /healthz port: 6443 scheme: HTTPS initialDelaySeconds: 15 timeoutSeconds: 15 name: kube-apiserver resources: requests: cpu: 250m volumeMounts: - mountPath: /etc/ssl/certs name: ca-certs readOnly: true - mountPath: /etc/pki name: etc-pki readOnly: true - mountPath: /etc/kubernetes/pki name: k8s-certs readOnly: true hostNetwork: true priorityClassName: system-cluster-critical volumes: - hostPath: path: /etc/ssl/certs type: DirectoryOrCreate name: ca-certs - hostPath: path: /etc/pki type: DirectoryOrCreate name: etc-pki - hostPath: path: /etc/kubernetes/pki type: DirectoryOrCreate name: k8s-certs status: {} [dryrun] Would write file "/etc/kubernetes/manifests/kube-controller-manager.yaml" with content: apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: component: kube-controller-manager tier: control-plane name: kube-controller-manager namespace: kube-system spec: containers: - command: - kube-controller-manager - --allocate-node-cidrs=true - --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf - --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf - --bind-address=127.0.0.1 - --client-ca-file=/etc/kubernetes/pki/ca.crt - --cluster-cidr=10.244.0.0/16 - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key - --controllers=*,bootstrapsigner,tokencleaner - --kubeconfig=/etc/kubernetes/controller-manager.conf - --leader-elect=true - --node-cidr-mask-size=24 - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt - --root-ca-file=/etc/kubernetes/pki/ca.crt - --service-account-private-key-file=/etc/kubernetes/pki/sa.key - --service-cluster-ip-range=10.96.0.0/12 - --use-service-account-credentials=true image: k8s.gcr.io/kube-controller-manager:v1.17.2 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 8 httpGet: host: 127.0.0.1 path: /healthz port: 10257 scheme: HTTPS initialDelaySeconds: 15 timeoutSeconds: 15 name: kube-controller-manager resources: requests: cpu: 200m volumeMounts: - mountPath: /etc/ssl/certs name: ca-certs readOnly: true - mountPath: /etc/pki name: etc-pki readOnly: true - mountPath: /usr/libexec/kubernetes/kubelet-plugins/volume/exec name: flexvolume-dir - mountPath: /etc/kubernetes/pki name: k8s-certs readOnly: true - mountPath: /etc/kubernetes/controller-manager.conf name: kubeconfig readOnly: true hostNetwork: true priorityClassName: system-cluster-critical volumes: - hostPath: path: /etc/ssl/certs type: DirectoryOrCreate name: ca-certs - hostPath: path: /etc/pki type: DirectoryOrCreate name: etc-pki - hostPath: path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec type: DirectoryOrCreate name: flexvolume-dir - hostPath: path: /etc/kubernetes/pki type: DirectoryOrCreate name: k8s-certs - hostPath: path: /etc/kubernetes/controller-manager.conf type: FileOrCreate name: kubeconfig status: {} [dryrun] Would write file "/etc/kubernetes/manifests/kube-scheduler.yaml" with content: apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: component: kube-scheduler tier: control-plane name: kube-scheduler namespace: kube-system spec: containers: - command: - kube-scheduler - --authentication-kubeconfig=/etc/kubernetes/scheduler.conf - --authorization-kubeconfig=/etc/kubernetes/scheduler.conf - --bind-address=127.0.0.1 - --kubeconfig=/etc/kubernetes/scheduler.conf - --leader-elect=true image: k8s.gcr.io/kube-scheduler:v1.17.2 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 8 httpGet: host: 127.0.0.1 path: /healthz port: 10259 scheme: HTTPS initialDelaySeconds: 15 timeoutSeconds: 15 name: kube-scheduler resources: requests: cpu: 100m volumeMounts: - mountPath: /etc/kubernetes/scheduler.conf name: kubeconfig readOnly: true hostNetwork: true priorityClassName: system-cluster-critical volumes: - hostPath: path: /etc/kubernetes/scheduler.conf type: FileOrCreate name: kubeconfig status: {} [dryrun] Would write file "/var/lib/kubelet/config.yaml" with content: apiVersion: kubelet.config.k8s.io/v1beta1 authentication: anonymous: enabled: false webhook: cacheTTL: 0s enabled: true x509: clientCAFile: /etc/kubernetes/pki/ca.crt authorization: mode: Webhook webhook: cacheAuthorizedTTL: 0s cacheUnauthorizedTTL: 0s clusterDNS: - 10.96.0.10 clusterDomain: cluster.local cpuManagerReconcilePeriod: 0s evictionPressureTransitionPeriod: 0s fileCheckFrequency: 0s healthzBindAddress: 127.0.0.1 healthzPort: 10248 httpCheckFrequency: 0s imageMinimumGCAge: 0s kind: KubeletConfiguration nodeStatusReportFrequency: 0s nodeStatusUpdateFrequency: 0s rotateCertificates: true runtimeRequestTimeout: 0s staticPodPath: /etc/kubernetes/manifests streamingConnectionIdleTimeout: 0s syncFrequency: 0s volumeStatsAggPeriod: 0s [dryrun] Would write file "/var/lib/kubelet/kubeadm-flags.env" with content: KUBELET_KUBEADM_ARGS="--cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1" [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/tmp/kubeadm-init-dryrun988930826". This can take up to 4m0s [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [dryrun] Would perform action CREATE on resource "configmaps" in API group "core/v1" [dryrun] Attached object: apiVersion: v1 data: ClusterConfiguration: | apiServer: extraArgs: authorization-mode: Node,RBAC timeoutForControlPlane: 4m0s apiVersion: kubeadm.k8s.io/v1beta2 certificatesDir: /etc/kubernetes/pki clusterName: kubernetes controllerManager: {} dns: type: CoreDNS etcd: local: dataDir: /var/lib/etcd imageRepository: k8s.gcr.io kind: ClusterConfiguration kubernetesVersion: v1.17.2 networking: dnsDomain: cluster.local podSubnet: 10.244.0.0/16 serviceSubnet: 10.96.0.0/12 scheduler: {} ClusterStatus: | apiEndpoints: master200.yinzhengjie.org.cn: advertiseAddress: 172.200.1.200 bindPort: 6443 apiVersion: kubeadm.k8s.io/v1beta2 kind: ClusterStatus kind: ConfigMap metadata: creationTimestamp: null name: kubeadm-config namespace: kube-system [dryrun] Would perform action CREATE on resource "roles" in API group "rbac.authorization.k8s.io/v1" [dryrun] Attached object: apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: creationTimestamp: null name: kubeadm:nodes-kubeadm-config namespace: kube-system rules: - apiGroups: - "" resourceNames: - kubeadm-config resources: - configmaps verbs: - get [dryrun] Would perform action CREATE on resource "rolebindings" in API group "rbac.authorization.k8s.io/v1" [dryrun] Attached object: apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: creationTimestamp: null name: kubeadm:nodes-kubeadm-config namespace: kube-system roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: kubeadm:nodes-kubeadm-config subjects: - kind: Group name: system:bootstrappers:kubeadm:default-node-token - kind: Group name: system:nodes [kubelet] Creating a ConfigMap "kubelet-config-1.17" in namespace kube-system with the configuration for the kubelets in the cluster [dryrun] Would perform action CREATE on resource "configmaps" in API group "core/v1" [dryrun] Attached object: apiVersion: v1 data: kubelet: | apiVersion: kubelet.config.k8s.io/v1beta1 authentication: anonymous: enabled: false webhook: cacheTTL: 0s enabled: true x509: clientCAFile: /etc/kubernetes/pki/ca.crt authorization: mode: Webhook webhook: cacheAuthorizedTTL: 0s cacheUnauthorizedTTL: 0s clusterDNS: - 10.96.0.10 clusterDomain: cluster.local cpuManagerReconcilePeriod: 0s evictionPressureTransitionPeriod: 0s fileCheckFrequency: 0s healthzBindAddress: 127.0.0.1 healthzPort: 10248 httpCheckFrequency: 0s imageMinimumGCAge: 0s kind: KubeletConfiguration nodeStatusReportFrequency: 0s nodeStatusUpdateFrequency: 0s rotateCertificates: true runtimeRequestTimeout: 0s staticPodPath: /etc/kubernetes/manifests streamingConnectionIdleTimeout: 0s syncFrequency: 0s volumeStatsAggPeriod: 0s kind: ConfigMap metadata: creationTimestamp: null name: kubelet-config-1.17 namespace: kube-system [dryrun] Would perform action CREATE on resource "roles" in API group "rbac.authorization.k8s.io/v1" [dryrun] Attached object: apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: creationTimestamp: null name: kubeadm:kubelet-config-1.17 namespace: kube-system rules: - apiGroups: - "" resourceNames: - kubelet-config-1.17 resources: - configmaps verbs: - get [dryrun] Would perform action CREATE on resource "rolebindings" in API group "rbac.authorization.k8s.io/v1" [dryrun] Attached object: apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: creationTimestamp: null name: kubeadm:kubelet-config-1.17 namespace: kube-system roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: kubeadm:kubelet-config-1.17 subjects: - kind: Group name: system:nodes - kind: Group name: system:bootstrappers:kubeadm:default-node-token [dryrun] Would perform action GET on resource "nodes" in API group "core/v1" [dryrun] Resource name: "master200.yinzhengjie.org.cn" [dryrun] Would perform action PATCH on resource "nodes" in API group "core/v1" [dryrun] Resource name: "master200.yinzhengjie.org.cn" [dryrun] Attached patch: {"metadata":{"annotations":{"kubeadm.alpha.kubernetes.io/cri-socket":"/var/run/dockershim.sock"}}} [upload-certs] Skipping phase. Please see --upload-certs [mark-control-plane] Marking the node master200.yinzhengjie.org.cn as control-plane by adding the label "node-role.kubernetes.io/master=''" [mark-control-plane] Marking the node master200.yinzhengjie.org.cn as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] [dryrun] Would perform action GET on resource "nodes" in API group "core/v1" [dryrun] Resource name: "master200.yinzhengjie.org.cn" [dryrun] Would perform action PATCH on resource "nodes" in API group "core/v1" [dryrun] Resource name: "master200.yinzhengjie.org.cn" [dryrun] Attached patch: {"metadata":{"labels":{"node-role.kubernetes.io/master":""}},"spec":{"taints":[{"effect":"NoSchedule","key":"node-role.kubernetes.io/master"}]}} [bootstrap-token] Using token: dp9kc3.s7ceexx1xblkqklh [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [dryrun] Would perform action GET on resource "secrets" in API group "core/v1" [dryrun] Resource name: "bootstrap-token-dp9kc3" [dryrun] Would perform action CREATE on resource "secrets" in API group "core/v1" [dryrun] Attached object: apiVersion: v1 data: auth-extra-groups: c3lzdGVtOmJvb3RzdHJhcHBlcnM6a3ViZWFkbTpkZWZhdWx0LW5vZGUtdG9rZW4= description: VGhlIGRlZmF1bHQgYm9vdHN0cmFwIHRva2VuIGdlbmVyYXRlZCBieSAna3ViZWFkbSBpbml0Jy4= expiration: MjAyMC0wMi0wNVQxOTozNzozMCswODowMA== token-id: ZHA5a2Mz token-secret: czdjZWV4eDF4YmxrcWtsaA== usage-bootstrap-authentication: dHJ1ZQ== usage-bootstrap-signing: dHJ1ZQ== kind: Secret metadata: creationTimestamp: null name: bootstrap-token-dp9kc3 namespace: kube-system type: bootstrap.kubernetes.io/token [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [dryrun] Would perform action CREATE on resource "clusterrolebindings" in API group "rbac.authorization.k8s.io/v1" [dryrun] Attached object: apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: creationTimestamp: null name: kubeadm:kubelet-bootstrap roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:node-bootstrapper subjects: - kind: Group name: system:bootstrappers:kubeadm:default-node-token [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [dryrun] Would perform action CREATE on resource "clusterrolebindings" in API group "rbac.authorization.k8s.io/v1" [dryrun] Attached object: apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: creationTimestamp: null name: kubeadm:node-autoapprove-bootstrap roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:certificates.k8s.io:certificatesigningrequests:nodeclient subjects: - kind: Group name: system:bootstrappers:kubeadm:default-node-token [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [dryrun] Would perform action CREATE on resource "clusterrolebindings" in API group "rbac.authorization.k8s.io/v1" [dryrun] Attached object: apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: creationTimestamp: null name: kubeadm:node-autoapprove-certificate-rotation roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient subjects: - kind: Group name: system:nodes [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [dryrun] Would perform action CREATE on resource "configmaps" in API group "core/v1" [dryrun] Attached object: apiVersion: v1 data: kubeconfig: | apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJd01ESXdOREV4TXpjeU4xb1hEVE13TURJd01URXhNemN5TjFvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSn VaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBSjgrCmhvL1ptZkNhSTJITDJEam5wZk9kT1l2QnNxQTF0YmtNT1NGKzdNOTFHQ1FWZE9STG5IbUFRZTZQRExIbjQ5TUEKbmVyemNLTnp6S203T0liNmdDRlFyRERjYUwvaUV0VWlFQWhPQXd3ZTIwTTAzQlFWblptY3VlcVFqbE1BSUVrdApNamRRT3lhYnluUkt5bHUwSXZPRHJaeUE5Mjc2dEFFSWNnZlFEVEUrU2hwSmU1aUIveDNXOTMwZGN1T3F1cVgvClZvRzkxcHVDdjNGSlF5aEdBSE91emhQMmtodmVuemdkZXEyc2RoUlVReEU4OXJVU294Q3VBS2Z3TjVONGVaR0QKbWRVc0dHS0pGblRLOXA5YkpFNncxWGh1d1VqTmltZmdhc2RvbVBOMmxVaU84MFFCcDdWM3ZlNjNhbVVYRVhVWQpVYk1oUFVlL21mZ1hyQnJZME1FQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFCMXdtWW9tYzBXN210aEVlK0ErK0lkTGVrOEoKRUwyNnkyVXJFR3g5MVRkeTRvZFpCcVVpRzltNERmdTNlZlU2T0g3V0gyTG1RNGNXbWtLcitDdmJrRThRK3FHMwo3TGRvNTdzbHhwN2NyYU5tQ3dmb1ZIOCtLbVl4UUlGOWpxMGJ4RjlqRlNMSnJHa0N6cDBwMlhaQi9Tc1VFb3M4ClNsU0hvWEhuemQ2ekZSTmJLKzYyMFkvV1ZrU2h1V3VuRW5rV1kva0hHdTB6eHJqOUJiQ25yM0FETzVqZ0tyM08KN24wdmhnZzlJaE1Rb25lYldJTEdNVXV1b21VNlBvWDN3eHprOVJkSjV6T05samN6bXhVc2IvUFJEcnQ3bkFGUQpjYmdGclcrOUo1VmkyMFNMLzZmK1ZXZnV3aHNpVXFaU1VLWE10dE9OQitjYkRlVFVqYjN3MHErSjNPbz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= server: https://172.200.1.200:6443 name: "" contexts: null current-context: "" kind: Config preferences: {} users: null kind: ConfigMap metadata: creationTimestamp: null name: cluster-info namespace: kube-public [dryrun] Would perform action CREATE on resource "roles" in API group "rbac.authorization.k8s.io/v1" [dryrun] Attached object: apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: creationTimestamp: null name: kubeadm:bootstrap-signer-clusterinfo namespace: kube-public rules: - apiGroups: - "" resourceNames: - cluster-info resources: - configmaps verbs: - get [dryrun] Would perform action CREATE on resource "rolebindings" in API group "rbac.authorization.k8s.io/v1" [dryrun] Attached object: apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: creationTimestamp: null name: kubeadm:bootstrap-signer-clusterinfo namespace: kube-public roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: kubeadm:bootstrap-signer-clusterinfo subjects: - kind: User name: system:anonymous [dryrun] Would perform action GET on resource "configmaps" in API group "core/v1" [dryrun] Resource name: "kube-dns" [dryrun] Would perform action GET on resource "configmaps" in API group "core/v1" [dryrun] Resource name: "coredns" [dryrun] Would perform action CREATE on resource "configmaps" in API group "core/v1" [dryrun] Attached object: apiVersion: v1 data: Corefile: | .:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa ttl 30 } prometheus :9153 forward . /etc/resolv.conf cache 30 loop reload loadbalance } kind: ConfigMap metadata: creationTimestamp: null name: coredns namespace: kube-system [dryrun] Would perform action CREATE on resource "clusterroles" in API group "rbac.authorization.k8s.io/v1" [dryrun] Attached object: apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: creationTimestamp: null name: system:coredns rules: - apiGroups: - "" resources: - endpoints - services - pods - namespaces verbs: - list - watch - apiGroups: - "" resources: - nodes verbs: - get [dryrun] Would perform action CREATE on resource "clusterrolebindings" in API group "rbac.authorization.k8s.io/v1" [dryrun] Attached object: apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: creationTimestamp: null name: system:coredns roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:coredns subjects: - kind: ServiceAccount name: coredns namespace: kube-system [dryrun] Would perform action CREATE on resource "serviceaccounts" in API group "core/v1" [dryrun] Attached object: apiVersion: v1 kind: ServiceAccount metadata: creationTimestamp: null name: coredns namespace: kube-system [dryrun] Would perform action CREATE on resource "deployments" in API group "apps/v1" [dryrun] Attached object: apiVersion: apps/v1 kind: Deployment metadata: creationTimestamp: null labels: k8s-app: kube-dns name: coredns namespace: kube-system spec: replicas: 2 selector: matchLabels: k8s-app: kube-dns strategy: rollingUpdate: maxUnavailable: 1 type: RollingUpdate template: metadata: creationTimestamp: null labels: k8s-app: kube-dns spec: containers: - args: - -conf - /etc/coredns/Corefile image: k8s.gcr.io/coredns:1.6.5 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 5 httpGet: path: /health port: 8080 scheme: HTTP initialDelaySeconds: 60 successThreshold: 1 timeoutSeconds: 5 name: coredns ports: - containerPort: 53 name: dns protocol: UDP - containerPort: 53 name: dns-tcp protocol: TCP - containerPort: 9153 name: metrics protocol: TCP readinessProbe: httpGet: path: /ready port: 8181 scheme: HTTP resources: limits: memory: 170Mi requests: cpu: 100m memory: 70Mi securityContext: allowPrivilegeEscalation: false capabilities: add: - NET_BIND_SERVICE drop: - all readOnlyRootFilesystem: true volumeMounts: - mountPath: /etc/coredns name: config-volume readOnly: true dnsPolicy: Default nodeSelector: beta.kubernetes.io/os: linux priorityClassName: system-cluster-critical serviceAccountName: coredns tolerations: - key: CriticalAddonsOnly operator: Exists - effect: NoSchedule key: node-role.kubernetes.io/master volumes: - configMap: items: - key: Corefile path: Corefile name: coredns name: config-volume status: {} [dryrun] Would perform action CREATE on resource "services" in API group "core/v1" [dryrun] Attached object: apiVersion: v1 kind: Service metadata: annotations: prometheus.io/port: "9153" prometheus.io/scrape: "true" creationTimestamp: null labels: k8s-app: kube-dns kubernetes.io/cluster-service: "true" kubernetes.io/name: KubeDNS name: kube-dns namespace: kube-system resourceVersion: "0" spec: clusterIP: 10.96.0.10 ports: - name: dns port: 53 protocol: UDP targetPort: 53 - name: dns-tcp port: 53 protocol: TCP targetPort: 53 - name: metrics port: 9153 protocol: TCP targetPort: 9153 selector: k8s-app: kube-dns status: loadBalancer: {} [addons] Applied essential addon: CoreDNS [dryrun] Would perform action CREATE on resource "serviceaccounts" in API group "core/v1" [dryrun] Attached object: apiVersion: v1 kind: ServiceAccount metadata: creationTimestamp: null name: kube-proxy namespace: kube-system [dryrun] Would perform action CREATE on resource "configmaps" in API group "core/v1" [dryrun] Attached object: apiVersion: v1 data: config.conf: |- apiVersion: kubeproxy.config.k8s.io/v1alpha1 bindAddress: 0.0.0.0 clientConnection: acceptContentTypes: "" burst: 0 contentType: "" kubeconfig: /var/lib/kube-proxy/kubeconfig.conf qps: 0 clusterCIDR: 10.244.0.0/16 configSyncPeriod: 0s conntrack: maxPerCore: null min: null tcpCloseWaitTimeout: null tcpEstablishedTimeout: null enableProfiling: false healthzBindAddress: "" hostnameOverride: "" iptables: masqueradeAll: false masqueradeBit: null minSyncPeriod: 0s syncPeriod: 0s ipvs: excludeCIDRs: null minSyncPeriod: 0s scheduler: "" strictARP: false syncPeriod: 0s kind: KubeProxyConfiguration metricsBindAddress: "" mode: "" nodePortAddresses: null oomScoreAdj: null portRange: "" udpIdleTimeout: 0s winkernel: enableDSR: false networkName: "" sourceVip: "" kubeconfig.conf: |- apiVersion: v1 kind: Config clusters: - cluster: certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt server: https://172.200.1.200:6443 name: default contexts: - context: cluster: default namespace: default user: default name: default current-context: default users: - name: default user: tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token kind: ConfigMap metadata: creationTimestamp: null labels: app: kube-proxy name: kube-proxy namespace: kube-system [dryrun] Would perform action CREATE on resource "daemonsets" in API group "apps/v1" [dryrun] Attached object: apiVersion: apps/v1 kind: DaemonSet metadata: creationTimestamp: null labels: k8s-app: kube-proxy name: kube-proxy namespace: kube-system spec: selector: matchLabels: k8s-app: kube-proxy template: metadata: creationTimestamp: null labels: k8s-app: kube-proxy spec: containers: - command: - /usr/local/bin/kube-proxy - --config=/var/lib/kube-proxy/config.conf - --hostname-override=$(NODE_NAME) env: - name: NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName image: k8s.gcr.io/kube-proxy:v1.17.2 imagePullPolicy: IfNotPresent name: kube-proxy resources: {} securityContext: privileged: true volumeMounts: - mountPath: /var/lib/kube-proxy name: kube-proxy - mountPath: /run/xtables.lock name: xtables-lock - mountPath: /lib/modules name: lib-modules readOnly: true hostNetwork: true nodeSelector: beta.kubernetes.io/os: linux priorityClassName: system-node-critical serviceAccountName: kube-proxy tolerations: - key: CriticalAddonsOnly operator: Exists - operator: Exists volumes: - configMap: name: kube-proxy name: kube-proxy - hostPath: path: /run/xtables.lock type: FileOrCreate name: xtables-lock - hostPath: path: /lib/modules name: lib-modules updateStrategy: type: RollingUpdate status: currentNumberScheduled: 0 desiredNumberScheduled: 0 numberMisscheduled: 0 numberReady: 0 [dryrun] Would perform action CREATE on resource "clusterrolebindings" in API group "rbac.authorization.k8s.io/v1" [dryrun] Attached object: apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: creationTimestamp: null name: kubeadm:node-proxier roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:node-proxier subjects: - kind: ServiceAccount name: kube-proxy namespace: kube-system [dryrun] Would perform action CREATE on resource "roles" in API group "rbac.authorization.k8s.io/v1" [dryrun] Attached object: apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: creationTimestamp: null name: kube-proxy namespace: kube-system rules: - apiGroups: - "" resourceNames: - kube-proxy resources: - configmaps verbs: - get [dryrun] Would perform action CREATE on resource "rolebindings" in API group "rbac.authorization.k8s.io/v1" [dryrun] Attached object: apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: creationTimestamp: null name: kube-proxy namespace: kube-system roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: kube-proxy subjects: - kind: Group name: system:bootstrappers:kubeadm:default-node-token [addons] Applied essential addon: kube-proxy Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/tmp/kubeadm-init-dryrun988930826/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 172.200.1.200:6443 --token dp9kc3.s7ceexx1xblkqklh --discovery-token-ca-cert-hash sha256:c87a7b40890a61c0b9c23d71418f1b63987a74d943839b018eb836d1cdded4eb [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# kubeadm init --kubernetes-version="v1.17.2" --pod-network-cidr="10.244.0.0/16" W0204 19:39:12.076223 32177 validation.go:28] Cannot validate kube-proxy config - no validator is available W0204 19:39:12.076260 32177 validation.go:28] Cannot validate kubelet config - no validator is available [init] Using Kubernetes version: v1.17.2 [preflight] Running pre-flight checks [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/ [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service' [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Starting the kubelet [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [master200.yinzhengjie.org.cn kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.200.1.200] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [master200.yinzhengjie.org.cn localhost] and IPs [172.200.1.200 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [master200.yinzhengjie.org.cn localhost] and IPs [172.200.1.200 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "kubelet.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" W0204 19:39:14.609238 32177 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC" [control-plane] Creating static Pod manifest for "kube-scheduler" W0204 19:39:14.609827 32177 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC" [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [apiclient] All control plane components are healthy after 18.505316 seconds [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config-1.17" in namespace kube-system with the configuration for the kubelets in the cluster [upload-certs] Skipping phase. Please see --upload-certs [mark-control-plane] Marking the node master200.yinzhengjie.org.cn as control-plane by adding the label "node-role.kubernetes.io/master=''" [mark-control-plane] Marking the node master200.yinzhengjie.org.cn as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] [bootstrap-token] Using token: gu02ed.9k1rpyl4mtkhvdpk [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 172.200.1.200:6443 --token gu02ed.9k1rpyl4mtkhvdpk --discovery-token-ca-cert-hash sha256:755ff7d07588b75aec769e566e4257d832cf6d1ec79bcdbd655dad80f30a6794 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# echo $? 0 [root@master200.yinzhengjie.org.cn ~]#
6>.部署网络插件
[root@master200.yinzhengjie.org.cn ~]# mkdir .kube [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# cp /etc/kubernetes/admin.conf .kube/config [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# kubectl get nodes #kubectl是k8s的apiserver的客户端工具，我们可以通过该命令查看节点的状态，很明显是"NotReady"状态，我们需要执行吓一跳命令下载网络插件 NAME STATUS ROLES AGE VERSION master200.yinzhengjie.org.cn NotReady master 6m21s v1.17.2 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml #下载flannel网络插件 podsecuritypolicy.policy/psp.flannel.unprivileged created clusterrole.rbac.authorization.k8s.io/flannel created clusterrolebinding.rbac.authorization.k8s.io/flannel created serviceaccount/flannel created configmap/kube-flannel-cfg created daemonset.apps/kube-flannel-ds-amd64 created daemonset.apps/kube-flannel-ds-arm64 created daemonset.apps/kube-flannel-ds-arm created daemonset.apps/kube-flannel-ds-ppc64le created daemonset.apps/kube-flannel-ds-s390x created [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# kubectl get pods -n kube-system #使用"-n"查看名称"kube-system"的名称空间,如下图所示，所有组件都是"Running"状态，因此我们的master节点应该也是"Ready"状态啦~ NAME READY STATUS RESTARTS AGE coredns-6955765f44-455fh 0/1 Running 0 8m17s coredns-6955765f44-q6zqj 0/1 Running 0 8m17s etcd-master200.yinzhengjie.org.cn 1/1 Running 0 8m31s kube-apiserver-master200.yinzhengjie.org.cn 1/1 Running 0 8m31s kube-controller-manager-master200.yinzhengjie.org.cn 1/1 Running 0 8m31s kube-flannel-ds-amd64-hnnhb 1/1 Running 0 33s kube-proxy-6r9dx 1/1 Running 0 8m17s kube-scheduler-master200.yinzhengjie.org.cn 1/1 Running 0 8m31s [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# kubectl get nodes #安装flannel(由CoreOS公司研发)网络插件后,它会自动运行,之后咱们的k8s master节点的状态成功变为"Ready"。 NAME STATUS ROLES AGE VERSION master200.yinzhengjie.org.cn Ready master 9m40s v1.17.2 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
7>.将k8s master节点的镜像打包并发送到k8s node节点
[root@master200.yinzhengjie.org.cn ~]# docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE k8s.gcr.io/kube-proxy v1.17.2 cba2a99699bd 2 weeks ago 116MB k8s.gcr.io/kube-apiserver v1.17.2 41ef50a5f06a 2 weeks ago 171MB k8s.gcr.io/kube-controller-manager v1.17.2 da5fd66c4068 2 weeks ago 161MB k8s.gcr.io/kube-scheduler v1.17.2 f52d4c527ef2 2 weeks ago 94.4MB k8s.gcr.io/coredns 1.6.5 70f311871ae1 3 months ago 41.6MB k8s.gcr.io/etcd 3.4.3-0 303ce5db0e90 3 months ago 288MB quay.io/coreos/flannel v0.11.0-amd64 ff281650a721 12 months ago 52.6MB k8s.gcr.io/pause 3.1 da86e6ba6ca1 2 years ago 742kB [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# ll total 0 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# docker image save k8s.gcr.io/kube-proxy:v1.17.2 k8s.gcr.io/pause:3.1 quay.io/coreos/flannel:v0.11.0-amd64 -o k8s-node-v1.17.2.tar [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# ll total 170028 -rw------- 1 root root 174108672 Feb 4 20:18 k8s-node-v1.17.2.tar [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# ll -h total 167M -rw------- 1 root root 167M Feb 4 20:18 k8s-node-v1.17.2.tar [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# scp k8s-node-v1.17.2.tar node201.yinzhengjie.org.cn:~ root@node201.yinzhengjie.org.cn's password: k8s-node-v1.17.2.tar 100% 166MB 119.7MB/s 00:01 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# scp k8s-node-v1.17.2.tar node202.yinzhengjie.org.cn:~ root@node202.yinzhengjie.org.cn's password: k8s-node-v1.17.2.tar 100% 166MB 118.3MB/s 00:01 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# scp k8s-node-v1.17.2.tar node203.yinzhengjie.org.cn:~ root@node203.yinzhengjie.org.cn's password: k8s-node-v1.17.2.tar 100% 166MB 122.9MB/s 00:01 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
8>.各个k8s node节点导入镜像
[root@node201.yinzhengjie.org.cn ~]# ll total 170028 -rw------- 1 root root 174108672 Feb 4 20:19 k8s-node-v1.17.2.tar [root@node201.yinzhengjie.org.cn ~]# [root@node201.yinzhengjie.org.cn ~]# docker image load -i k8s-node-v1.17.2.tar 7bff100f35cb: Loading layer [==================================================>] 4.672MB/4.672MB 5d3f68f6da8f: Loading layer [==================================================>] 9.526MB/9.526MB 9b48060f404d: Loading layer [==================================================>] 5.912MB/5.912MB 3f3a4ce2b719: Loading layer [==================================================>] 35.25MB/35.25MB 9ce0bb155166: Loading layer [==================================================>] 5.12kB/5.12kB Loaded image: quay.io/coreos/flannel:v0.11.0-amd64 fc4976bd934b: Loading layer [==================================================>] 53.88MB/53.88MB 682fbb19de80: Loading layer [==================================================>] 21.06MB/21.06MB 2dc2f2423ad1: Loading layer [==================================================>] 5.168MB/5.168MB ad9fb2411669: Loading layer [==================================================>] 4.608kB/4.608kB 597151d24476: Loading layer [==================================================>] 8.192kB/8.192kB 0d8d54147a3a: Loading layer [==================================================>] 8.704kB/8.704kB ca7fe3329548: Loading layer [==================================================>] 37.81MB/37.81MB Loaded image: k8s.gcr.io/kube-proxy:v1.17.2 e17133b79956: Loading layer [==================================================>] 744.4kB/744.4kB Loaded image: k8s.gcr.io/pause:3.1 [root@node201.yinzhengjie.org.cn ~]# [root@node201.yinzhengjie.org.cn ~]# docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE k8s.gcr.io/kube-proxy v1.17.2 cba2a99699bd 2 weeks ago 116MB quay.io/coreos/flannel v0.11.0-amd64 ff281650a721 12 months ago 52.6MB k8s.gcr.io/pause 3.1 da86e6ba6ca1 2 years ago 742kB [root@node201.yinzhengjie.org.cn ~]# [root@node201.yinzhengjie.org.cn ~]#
[root@node202.yinzhengjie.org.cn ~]# ll total 170028 -rw------- 1 root root 174108672 Feb 4 20:20 k8s-node-v1.17.2.tar [root@node202.yinzhengjie.org.cn ~]# [root@node202.yinzhengjie.org.cn ~]# docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE [root@node202.yinzhengjie.org.cn ~]# [root@node202.yinzhengjie.org.cn ~]# docker image load -i k8s-node-v1.17.2.tar 7bff100f35cb: Loading layer [==================================================>] 4.672MB/4.672MB 5d3f68f6da8f: Loading layer [==================================================>] 9.526MB/9.526MB 9b48060f404d: Loading layer [==================================================>] 5.912MB/5.912MB 3f3a4ce2b719: Loading layer [==================================================>] 35.25MB/35.25MB 9ce0bb155166: Loading layer [==================================================>] 5.12kB/5.12kB Loaded image: quay.io/coreos/flannel:v0.11.0-amd64 fc4976bd934b: Loading layer [==================================================>] 53.88MB/53.88MB 682fbb19de80: Loading layer [==================================================>] 21.06MB/21.06MB 2dc2f2423ad1: Loading layer [==================================================>] 5.168MB/5.168MB ad9fb2411669: Loading layer [==================================================>] 4.608kB/4.608kB 597151d24476: Loading layer [==================================================>] 8.192kB/8.192kB 0d8d54147a3a: Loading layer [==================================================>] 8.704kB/8.704kB ca7fe3329548: Loading layer [==================================================>] 37.81MB/37.81MB Loaded image: k8s.gcr.io/kube-proxy:v1.17.2 e17133b79956: Loading layer [==================================================>] 744.4kB/744.4kB Loaded image: k8s.gcr.io/pause:3.1 [root@node202.yinzhengjie.org.cn ~]# [root@node202.yinzhengjie.org.cn ~]# [root@node202.yinzhengjie.org.cn ~]# ll total 170028 -rw------- 1 root root 174108672 Feb 4 20:20 k8s-node-v1.17.2.tar [root@node202.yinzhengjie.org.cn ~]# [root@node202.yinzhengjie.org.cn ~]# docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE k8s.gcr.io/kube-proxy v1.17.2 cba2a99699bd 2 weeks ago 116MB quay.io/coreos/flannel v0.11.0-amd64 ff281650a721 12 months ago 52.6MB k8s.gcr.io/pause 3.1 da86e6ba6ca1 2 years ago 742kB [root@node202.yinzhengjie.org.cn ~]# [root@node202.yinzhengjie.org.cn ~]#
[root@node203.yinzhengjie.org.cn ~]# docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE [root@node203.yinzhengjie.org.cn ~]# [root@node203.yinzhengjie.org.cn ~]# ll total 170028 -rw------- 1 root root 174108672 Feb 4 20:20 k8s-node-v1.17.2.tar [root@node203.yinzhengjie.org.cn ~]# [root@node203.yinzhengjie.org.cn ~]# docker image load -i k8s-node-v1.17.2.tar 7bff100f35cb: Loading layer [==================================================>] 4.672MB/4.672MB 5d3f68f6da8f: Loading layer [==================================================>] 9.526MB/9.526MB 9b48060f404d: Loading layer [==================================================>] 5.912MB/5.912MB 3f3a4ce2b719: Loading layer [==================================================>] 35.25MB/35.25MB 9ce0bb155166: Loading layer [==================================================>] 5.12kB/5.12kB Loaded image: quay.io/coreos/flannel:v0.11.0-amd64 fc4976bd934b: Loading layer [==================================================>] 53.88MB/53.88MB 682fbb19de80: Loading layer [==================================================>] 21.06MB/21.06MB 2dc2f2423ad1: Loading layer [==================================================>] 5.168MB/5.168MB ad9fb2411669: Loading layer [==================================================>] 4.608kB/4.608kB 597151d24476: Loading layer [==================================================>] 8.192kB/8.192kB 0d8d54147a3a: Loading layer [==================================================>] 8.704kB/8.704kB ca7fe3329548: Loading layer [==================================================>] 37.81MB/37.81MB Loaded image: k8s.gcr.io/kube-proxy:v1.17.2 e17133b79956: Loading layer [==================================================>] 744.4kB/744.4kB Loaded image: k8s.gcr.io/pause:3.1 [root@node203.yinzhengjie.org.cn ~]# [root@node203.yinzhengjie.org.cn ~]# docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE k8s.gcr.io/kube-proxy v1.17.2 cba2a99699bd 2 weeks ago 116MB quay.io/coreos/flannel v0.11.0-amd64 ff281650a721 12 months ago 52.6MB k8s.gcr.io/pause 3.1 da86e6ba6ca1 2 years ago 742kB [root@node203.yinzhengjie.org.cn ~]# [root@node203.yinzhengjie.org.cn ~]#
四.将k8s node节点加入到上一步初始化的k8s集群
1>.将k8s master节点的k8s软件源拷贝到其它3个k8s node节点
[root@master200.yinzhengjie.org.cn ~]# scp /etc/yum.repos.d/kubernetes.repo node201.yinzhengjie.org.cn:/etc/yum.repos.d/kubernetes.repo root@node201.yinzhengjie.org.cn's password: kubernetes.repo 100% 275 125.6KB/s 00:00 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# scp /etc/yum.repos.d/kubernetes.repo node202.yinzhengjie.org.cn:/etc/yum.repos.d/kubernetes.repo root@node202.yinzhengjie.org.cn's password: kubernetes.repo 100% 275 199.6KB/s 00:00 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# scp /etc/yum.repos.d/kubernetes.repo node203.yinzhengjie.org.cn:/etc/yum.repos.d/kubernetes.repo root@node203.yinzhengjie.org.cn's password: kubernetes.repo 100% 275 63.0KB/s 00:00 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
2>.三个节点均安装k8s相关软件包
[root@node201.yinzhengjie.org.cn ~]# yum -y install kubelet kubeadm Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.aliyun.com * extras: mirrors.aliyun.com * updates: mirrors.aliyun.com docker-ce-stable | 3.5 kB 00:00:00 kubernetes | 1.4 kB 00:00:00 kubernetes/primary | 63 kB 00:00:00 kubernetes 460/460 Resolving Dependencies --> Running transaction check ---> Package kubeadm.x86_64 0:1.17.2-0 will be installed --> Processing Dependency: kubernetes-cni >= 0.7.5 for package: kubeadm-1.17.2-0.x86_64 --> Processing Dependency: kubectl >= 1.13.0 for package: kubeadm-1.17.2-0.x86_64 --> Processing Dependency: cri-tools >= 1.13.0 for package: kubeadm-1.17.2-0.x86_64 ---> Package kubelet.x86_64 0:1.17.2-0 will be installed --> Processing Dependency: socat for package: kubelet-1.17.2-0.x86_64 --> Processing Dependency: conntrack for package: kubelet-1.17.2-0.x86_64 --> Running transaction check ---> Package conntrack-tools.x86_64 0:1.4.4-5.el7_7.2 will be installed --> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.1)(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.0)(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_cthelper.so.0(LIBNETFILTER_CTHELPER_1.0)(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_queue.so.1()(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_cttimeout.so.1()(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_cthelper.so.0()(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 ---> Package cri-tools.x86_64 0:1.13.0-0 will be installed ---> Package kubectl.x86_64 0:1.17.2-0 will be installed ---> Package kubernetes-cni.x86_64 0:0.7.5-0 will be installed ---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed --> Running transaction check ---> Package libnetfilter_cthelper.x86_64 0:1.0.0-10.el7_7.1 will be installed ---> Package libnetfilter_cttimeout.x86_64 0:1.0.0-6.el7_7.1 will be installed ---> Package libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 will be installed --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================================================================================================================================================================== Package Arch Version Repository Size ============================================================================================================================================================================================================================================================================== Installing: kubeadm x86_64 1.17.2-0 kubernetes 8.7 M kubelet x86_64 1.17.2-0 kubernetes 20 M Installing for dependencies: conntrack-tools x86_64 1.4.4-5.el7_7.2 updates 187 k cri-tools x86_64 1.13.0-0 kubernetes 5.1 M kubectl x86_64 1.17.2-0 kubernetes 9.4 M kubernetes-cni x86_64 0.7.5-0 kubernetes 10 M libnetfilter_cthelper x86_64 1.0.0-10.el7_7.1 updates 18 k libnetfilter_cttimeout x86_64 1.0.0-6.el7_7.1 updates 18 k libnetfilter_queue x86_64 1.0.2-2.el7_2 base 23 k socat x86_64 1.7.3.2-2.el7 base 290 k Transaction Summary ============================================================================================================================================================================================================================================================================== Install 2 Packages (+8 Dependent packages) Total download size: 54 M Installed size: 243 M Downloading packages: (1/10): conntrack-tools-1.4.4-5.el7_7.2.x86_64.rpm | 187 kB 00:00:00 warning: /var/cache/yum/x86_64/7/kubernetes/packages/14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 3e1ba8d5: NOKEY ] 3.5 MB/s | 6.9 MB 00:00:13 ETA Public key for 14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm is not installed (2/10): 14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm | 5.1 MB 00:00:01 (3/10): 105d89f0607c7baf91305ba352e78000bd20aad5cdf706bffff3b31cd546dbf3-kubeadm-1.17.2-0.x86_64.rpm | 8.7 MB 00:00:02 (4/10): b44630896c69cd411db53be1d5cb5ae899a40aba7c0766317ea904390fcfc45b-kubectl-1.17.2-0.x86_64.rpm | 9.4 MB 00:00:01 (5/10): libnetfilter_cttimeout-1.0.0-6.el7_7.1.x86_64.rpm | 18 kB 00:00:00 (6/10): libnetfilter_queue-1.0.2-2.el7_2.x86_64.rpm | 23 kB 00:00:00 (7/10): 548a0dcd865c16a50980420ddfa5fbccb8b59621179798e6dc905c9bf8af3b34-kubernetes-cni-0.7.5-0.x86_64.rpm | 10 MB 00:00:01 (8/10): socat-1.7.3.2-2.el7.x86_64.rpm | 290 kB 00:00:01 (9/10): libnetfilter_cthelper-1.0.0-10.el7_7.1.x86_64.rpm | 18 kB 00:00:02 (10/10): 3ee7f2dff78e6fbb3ac3af8acb1a907f4bec1b1ef4cf627cbe02fa553707f2e9-kubelet-1.17.2-0.x86_64.rpm | 20 MB 00:00:04 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Total 8.6 MB/s | 54 MB 00:00:06 Retrieving key from https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg Importing GPG key 0x3E1BA8D5: Userid : "Google Cloud Packages RPM Signing Key <gc-team@google.com>" Fingerprint: 3749 e1ba 95a8 6ce0 5454 6ed2 f09c 394c 3e1b a8d5 From : https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg Retrieving key from https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg Importing GPG key 0xA7317B0F: Userid : "Google Cloud Packages Automatic Signing Key <gc-team@google.com>" Fingerprint: d0bc 747f d8ca f711 7500 d6fa 3746 c208 a731 7b0f From : https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : libnetfilter_cttimeout-1.0.0-6.el7_7.1.x86_64 1/10 Installing : socat-1.7.3.2-2.el7.x86_64 2/10 Installing : cri-tools-1.13.0-0.x86_64 3/10 Installing : kubectl-1.17.2-0.x86_64 4/10 Installing : libnetfilter_queue-1.0.2-2.el7_2.x86_64 5/10 Installing : libnetfilter_cthelper-1.0.0-10.el7_7.1.x86_64 6/10 Installing : conntrack-tools-1.4.4-5.el7_7.2.x86_64 7/10 Installing : kubernetes-cni-0.7.5-0.x86_64 8/10 Installing : kubelet-1.17.2-0.x86_64 9/10 Installing : kubeadm-1.17.2-0.x86_64 10/10 Verifying : libnetfilter_cthelper-1.0.0-10.el7_7.1.x86_64 1/10 Verifying : conntrack-tools-1.4.4-5.el7_7.2.x86_64 2/10 Verifying : libnetfilter_queue-1.0.2-2.el7_2.x86_64 3/10 Verifying : kubectl-1.17.2-0.x86_64 4/10 Verifying : cri-tools-1.13.0-0.x86_64 5/10 Verifying : kubeadm-1.17.2-0.x86_64 6/10 Verifying : kubernetes-cni-0.7.5-0.x86_64 7/10 Verifying : socat-1.7.3.2-2.el7.x86_64 8/10 Verifying : kubelet-1.17.2-0.x86_64 9/10 Verifying : libnetfilter_cttimeout-1.0.0-6.el7_7.1.x86_64 10/10 Installed: kubeadm.x86_64 0:1.17.2-0 kubelet.x86_64 0:1.17.2-0 Dependency Installed: conntrack-tools.x86_64 0:1.4.4-5.el7_7.2 cri-tools.x86_64 0:1.13.0-0 kubectl.x86_64 0:1.17.2-0 kubernetes-cni.x86_64 0:0.7.5-0 libnetfilter_cthelper.x86_64 0:1.0.0-10.el7_7.1 libnetfilter_cttimeout.x86_64 0:1.0.0-6.el7_7.1 libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 socat.x86_64 0:1.7.3.2-2.el7 Complete! [root@node201.yinzhengjie.org.cn ~]#
[root@node202.yinzhengjie.org.cn ~]# yum -y install kubelet kubeadm Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirror.bit.edu.cn * extras: mirrors.aliyun.com * updates: mirrors.aliyun.com docker-ce-stable | 3.5 kB 00:00:00 kubernetes | 1.4 kB 00:00:00 kubernetes/primary | 63 kB 00:00:00 kubernetes 460/460 Resolving Dependencies --> Running transaction check ---> Package kubeadm.x86_64 0:1.17.2-0 will be installed --> Processing Dependency: kubernetes-cni >= 0.7.5 for package: kubeadm-1.17.2-0.x86_64 --> Processing Dependency: kubectl >= 1.13.0 for package: kubeadm-1.17.2-0.x86_64 --> Processing Dependency: cri-tools >= 1.13.0 for package: kubeadm-1.17.2-0.x86_64 ---> Package kubelet.x86_64 0:1.17.2-0 will be installed --> Processing Dependency: socat for package: kubelet-1.17.2-0.x86_64 --> Processing Dependency: conntrack for package: kubelet-1.17.2-0.x86_64 --> Running transaction check ---> Package conntrack-tools.x86_64 0:1.4.4-5.el7_7.2 will be installed --> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.1)(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.0)(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_cthelper.so.0(LIBNETFILTER_CTHELPER_1.0)(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_queue.so.1()(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_cttimeout.so.1()(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_cthelper.so.0()(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 ---> Package cri-tools.x86_64 0:1.13.0-0 will be installed ---> Package kubectl.x86_64 0:1.17.2-0 will be installed ---> Package kubernetes-cni.x86_64 0:0.7.5-0 will be installed ---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed --> Running transaction check ---> Package libnetfilter_cthelper.x86_64 0:1.0.0-10.el7_7.1 will be installed ---> Package libnetfilter_cttimeout.x86_64 0:1.0.0-6.el7_7.1 will be installed ---> Package libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 will be installed --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================================================================================================================================================================== Package Arch Version Repository Size ============================================================================================================================================================================================================================================================================== Installing: kubeadm x86_64 1.17.2-0 kubernetes 8.7 M kubelet x86_64 1.17.2-0 kubernetes 20 M Installing for dependencies: conntrack-tools x86_64 1.4.4-5.el7_7.2 updates 187 k cri-tools x86_64 1.13.0-0 kubernetes 5.1 M kubectl x86_64 1.17.2-0 kubernetes 9.4 M kubernetes-cni x86_64 0.7.5-0 kubernetes 10 M libnetfilter_cthelper x86_64 1.0.0-10.el7_7.1 updates 18 k libnetfilter_cttimeout x86_64 1.0.0-6.el7_7.1 updates 18 k libnetfilter_queue x86_64 1.0.2-2.el7_2 base 23 k socat x86_64 1.7.3.2-2.el7 base 290 k Transaction Summary ============================================================================================================================================================================================================================================================================== Install 2 Packages (+8 Dependent packages) Total download size: 54 M Installed size: 243 M Downloading packages: (1/10): conntrack-tools-1.4.4-5.el7_7.2.x86_64.rpm | 187 kB 00:00:00 warning: /var/cache/yum/x86_64/7/kubernetes/packages/14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 3e1ba8d5: NOKEY ] 1.8 MB/s | 3.7 MB 00:00:28 ETA Public key for 14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm is not installed (2/10): 14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm | 5.1 MB 00:00:01 (3/10): b44630896c69cd411db53be1d5cb5ae899a40aba7c0766317ea904390fcfc45b-kubectl-1.17.2-0.x86_64.rpm | 9.4 MB 00:00:01 (4/10): 3ee7f2dff78e6fbb3ac3af8acb1a907f4bec1b1ef4cf627cbe02fa553707f2e9-kubelet-1.17.2-0.x86_64.rpm | 20 MB 00:00:02 (5/10): libnetfilter_cthelper-1.0.0-10.el7_7.1.x86_64.rpm | 18 kB 00:00:00 (6/10): libnetfilter_cttimeout-1.0.0-6.el7_7.1.x86_64.rpm | 18 kB 00:00:00 (7/10): libnetfilter_queue-1.0.2-2.el7_2.x86_64.rpm | 23 kB 00:00:00 (8/10): socat-1.7.3.2-2.el7.x86_64.rpm | 290 kB 00:00:00 (9/10): 548a0dcd865c16a50980420ddfa5fbccb8b59621179798e6dc905c9bf8af3b34-kubernetes-cni-0.7.5-0.x86_64.rpm | 10 MB 00:00:01 (10/10): 105d89f0607c7baf91305ba352e78000bd20aad5cdf706bffff3b31cd546dbf3-kubeadm-1.17.2-0.x86_64.rpm | 8.7 MB 00:00:18 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Total 2.9 MB/s | 54 MB 00:00:18 Retrieving key from https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg Importing GPG key 0x3E1BA8D5: Userid : "Google Cloud Packages RPM Signing Key <gc-team@google.com>" Fingerprint: 3749 e1ba 95a8 6ce0 5454 6ed2 f09c 394c 3e1b a8d5 From : https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg Retrieving key from https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg Importing GPG key 0xA7317B0F: Userid : "Google Cloud Packages Automatic Signing Key <gc-team@google.com>" Fingerprint: d0bc 747f d8ca f711 7500 d6fa 3746 c208 a731 7b0f From : https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : libnetfilter_cttimeout-1.0.0-6.el7_7.1.x86_64 1/10 Installing : socat-1.7.3.2-2.el7.x86_64 2/10 Installing : cri-tools-1.13.0-0.x86_64 3/10 Installing : kubectl-1.17.2-0.x86_64 4/10 Installing : libnetfilter_queue-1.0.2-2.el7_2.x86_64 5/10 Installing : libnetfilter_cthelper-1.0.0-10.el7_7.1.x86_64 6/10 Installing : conntrack-tools-1.4.4-5.el7_7.2.x86_64 7/10 Installing : kubernetes-cni-0.7.5-0.x86_64 8/10 Installing : kubelet-1.17.2-0.x86_64 9/10 Installing : kubeadm-1.17.2-0.x86_64 10/10 Verifying : libnetfilter_cthelper-1.0.0-10.el7_7.1.x86_64 1/10 Verifying : conntrack-tools-1.4.4-5.el7_7.2.x86_64 2/10 Verifying : libnetfilter_queue-1.0.2-2.el7_2.x86_64 3/10 Verifying : kubectl-1.17.2-0.x86_64 4/10 Verifying : cri-tools-1.13.0-0.x86_64 5/10 Verifying : kubeadm-1.17.2-0.x86_64 6/10 Verifying : kubernetes-cni-0.7.5-0.x86_64 7/10 Verifying : socat-1.7.3.2-2.el7.x86_64 8/10 Verifying : kubelet-1.17.2-0.x86_64 9/10 Verifying : libnetfilter_cttimeout-1.0.0-6.el7_7.1.x86_64 10/10 Installed: kubeadm.x86_64 0:1.17.2-0 kubelet.x86_64 0:1.17.2-0 Dependency Installed: conntrack-tools.x86_64 0:1.4.4-5.el7_7.2 cri-tools.x86_64 0:1.13.0-0 kubectl.x86_64 0:1.17.2-0 kubernetes-cni.x86_64 0:0.7.5-0 libnetfilter_cthelper.x86_64 0:1.0.0-10.el7_7.1 libnetfilter_cttimeout.x86_64 0:1.0.0-6.el7_7.1 libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 socat.x86_64 0:1.7.3.2-2.el7 Complete! [root@node202.yinzhengjie.org.cn ~]#
[root@node203.yinzhengjie.org.cn ~]# yum -y install kubelet kubeadm Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirrors.aliyun.com * extras: mirrors.aliyun.com * updates: mirrors.aliyun.com docker-ce-stable | 3.5 kB 00:00:00 kubernetes | 1.4 kB 00:00:00 kubernetes/primary | 63 kB 00:00:01 kubernetes 460/460 Resolving Dependencies --> Running transaction check ---> Package kubeadm.x86_64 0:1.17.2-0 will be installed --> Processing Dependency: kubernetes-cni >= 0.7.5 for package: kubeadm-1.17.2-0.x86_64 --> Processing Dependency: kubectl >= 1.13.0 for package: kubeadm-1.17.2-0.x86_64 --> Processing Dependency: cri-tools >= 1.13.0 for package: kubeadm-1.17.2-0.x86_64 ---> Package kubelet.x86_64 0:1.17.2-0 will be installed --> Processing Dependency: socat for package: kubelet-1.17.2-0.x86_64 --> Processing Dependency: conntrack for package: kubelet-1.17.2-0.x86_64 --> Running transaction check ---> Package conntrack-tools.x86_64 0:1.4.4-5.el7_7.2 will be installed --> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.1)(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.0)(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_cthelper.so.0(LIBNETFILTER_CTHELPER_1.0)(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_queue.so.1()(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_cttimeout.so.1()(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 --> Processing Dependency: libnetfilter_cthelper.so.0()(64bit) for package: conntrack-tools-1.4.4-5.el7_7.2.x86_64 ---> Package cri-tools.x86_64 0:1.13.0-0 will be installed ---> Package kubectl.x86_64 0:1.17.2-0 will be installed ---> Package kubernetes-cni.x86_64 0:0.7.5-0 will be installed ---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed --> Running transaction check ---> Package libnetfilter_cthelper.x86_64 0:1.0.0-10.el7_7.1 will be installed ---> Package libnetfilter_cttimeout.x86_64 0:1.0.0-6.el7_7.1 will be installed ---> Package libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 will be installed --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================================================================================================================================================================== Package Arch Version Repository Size ============================================================================================================================================================================================================================================================================== Installing: kubeadm x86_64 1.17.2-0 kubernetes 8.7 M kubelet x86_64 1.17.2-0 kubernetes 20 M Installing for dependencies: conntrack-tools x86_64 1.4.4-5.el7_7.2 updates 187 k cri-tools x86_64 1.13.0-0 kubernetes 5.1 M kubectl x86_64 1.17.2-0 kubernetes 9.4 M kubernetes-cni x86_64 0.7.5-0 kubernetes 10 M libnetfilter_cthelper x86_64 1.0.0-10.el7_7.1 updates 18 k libnetfilter_cttimeout x86_64 1.0.0-6.el7_7.1 updates 18 k libnetfilter_queue x86_64 1.0.2-2.el7_2 base 23 k socat x86_64 1.7.3.2-2.el7 base 290 k Transaction Summary ============================================================================================================================================================================================================================================================================== Install 2 Packages (+8 Dependent packages) Total download size: 54 M Installed size: 243 M Downloading packages: (1/10): conntrack-tools-1.4.4-5.el7_7.2.x86_64.rpm | 187 kB 00:00:00 warning: /var/cache/yum/x86_64/7/kubernetes/packages/14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 3e1ba8d5: NOKEY ] 1.4 MB/s | 7.3 MB 00:00:34 ETA Public key for 14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm is not installed (2/10): 14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm | 5.1 MB 00:00:03 (3/10): 105d89f0607c7baf91305ba352e78000bd20aad5cdf706bffff3b31cd546dbf3-kubeadm-1.17.2-0.x86_64.rpm | 8.7 MB 00:00:03 (4/10): b44630896c69cd411db53be1d5cb5ae899a40aba7c0766317ea904390fcfc45b-kubectl-1.17.2-0.x86_64.rpm | 9.4 MB 00:00:02 (5/10): libnetfilter_cthelper-1.0.0-10.el7_7.1.x86_64.rpm | 18 kB 00:00:00 (6/10): libnetfilter_cttimeout-1.0.0-6.el7_7.1.x86_64.rpm | 18 kB 00:00:00 (7/10): libnetfilter_queue-1.0.2-2.el7_2.x86_64.rpm | 23 kB 00:00:00 (8/10): socat-1.7.3.2-2.el7.x86_64.rpm | 290 kB 00:00:00 (9/10): 548a0dcd865c16a50980420ddfa5fbccb8b59621179798e6dc905c9bf8af3b34-kubernetes-cni-0.7.5-0.x86_64.rpm | 10 MB 00:00:03 (10/10): 3ee7f2dff78e6fbb3ac3af8acb1a907f4bec1b1ef4cf627cbe02fa553707f2e9-kubelet-1.17.2-0.x86_64.rpm | 20 MB 00:00:06 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Total 5.3 MB/s | 54 MB 00:00:10 Retrieving key from https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg Importing GPG key 0x3E1BA8D5: Userid : "Google Cloud Packages RPM Signing Key <gc-team@google.com>" Fingerprint: 3749 e1ba 95a8 6ce0 5454 6ed2 f09c 394c 3e1b a8d5 From : https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg Retrieving key from https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg Importing GPG key 0xA7317B0F: Userid : "Google Cloud Packages Automatic Signing Key <gc-team@google.com>" Fingerprint: d0bc 747f d8ca f711 7500 d6fa 3746 c208 a731 7b0f From : https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : libnetfilter_cttimeout-1.0.0-6.el7_7.1.x86_64 1/10 Installing : socat-1.7.3.2-2.el7.x86_64 2/10 Installing : cri-tools-1.13.0-0.x86_64 3/10 Installing : kubectl-1.17.2-0.x86_64 4/10 Installing : libnetfilter_queue-1.0.2-2.el7_2.x86_64 5/10 Installing : libnetfilter_cthelper-1.0.0-10.el7_7.1.x86_64 6/10 Installing : conntrack-tools-1.4.4-5.el7_7.2.x86_64 7/10 Installing : kubernetes-cni-0.7.5-0.x86_64 8/10 Installing : kubelet-1.17.2-0.x86_64 9/10 Installing : kubeadm-1.17.2-0.x86_64 10/10 Verifying : libnetfilter_cthelper-1.0.0-10.el7_7.1.x86_64 1/10 Verifying : conntrack-tools-1.4.4-5.el7_7.2.x86_64 2/10 Verifying : libnetfilter_queue-1.0.2-2.el7_2.x86_64 3/10 Verifying : kubectl-1.17.2-0.x86_64 4/10 Verifying : cri-tools-1.13.0-0.x86_64 5/10 Verifying : kubeadm-1.17.2-0.x86_64 6/10 Verifying : kubernetes-cni-0.7.5-0.x86_64 7/10 Verifying : socat-1.7.3.2-2.el7.x86_64 8/10 Verifying : kubelet-1.17.2-0.x86_64 9/10 Verifying : libnetfilter_cttimeout-1.0.0-6.el7_7.1.x86_64 10/10 Installed: kubeadm.x86_64 0:1.17.2-0 kubelet.x86_64 0:1.17.2-0 Dependency Installed: conntrack-tools.x86_64 0:1.4.4-5.el7_7.2 cri-tools.x86_64 0:1.13.0-0 kubectl.x86_64 0:1.17.2-0 kubernetes-cni.x86_64 0:0.7.5-0 libnetfilter_cthelper.x86_64 0:1.0.0-10.el7_7.1 libnetfilter_cttimeout.x86_64 0:1.0.0-6.el7_7.1 libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 socat.x86_64 0:1.7.3.2-2.el7 Complete! [root@node203.yinzhengjie.org.cn ~]#
[root@node201.yinzhengjie.org.cn ~]# systemctl list-unit-files | grep kubelet kubelet.service disabled [root@node201.yinzhengjie.org.cn ~]# [root@node201.yinzhengjie.org.cn ~]# systemctl enable kubelet.service Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service. [root@node201.yinzhengjie.org.cn ~]# [root@node201.yinzhengjie.org.cn ~]# systemctl list-unit-files | grep kubelet kubelet.service enabled [root@node201.yinzhengjie.org.cn ~]# [root@node201.yinzhengjie.org.cn ~]#
[root@node202.yinzhengjie.org.cn ~]# systemctl list-unit-files | grep kubelet kubelet.service disabled [root@node202.yinzhengjie.org.cn ~]# [root@node202.yinzhengjie.org.cn ~]# systemctl enable kubelet.service Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service. [root@node202.yinzhengjie.org.cn ~]# [root@node202.yinzhengjie.org.cn ~]# systemctl list-unit-files | grep kubelet kubelet.service enabled [root@node202.yinzhengjie.org.cn ~]# [root@node202.yinzhengjie.org.cn ~]#
[root@node203.yinzhengjie.org.cn ~]# systemctl list-unit-files | grep kubelet kubelet.service disabled [root@node203.yinzhengjie.org.cn ~]# [root@node203.yinzhengjie.org.cn ~]# systemctl enable kubelet.service Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service. [root@node203.yinzhengjie.org.cn ~]# [root@node203.yinzhengjie.org.cn ~]# systemctl list-unit-files | grep kubelet kubelet.service enabled [root@node203.yinzhengjie.org.cn ~]# [root@node203.yinzhengjie.org.cn ~]#
3>.将k8s master节点的"/etc/sysconfig/kubelet"拷贝到其它三个k8s node节点(如果你的节点已经禁用了swap分区，此步骤可直接忽略)
[root@master200.yinzhengjie.org.cn ~]# cat /etc/sysconfig/kubelet KUBELET_EXTRA_ARGS="--fail-swap-on=false" [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# scp /etc/sysconfig/kubelet node201.yinzhengjie.org.cn:/etc/sysconfig/kubelet root@node201.yinzhengjie.org.cn's password: kubelet 100% 42 23.3KB/s 00:00 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# scp /etc/sysconfig/kubelet node202.yinzhengjie.org.cn:/etc/sysconfig/kubelet root@node202.yinzhengjie.org.cn's password: kubelet 100% 42 32.2KB/s 00:00 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]# scp /etc/sysconfig/kubelet node203.yinzhengjie.org.cn:/etc/sysconfig/kubelet root@node203.yinzhengjie.org.cn's password: kubelet 100% 42 21.7KB/s 00:00 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
4>.各节点加入到k8s集群
[root@node201.yinzhengjie.org.cn ~]# kubeadm join 172.200.1.200:6443 --token gu02ed.9k1rpyl4mtkhvdpk > --discovery-token-ca-cert-hash sha256:755ff7d07588b75aec769e566e4257d832cf6d1ec79bcdbd655dad80f30a6794 W0204 20:11:11.564558 32762 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set. [preflight] Running pre-flight checks [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/ [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service' [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' [kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.17" ConfigMap in the kube-system namespace [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster. [root@node201.yinzhengjie.org.cn ~]# [root@node201.yinzhengjie.org.cn ~]# echo $? 0 [root@node201.yinzhengjie.org.cn ~]#
[root@node202.yinzhengjie.org.cn ~]# kubeadm join 172.200.1.200:6443 --token gu02ed.9k1rpyl4mtkhvdpk --discovery-token-ca-cert-hash sha256:755ff7d07588b75aec769e566e4257d832cf6d1ec79bcdbd655dad80f30a6794 W0204 20:26:07.049488 31891 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set. [preflight] Running pre-flight checks [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/ [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service' [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' [kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.17" ConfigMap in the kube-system namespace [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster. [root@node202.yinzhengjie.org.cn ~]# [root@node202.yinzhengjie.org.cn ~]# echo $? 0 [root@node202.yinzhengjie.org.cn ~]#
[root@node203.yinzhengjie.org.cn ~]# kubeadm join 172.200.1.200:6443 --token gu02ed.9k1rpyl4mtkhvdpk --discovery-token-ca-cert-hash sha256:755ff7d07588b75aec769e566e4257d832cf6d1ec79bcdbd655dad80f30a6794 W0204 20:26:19.457993 32429 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set. [preflight] Running pre-flight checks [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/ [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service' [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' [kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.17" ConfigMap in the kube-system namespace [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster. [root@node203.yinzhengjie.org.cn ~]#
5>.将节点加入成功后,再来k8s master节点查看节点是否加入成功
[root@master200.yinzhengjie.org.cn ~]# kubectl get nodes #如果k8s master和k8s node状态的状态为"Ready"说明k8s集群搭建完毕啦~ NAME STATUS ROLES AGE VERSION master200.yinzhengjie.org.cn Ready master 47m v1.17.2 node201.yinzhengjie.org.cn Ready <none> 16m v1.17.2 node202.yinzhengjie.org.cn Ready <none> 78s v1.17.2 node203.yinzhengjie.org.cn Ready <none> 66s v1.17.2 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
五.配置k8s node节点也可以管理api server服务器
1>.在各个k8s node节点创建"~/.kubu"目录
[root@node201.yinzhengjie.org.cn ~]# mkdir ~/.kube [root@node201.yinzhengjie.org.cn ~]# [root@node201.yinzhengjie.org.cn ~]# ll .kube/ total 0 [root@node201.yinzhengjie.org.cn ~]# [root@node201.yinzhengjie.org.cn ~]#
[root@node202.yinzhengjie.org.cn ~]# mkdir .kube [root@node202.yinzhengjie.org.cn ~]# [root@node202.yinzhengjie.org.cn ~]# ll .kube/ total 0 [root@node202.yinzhengjie.org.cn ~]# [root@node202.yinzhengjie.org.cn ~]#
[root@node203.yinzhengjie.org.cn ~]# mkdir ~/.kube [root@node203.yinzhengjie.org.cn ~]# [root@node203.yinzhengjie.org.cn ~]# ll .kube/ total 0 [root@node203.yinzhengjie.org.cn ~]# [root@node203.yinzhengjie.org.cn ~]#
2>.将k8s master节点的"/etc/kubernetes/admin.conf"配置文件拷贝到各个k8s node节点的"~/.kube/"目录并命名为"config"
[root@master200.yinzhengjie.org.cn ~]# scp /etc/kubernetes/admin.conf node201.yinzhengjie.org.cn:~/.kube/config root@node201.yinzhengjie.org.cn's password: admin.conf 100% 5453 2.2MB/s 00:00 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# scp /etc/kubernetes/admin.conf node202.yinzhengjie.org.cn:~/.kube/config root@node202.yinzhengjie.org.cn's password: admin.conf 100% 5453 3.6MB/s 00:00 [root@master200.yinzhengjie.org.cn ~]#
[root@master200.yinzhengjie.org.cn ~]# scp /etc/kubernetes/admin.conf node203.yinzhengjie.org.cn:~/.kube/config root@node203.yinzhengjie.org.cn's password: admin.conf 100% 5453 2.9MB/s 00:00 [root@master200.yinzhengjie.org.cn ~]# [root@master200.yinzhengjie.org.cn ~]#
3>.在k8s node节点查看node信息
[root@node201.yinzhengjie.org.cn ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION master200.yinzhengjie.org.cn Ready master 61m v1.17.2 node201.yinzhengjie.org.cn Ready <none> 29m v1.17.2 node202.yinzhengjie.org.cn Ready <none> 15m v1.17.2 node203.yinzhengjie.org.cn Ready <none> 14m v1.17.2 [root@node201.yinzhengjie.org.cn ~]# [root@node201.yinzhengjie.org.cn ~]# [root@node201.yinzhengjie.org.cn ~]# kubectl config view apiVersion: v1 clusters: - cluster: certificate-authority-data: DATA+OMITTED server: https://172.200.1.200:6443 name: kubernetes contexts: - context: cluster: kubernetes user: kubernetes-admin name: kubernetes-admin@kubernetes current-context: kubernetes-admin@kubernetes kind: Config preferences: {} users: - name: kubernetes-admin user: client-certificate-data: REDACTED client-key-data: REDACTED [root@node201.yinzhengjie.org.cn ~]# [root@node201.yinzhengjie.org.cn ~]#
4>.API Server的证书说明
早期版本的K8S的API Server支持2个套接字，分别监听两个不同的端口，一个是8080(http协议),另一个是6443(https),后来官方考虑http并不安全于是采用了只保留6443端口。
API Server是需要双向认证的，即除了API Server有自己的证书文件还要求客户端也提供证书文件，这就是为什么上面会将"/etc/kubernetes/admin.conf"拷贝到"~/.kube/config"的原因。
我们使用kubeadmin部署K8S集群时，API Server组件需要的证书会自动帮咱们生成，默认存放目录在"/etc/kubernetes/pki/",如下图所示。
5>.kubernetes快速入门
博主推荐阅读: https://www.cnblogs.com/yinzhengjie/p/12261980.html