一:Kubernetes介绍和环境准备
1. 什么是K8S
Kubernetes是一个全新的基于容器技术的分布式架构领先方案,是一个开放的开发平台。不局限于任何一种语言,没有限定任何编程接口。是一个完备的分布式系统支撑平台。它构建在docker之上,提供应用部署、维护、扩展机制等功能,利用Kubernetes能方便地管理跨机器运行容器化的应用。
2. 主要功能
- 使用Docker对应用程序包装、实例化;
- 以集群的方式运行、管理跨机器的容器;
- 解决Docker跨机器容器之间的通讯问题;
- Kubernetes的自我修复机制使得容器集群总是运行在用户期望的状态;
3. 组件介绍
Kubernetes中的:Node、Pod、Replication Controller、Service等都可以看作一种“资源对象”,几乎所有的资源对象都可以通过kubectl工具(API调用)执行增、删、改、查等操作并将其保存在etcd中持久化存储。
- Master:集群控制管理节点,所有的命令都经由master处理。
- Node:是kubernetes集群的工作负载节点。Master为其分配工作,当某个Node宕机时,Master会将其工作负载自动转移到其他节点。
- Pod:是kubernetes最重要也是最基本的概念。每个Pod都会包含一个 “根容器”,还会包含一个或者多个紧密相连的业务容器。
- ETCD:k8s中,所有数据的存储以及操作记录都在etcd中进行存储,一旦故障,可能导致整个集群的瘫痪或者数据丢失。
- Label:是一个key=value的键值对,其中key与value由用户自己指定。可以附加到各种资源对象上,一个资源对象可以定义任意数量的Label。
- RC:Replication Controller声明某个Pod的副本数在任意时刻都符合某个预期值。定义包含如下:
(1)Pod期待的副本数(replicas);
(2)用于筛选目标Pod的Label Selector;
(3)当Pod副本数小于期望时,用于新的创建Pod的模板template ;
(4)通过改变RC里的Pod副本数量,可以实现Pod的扩容或缩容功能;
(5)通过改变RC里Pod模板中的镜像版本,可以实现Pod的滚动升级功能;
3.1 master节点
- API Server
集群控制管理节点,所有的命令都经由master处理,所有资源增删改查的唯一入口。
提供集群管理的REST API接口,包括认证授权、数据校验以及集群状态变更等;
只有API Server才直接操作etcd;
其他模块通过API Server查询或修改数据;
提供其他模块之间的数据交互和通信的枢纽;
- Scheduler
负责分配调度Pod到集群内的node节点;
监听kube-apiserver,查询还未分配Node的Pod;
根据调度策略为这些Pod分配节点;
- Controller Manager
所有其他群集级别的功能,目前由控制器Manager执行。资源对象的自动化控制中心;
由一系列的控制器组成,它通过apiserver监控整个集群的状态,并确保集群处于预期的工作状态;
- ETCD
所有持久化的状态信息存储在ETCD中。
3.2 node节点
默认情况下,Node节点可动态增加到kubernetes集群中,一旦Node被纳入集群管理范围,kubelet会定时向Master汇报自身的情况,以及之前有哪些Pod在运行等,这样Master可以获知每个Node的资源使用情况,并实现高效均衡的资源调度策略。如果Node没有按时上报信息,则会被Master判断为失联,Node状态会被标记为Not Ready,随后Master会触发工作负载转移流程。
- Kubelet
管理Pods以及容器、镜像、 Volume等,实现对集群对节点的管理。
- Kube-proxy
提供网络代理以及负载均衡,实现与Service通信。
- Docker Engine
负责节点的容器的管理工作。
4. 实验环境准备
4.1 规划
本次实验是基于2个node+1个master,每个节点都跑一个ETCD。
1. 操作系统 CentOS-7.x-x86_64。 |
||||
主机名 | IP地址(NAT) | CPU | 内存 | 描述 |
k8s-master | eth0 : 10.0.0.25 | 1VCPU | 2G | Kubernets Master节点/Etcd节点 |
k8s-node-1 | eth0 : 10.0.0.26 | 1VCPU | 2G | Kubernets Node节点/ Etcd节点 |
k8s-node-1 | eth0 : 10.0.0.27 | 1VCPU | 2G | Kubernets Node节点/ Etcd节点 |
4.2 网络设置
4.3 配置静态IP地址
#将 UUID 和 MAC 地址已经其它配置删除掉,3个节点除了IP和主机名不同其他相同。
[root@k8s-master ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0 TYPE=Ethernet BOOTPROTO=static NAME=eth0 DEVICE=eth0 ONBOOT=yes IPADDR=10.0.0.25 NETMASK=255.255.255.0 GATEWAY=10.0.0.254 DNS=223.5.5.5 #重启网络服务 [root@k8s-master ~]# systemctl restart network #设置 DNS 解析 [root@k8s-master ~]# vi /etc/resolv.conf nameserver 223.5.5.5
4.4 关闭selinux、防火墙
setenforce 0 sed -i 's#SELINUX=enforcing#SELINUX=disabled#' /etc/selinux/config systemctl disable firewalld.service systemctl stop firewalld.service systemctl stop NetworkManager systemctl disable NetworkManager
4.5 设置主机名解析
3个节点都做
#3个节点都一样 [root@k8s-master ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 10.0.0.25 k8s-master 10.0.0.26 k8s-node-1 10.0.0.27 k8s-node-2
4.6 配置epel源
3个节点都做
rpm -ivh http://mirrors.aliyun.com/epel/epel-release-latest-7.noarch.rpm #下载常用命令 yum install -y net-tools vim lrzsz tree screen lsof tcpdump nc mtr nmap #保证能上网 [root@k8s-master ~]# ping www.baidu.com -c3 PING www.a.shifen.com (61.135.169.121) 56(84) bytes of data. 64 bytes from 61.135.169.121: icmp_seq=1 ttl=128 time=5.41 ms 64 bytes from 61.135.169.121: icmp_seq=2 ttl=128 time=6.55 ms 64 bytes from 61.135.169.121: icmp_seq=3 ttl=128 time=8.97 ms --- www.a.shifen.com ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2023ms rtt min/avg/max/mdev = 5.418/6.981/8.974/1.486 ms
4.7 配置免秘钥登录
只在master节点做
[root@k8s-master ~]# ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: b1:a0:5b:02:57:0e:8f:1e:25:bf:46:1f:d1:f3:24:c4 root@k8s-master The key's randomart image is: +--[ RSA 2048]----+ | o o .+. | | X .E . | | . + * o = | | + + + + . | | + + S | | = | | . | | | | | +-----------------+ [root@k8s-master ~]# ssh-copy-id k8s-master The authenticity of host 'k8s-master (10.0.0.25)' can't be established. ECDSA key fingerprint is 75:5c:83:a1:b4:cc:bf:28:71:a5:d5:d1:94:35:3c:9a. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@k8s-master's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'k8s-master'" and check to make sure that only the key(s) you wanted were added. [root@k8s-master ~]# ssh-copy-id k8s-node-1 The authenticity of host 'k8s-node-1 (10.0.0.26)' can't be established. ECDSA key fingerprint is 75:5c:83:a1:b4:cc:bf:28:71:a5:d5:d1:94:35:3c:9a. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@k8s-node-1's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'k8s-node-1'" and check to make sure that only the key(s) you wanted were added. [root@k8s-master ~]# ssh-copy-id k8s-node-2 The authenticity of host 'k8s-node-2 (10.0.0.27)' can't be established. ECDSA key fingerprint is 75:5c:83:a1:b4:cc:bf:28:71:a5:d5:d1:94:35:3c:9a. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@k8s-node-2's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'k8s-node-2'" and check to make sure that only the key(s) you wanted were added.
二:Kubernetes集群初始化
1. 安装Docker
3个节点都做
#第一步:使用国内Docker源 [root@k8s-master ~]# cd /etc/yum.repos.d/ [root@k8s-master yum.repos.d]# wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo #第二步:Docker安装: [root@k8s-master ~]# yum install -y docker-ce #第三步:启动后台进程: [root@k8s-master ~]# systemctl start docker
[root@k8s-master ~]# systemctl enable docker
2. 准备部署目录
3个节点都做
[root@k8s-master ~]# mkdir -p /opt/kubernetes/{cfg,bin,ssl,log} [root@k8s-master ~]# tree -L 1 /opt/kubernetes/ /opt/kubernetes/ ├── bin ├── cfg ├── log └── ssl
3. 准备软件包
百度网盘下载地址:
链接: https://pan.baidu.com/s/1kUNV7t8SSF_yuP_WDaXj5g
密码: wexh
4. 解压软件包
master节点做
#将下载软件包上传到/usr/local/src/ [root@k8s-master ~]# ll /usr/local/src/ 总用量 579812 -rw-r--r-- 1 root root 593725046 6月 2 13:27 k8s-v1.10.1-manual.zip #解压 cd /usr/local/src/ unzip k8s-v1.10.1-manual.zip cd /usr/local/src/k8s-v1.10.1-manual/k8s-v1.10.1 tar zxf kubernetes.tar.gz tar zxf kubernetes-server-linux-amd64.tar.gz tar zxf kubernetes-client-linux-amd64.tar.gz tar zxf kubernetes-node-linux-amd64.tar.gz mv ./* /usr/local/src/
5. 修改环境变量
3个节点都做
[root@k8s-master ~]# sed -i 's#PATH=$PATH:$HOME/bin#PATH=$PATH:$HOME/bin:/opt/kubernetes/bin#g' /root/.bash_profile [root@k8s-master ~]# source /root/.bash_profile
6. 手动制作CA证书
6.1 安装 CFSSL
master节点做
[root@k8s-master ~]# cd /usr/local/src/ [root@k8s-master src]# wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 [root@k8s-master src]# wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 [root@k8s-master src]# wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 [root@k8s-master src]# chmod +x cfssl* [root@k8s-master src]# mv cfssl-certinfo_linux-amd64 /opt/kubernetes/bin/cfssl-certinfo [root@k8s-master src]# mv cfssljson_linux-amd64 /opt/kubernetes/bin/cfssljson [root@k8s-master src]# mv cfssl_linux-amd64 /opt/kubernetes/bin/cfssl #复制cfssl命令文件到k8s-node1和k8s-node2节点。如果实际中多个节点,就都需要同步复制。 [root@k8s-master src]# scp /opt/kubernetes/bin/cfssl* 10.0.0.26: /opt/kubernetes/bin [root@k8s-master src]# scp /opt/kubernetes/bin/cfssl* 10.0.0.27: /opt/kubernetes/bin
6.2 初始化CFSSL
master节点做
[root@k8s-master src]# mkdir ssl && cd ssl [root@k8s-master ssl]# cfssl print-defaults config > config.json [root@k8s-master ssl]# cfssl print-defaults csr > csr.json
6.3 创建用来生成 CA 文件的 JSON 配置文件
master节点做
[root@k8s-master ssl]# cd /usr/local/src/ssl [root@k8s-master ssl]# vim ca-config.json { "signing": { "default": { "expiry": "8760h" }, "profiles": { "kubernetes": { "usages": [ "signing", "key encipherment", "server auth", "client auth" ], "expiry": "8760h" } } } }
6.4 创建用来生成 CA 证书签名请求(CSR)的 JSON 配置文件
master节点做
[root@k8s-master ssl]# cd /usr/local/src/ssl [root@k8s-master ssl]# vim ca-csr.json { "CN": "kubernetes", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "System" } ] }
6.5 生成CA证书(ca.pem)和密钥(ca-key.pem)
master节点做
[root@k8s-master ssl]# cd /usr/local/src/ssl [root@k8s-master ssl]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca [root@k8s-master ssl]# ls -l ca* -rw-r--r-- 1 root root 290 6月 2 15:58 ca-config.json -rw-r--r-- 1 root root 1001 6月 2 15:58 ca.csr -rw-r--r-- 1 root root 208 6月 2 15:58 ca-csr.json -rw------- 1 root root 1675 6月 2 15:58 ca-key.pem -rw-r--r-- 1 root root 1359 6月 2 15:58 ca.pem
6.6 分发证书
master节点做
[root@k8s-master ssl]# cp ca.csr ca.pem ca-key.pem ca-config.json /opt/kubernetes/ssl [root@k8s-master ssl]# scp ca.csr ca.pem ca-key.pem ca-config.json 10.0.0.26:/opt/kubernetes/ssl [root@k8s-master ssl]# scp ca.csr ca.pem ca-key.pem ca-config.json 10.0.0.27:/opt/kubernetes/ssl
7. ETCD集群部署
7.1 准备etcd软件包
master节点做
[root@k8s-master ~]# cd /usr/local/src/ [root@k8s-master src]# wget https://github.com/coreos/etcd/releases/download/v3.2.18/etcd-v3.2.18-linux-amd64.tar.gz [root@k8s-master src]# tar zxf etcd-v3.2.18-linux-amd64.tar.gz [root@k8s-master src]# cd etcd-v3.2.18-linux-amd64 [root@k8s-master etcd-v3.2.18-linux-amd64]# cp etcd etcdctl /opt/kubernetes/bin/ [root@k8s-master etcd-v3.2.18-linux-amd64]# scp etcd etcdctl 10.0.0.26:/opt/kubernetes/bin/ [root@k8s-master etcd-v3.2.18-linux-amd64]# scp etcd etcdctl 10.0.0.27:/opt/kubernetes/bin/
7.2 创建 etcd 证书签名请求
master节点做
[root@k8s-master ~]# cd /usr/local/src/ssl/ [root@k8s-master ssl]# vim etcd-csr.json { "CN": "etcd", "hosts": [ "127.0.0.1", "10.0.0.25", "10.0.0.26", "10.0.0.27" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "System" } ] }
7.4 生成 etcd 证书和私钥
master节点做
[root@k8s-master ssl]# cd /usr/local/src/ssl/ [root@k8s-master ssl]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem -ca-key=/opt/kubernetes/ssl/ca-key.pem -config=/opt/kubernetes/ssl/ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd #生成以下证书文件 [root@k8s-master ssl]# ls -l etcd* -rw-r--r-- 1 root root 1062 6月 3 10:48 etcd.csr -rw-r--r-- 1 root root 275 6月 3 10:45 etcd-csr.json -rw------- 1 root root 1675 6月 3 10:48 etcd-key.pem -rw-r--r-- 1 root root 1436 6月 3 10:48 etcd.pem [root@k8s-master ssl]#
7.4 证书移动到/opt/kubernetes/ssl目录下
master节点做
[root@k8s-master ssl]# cp etcd*.pem /opt/kubernetes/ssl [root@k8s-master ssl]# scp etcd*.pem 10.0.0.26:/opt/kubernetes/ssl [root@k8s-master ssl]# scp etcd*.pem 10.0.0.27:/opt/kubernetes/ssl
7.5 设置ETCD配置文件
master节点配置文件
[root@k8s-master ~]# vim /opt/kubernetes/cfg/etcd.conf #[member] ETCD_NAME="etcd-master" ETCD_DATA_DIR="/var/lib/etcd/default.etcd" #ETCD_SNAPSHOT_COUNTER="10000" #ETCD_HEARTBEAT_INTERVAL="100" #ETCD_ELECTION_TIMEOUT="1000" ETCD_LISTEN_PEER_URLS="https://10.0.0.25:2380" ETCD_LISTEN_CLIENT_URLS="https://10.0.0.25:2379,https://127.0.0.1:2379" #ETCD_MAX_SNAPSHOTS="5" #ETCD_MAX_WALS="5" #ETCD_CORS="" #[cluster] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.0.0.25:2380" # if you use different ETCD_NAME (e.g. test), # set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..." ETCD_INITIAL_CLUSTER="etcd-master=https://10.0.0.25:2380,etcd-node1=https://10.0.0.26:2380,etcd-node2=https://10.0.0.27:2380" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster" ETCD_ADVERTISE_CLIENT_URLS="https://10.0.0.25:2379" #[security] CLIENT_CERT_AUTH="true" ETCD_CA_FILE="/opt/kubernetes/ssl/ca.pem" ETCD_CERT_FILE="/opt/kubernetes/ssl/etcd.pem" ETCD_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem" PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_CA_FILE="/opt/kubernetes/ssl/ca.pem" ETCD_PEER_CERT_FILE="/opt/kubernetes/ssl/etcd.pem" ETCD_PEER_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"
node1节点配置文件
[root@k8s-node-1 ~]# vim /opt/kubernetes/cfg/etcd.conf #[member] ETCD_NAME="etcd-node1" ETCD_DATA_DIR="/var/lib/etcd/default.etcd" #ETCD_SNAPSHOT_COUNTER="10000" #ETCD_HEARTBEAT_INTERVAL="100" #ETCD_ELECTION_TIMEOUT="1000" ETCD_LISTEN_PEER_URLS="https://10.0.0.26:2380" ETCD_LISTEN_CLIENT_URLS="https://10.0.0.26:2379,https://127.0.0.1:2379" #ETCD_MAX_SNAPSHOTS="5" #ETCD_MAX_WALS="5" #ETCD_CORS="" #[cluster] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.0.0.26:2380" # if you use different ETCD_NAME (e.g. test), # set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..." ETCD_INITIAL_CLUSTER="etcd-master=https://10.0.0.25:2380,etcd-node1=https://10.0.0.26:2380,etcd-node2=https://10.0.0.27:2380" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster" ETCD_ADVERTISE_CLIENT_URLS="https://10.0.0.26:2379" #[security] CLIENT_CERT_AUTH="true" ETCD_CA_FILE="/opt/kubernetes/ssl/ca.pem" ETCD_CERT_FILE="/opt/kubernetes/ssl/etcd.pem" ETCD_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem" PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_CA_FILE="/opt/kubernetes/ssl/ca.pem" ETCD_PEER_CERT_FILE="/opt/kubernetes/ssl/etcd.pem" ETCD_PEER_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"
node2节点配置文件
[root@k8s-node-2 ~]# vim /opt/kubernetes/cfg/etcd.conf #[member] ETCD_NAME="etcd-node2" ETCD_DATA_DIR="/var/lib/etcd/default.etcd" #ETCD_SNAPSHOT_COUNTER="10000" #ETCD_HEARTBEAT_INTERVAL="100" #ETCD_ELECTION_TIMEOUT="1000" ETCD_LISTEN_PEER_URLS="https://10.0.0.27:2380" ETCD_LISTEN_CLIENT_URLS="https://10.0.0.27:2379,https://127.0.0.1:2379" #ETCD_MAX_SNAPSHOTS="5" #ETCD_MAX_WALS="5" #ETCD_CORS="" #[cluster] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.0.0.27:2380" # if you use different ETCD_NAME (e.g. test), # set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..." ETCD_INITIAL_CLUSTER="etcd-master=https://10.0.0.25:2380,etcd-node1=https://10.0.0.26:2380,etcd-node2=https://10.0.0.27:2380" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster" ETCD_ADVERTISE_CLIENT_URLS="https://10.0.0.27:2379" #[security] CLIENT_CERT_AUTH="true" ETCD_CA_FILE="/opt/kubernetes/ssl/ca.pem" ETCD_CERT_FILE="/opt/kubernetes/ssl/etcd.pem" ETCD_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem" PEER_CLIENT_CERT_AUTH="true" ETCD_PEER_CA_FILE="/opt/kubernetes/ssl/ca.pem" ETCD_PEER_CERT_FILE="/opt/kubernetes/ssl/etcd.pem" ETCD_PEER_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"
7.6 创建ETCD系统服务
master节点做
[root@k8s-master ~]# vim /etc/systemd/system/etcd.service [Unit] Description=Etcd Server After=network.target [Service] Type=simple WorkingDirectory=/var/lib/etcd EnvironmentFile=-/opt/kubernetes/cfg/etcd.conf # set GOMAXPROCS to number of processors ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /opt/kubernetes/bin/etcd" Type=notify [Install] WantedBy=multi-user.target
[root@k8s-master ~]# scp /etc/systemd/system/etcd.service 10.0.0.26:/etc/systemd/system/
[root@k8s-master ~]# scp /etc/systemd/system/etcd.service 10.0.0.27:/etc/systemd/system/
7.7 重新加载系统服务
3个节点都做
systemctl daemon-reload mkdir /var/lib/etcd systemctl enable etcd systemctl start etcd systemctl status etcd
7.8 验证集群
master节点做
[root@k8s-master ~]# etcdctl --endpoints=https://10.0.0.25:2379 --ca-file=/opt/kubernetes/ssl/ca.pem --cert-file=/opt/kubernetes/ssl/etcd.pem --key-file=/opt/kubernetes/ssl/etcd-key.pem cluster-health
如下图所示证明ETCD集群部署成功
三:Kubernetes的master节点部署
1. 部署API server服务
1.1 准备软件包
master节点做
因为K8S是用二进制文件构写的所以不需要编译,只需要拷贝文件就行。
[root@k8s-master ~]# cd /usr/local/src/kubernetes [root@k8s-master kubernetes]# cp server/bin/kube-apiserver /opt/kubernetes/bin/ [root@k8s-master kubernetes]# cp server/bin/kube-controller-manager /opt/kubernetes/bin/ [root@k8s-master kubernetes]# cp server/bin/kube-scheduler /opt/kubernetes/bin/
1.2 创建生成CSR的 JSON 配置文件
master节点做
[root@k8s-master src]# cd /usr/local/src/ssl/ [root@k8s-master ssl]# vim kubernetes-csr.json { "CN": "kubernetes", "hosts": [ "127.0.0.1", "10.0.0.25", "10.1.0.1", "kubernetes", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "System" } ] }
1.3 生成 kubernetes 证书和私钥
master节点做
[root@k8s-master ssl]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem -ca-key=/opt/kubernetes/ssl/ca-key.pem -config=/opt/kubernetes/ssl/ca-config.json -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes [root@k8s-master ssl]# cp kubernetes*.pem /opt/kubernetes/ssl/ [root@k8s-master ssl]# scp kubernetes*.pem 10.0.0.26:/opt/kubernetes/ssl/ [root@k8s-master ssl]# scp kubernetes*.pem 10.0.0.27:/opt/kubernetes/ssl/
1.4 创建 kube-apiserver 使用的客户端 token 文件
master节点做
[root@k8s-master ~]# head -c 16 /dev/urandom | od -An -t x | tr -d ' ' 841276913ac026513f75d267f0fc5212 [root@k8s-master ~]# vim /opt/kubernetes/ssl/bootstrap-token.csv 841276913ac026513f75d267f0fc5212,kubelet-bootstrap,10001,"system:kubelet-bootstrap" #根据上面自己生成的填写
1.5 创建基础用户名/密码认证配置
master节点做
[root@k8s-master ~]# vim /opt/kubernetes/ssl/basic-auth.csv admin,admin,1 readonly,readonly,2
1.6 部署Kubernetes API Server
master节点做
[root@k8s-master ~]# vim /usr/lib/systemd/system/kube-apiserver.service [Unit] Description=Kubernetes API Server Documentation=https://github.com/GoogleCloudPlatform/kubernetes After=network.target [Service] ExecStart=/opt/kubernetes/bin/kube-apiserver --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota,NodeRestriction --bind-address=10.0.0.25 --insecure-bind-address=127.0.0.1 --authorization-mode=Node,RBAC --runtime-config=rbac.authorization.k8s.io/v1 --kubelet-https=true --anonymous-auth=false --basic-auth-file=/opt/kubernetes/ssl/basic-auth.csv --enable-bootstrap-token-auth --token-auth-file=/opt/kubernetes/ssl/bootstrap-token.csv --service-cluster-ip-range=10.1.0.0/16 --service-node-port-range=20000-40000 --tls-cert-file=/opt/kubernetes/ssl/kubernetes.pem --tls-private-key-file=/opt/kubernetes/ssl/kubernetes-key.pem --client-ca-file=/opt/kubernetes/ssl/ca.pem --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem --etcd-cafile=/opt/kubernetes/ssl/ca.pem --etcd-certfile=/opt/kubernetes/ssl/kubernetes.pem --etcd-keyfile=/opt/kubernetes/ssl/kubernetes-key.pem --etcd-servers=https://10.0.0.25:2379,https://10.0.0.26:2379,https://10.0.0.27:2379 --enable-swagger-ui=true --allow-privileged=true --audit-log-maxage=30 --audit-log-maxbackup=3 --audit-log-maxsize=100 --audit-log-path=/opt/kubernetes/log/api-audit.log --event-ttl=1h --v=2 --logtostderr=false --log-dir=/opt/kubernetes/log Restart=on-failure RestartSec=5 Type=notify LimitNOFILE=65536 [Install] WantedBy=multi-user.target
1.7 启动APIserver
master节点做
[root@k8s-master ~]# systemctl daemon-reload [root@k8s-master ~]# systemctl enable kube-apiserver [root@k8s-master ~]# systemctl start kube-apiserver [root@k8s-master ~]# systemctl status kube-apiserver
2. 部署Controller Manager服务
master节点做
[root@k8s-master ~]# vim /usr/lib/systemd/system/kube-controller-manager.service [Unit] Description=Kubernetes Controller Manager Documentation=https://github.com/GoogleCloudPlatform/kubernetes [Service] ExecStart=/opt/kubernetes/bin/kube-controller-manager --address=127.0.0.1 --master=http://127.0.0.1:8080 --allocate-node-cidrs=true --service-cluster-ip-range=10.1.0.0/16 --cluster-cidr=10.2.0.0/16 --cluster-name=kubernetes --cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem --cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem --service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem --root-ca-file=/opt/kubernetes/ssl/ca.pem --leader-elect=true --v=2 --logtostderr=false --log-dir=/opt/kubernetes/log Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target ###启动Controller Manager [root@k8s-master ~]# systemctl daemon-reload [root@k8s-master ~]# systemctl enable kube-controller-manager [root@k8s-master ~]# systemctl start kube-controller-manager [root@k8s-master ~]# systemctl status kube-controller-manager
3. 部署Kubernetes Scheduler服务
master节点上做
[root@k8s-master ~]# vim /usr/lib/systemd/system/kube-scheduler.service [Unit] Description=Kubernetes Scheduler Documentation=https://github.com/GoogleCloudPlatform/kubernetes [Service] ExecStart=/opt/kubernetes/bin/kube-scheduler --address=127.0.0.1 --master=http://127.0.0.1:8080 --leader-elect=true --v=2 --logtostderr=false --log-dir=/opt/kubernetes/log Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target ###启动scheduler服务 [root@k8s-master ~]# systemctl daemon-reload [root@k8s-master ~]# systemctl enable kube-scheduler [root@k8s-master ~]# systemctl start kube-scheduler [root@k8s-master ~]# systemctl status kube-scheduler
4. 部署kubectl 命令行工具
master节点做
###准备二进制命令包 [root@k8s-master ~]# cd /usr/local/src/kubernetes/client/bin [root@k8s-master bin]# cp kubectl /opt/kubernetes/bin/ ###创建 admin 证书签名请求 [root@linux-node1 ~]# cd /usr/local/src/ssl/ [root@linux-node1 ssl]# vim admin-csr.json { "CN": "admin", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "system:masters", "OU": "System" } ] } ###生成 admin 证书和私钥 [root@k8s-master ssl]# cd /usr/local/src/ssl/ [root@k8s-master ssl]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem -ca-key=/opt/kubernetes/ssl/ca-key.pem -config=/opt/kubernetes/ssl/ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin [root@k8s-master ssl]# mv admin*.pem /opt/kubernetes/ssl/ ###设置集群参数 [root@k8s-master ssl]# kubectl config set-cluster kubernetes --certificate-authority=/opt/kubernetes/ssl/ca.pem --embed-certs=true --server=https://10.0.0.25:6443 ###设置客户端认证参数 [root@k8s-master ssl]# kubectl config set-credentials admin --client-certificate=/opt/kubernetes/ssl/admin.pem --embed-certs=true --client-key=/opt/kubernetes/ssl/admin-key.pem ###设置上下文参数 [root@k8s-master ssl]# kubectl config set-context kubernetes --cluster=kubernetes --user=admin ###使用kubectl工具 [root@k8s-master ~]# ll /root/.kube/config [root@k8s-master ~]# kubectl get cs
如下图所示证明master节点部署成功
四:Kubernetes的node节点部署
1. kubelet部署
master节点做
###二进制包准备 [root@k8s-master ~]# cd /usr/local/src/kubernetes/server/bin/ [root@k8s-master bin]# cp kubelet kube-proxy /opt/kubernetes/bin/ [root@k8s-master bin]# scp kubelet kube-proxy 10.0.0.26:/opt/kubernetes/bin/ [root@k8s-master bin]# scp kubelet kube-proxy 10.0.0.27:/opt/kubernetes/bin/ ###创建角色绑定 [root@k8s-master ~]# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap
###设置集群参数
[root@k8s-master ~]# kubectl config set-cluster kubernetes
--certificate-authority=/opt/kubernetes/ssl/ca.pem
--embed-certs=true
--server=https://10.0.0.25:6443
--kubeconfig=bootstrap.kubeconfig
###设置客户端认证参数
[root@k8s-master ~]# kubectl config set-credentials kubelet-bootstrap
--token=30e6006d17ccc1ef829d26d1323cc7d6
--kubeconfig=bootstrap.kubeconfig
###设置上下文参数
[root@k8s-master ~]# kubectl config set-context default
--cluster=kubernetes
--user=kubelet-bootstrap
--kubeconfig=bootstrap.kubeconfig
###选择默认上下文 [root@k8s-master ssl]# kubectl config use-context default --kubeconfig=bootstrap.kubeconfig [root@k8s-master ssl]# cp bootstrap.kubeconfig /opt/kubernetes/cfg [root@k8s-master ssl]# scp bootstrap.kubeconfig 10.0.0.26:/opt/kubernetes/cfg [root@k8s-master ssl]# scp bootstrap.kubeconfig 10.0.0.27:/opt/kubernetes/cfg ###设置CNI支持 3个节点都创建 mkdir -p /etc/cni/net.d vim /etc/cni/net.d/10-default.conf { "name": "flannel", "type": "flannel", "delegate": { "bridge": "docker0", "isDefaultGateway": true, "mtu": 1400 } } [root@k8s-master ~]# scp /etc/cni/net.d/10-default.conf 10.0.0.26:/etc/cni/net.d/10-default.conf [root@k8s-master ~]# scp /etc/cni/net.d/10-default.conf 10.0.0.27:/etc/cni/net.d/10-default.conf ###创建kubelet目录 #3个节点都创建 mkdir /var/lib/kubelet ###master节点配置文件 [root@k8s-master ~]# vim /usr/lib/systemd/system/kubelet.service [Unit] Description=Kubernetes Kubelet Documentation=https://github.com/GoogleCloudPlatform/kubernetes After=docker.service Requires=docker.service [Service] WorkingDirectory=/var/lib/kubelet ExecStart=/opt/kubernetes/bin/kubelet --address=10.0.0.25 --hostname-override=10.0.0.25 --pod-infra-container-image=mirrorgooglecontainers/pause-amd64:3.0 --experimental-bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig --cert-dir=/opt/kubernetes/ssl --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/kubernetes/bin/cni --cluster-dns=10.1.0.2 --cluster-domain=cluster.local. --hairpin-mode hairpin-veth --allow-privileged=true --fail-swap-on=false --logtostderr=true --v=2 --logtostderr=false --log-dir=/opt/kubernetes/log Restart=on-failure RestartSec=5 ###node1节点配置文件 [root@k8s-node-1 ~]# vim /usr/lib/systemd/system/kubelet.service [Unit] Description=Kubernetes Kubelet Documentation=https://github.com/GoogleCloudPlatform/kubernetes After=docker.service Requires=docker.service [Service] WorkingDirectory=/var/lib/kubelet ExecStart=/opt/kubernetes/bin/kubelet --address=10.0.0.26 --hostname-override=10.0.0.26 --pod-infra-container-image=mirrorgooglecontainers/pause-amd64:3.0 --experimental-bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig --cert-dir=/opt/kubernetes/ssl --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/kubernetes/bin/cni --cluster-dns=10.1.0.2 --cluster-domain=cluster.local. --hairpin-mode hairpin-veth --allow-privileged=true --fail-swap-on=false --logtostderr=true --v=2 --logtostderr=false --log-dir=/opt/kubernetes/log Restart=on-failure RestartSec=5 ###node2节点配置文件 [root@k8s-node-2 ~]# vim /usr/lib/systemd/system/kubelet.service [Unit] Description=Kubernetes Kubelet Documentation=https://github.com/GoogleCloudPlatform/kubernetes After=docker.service Requires=docker.service [Service] WorkingDirectory=/var/lib/kubelet ExecStart=/opt/kubernetes/bin/kubelet --address=10.0.0.27 --hostname-override=10.0.0.27 --pod-infra-container-image=mirrorgooglecontainers/pause-amd64:3.0 --experimental-bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig --cert-dir=/opt/kubernetes/ssl --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/kubernetes/bin/cni --cluster-dns=10.1.0.2 --cluster-domain=cluster.local. --hairpin-mode hairpin-veth --allow-privileged=true --fail-swap-on=false --logtostderr=true --v=2 --logtostderr=false --log-dir=/opt/kubernetes/log Restart=on-failure RestartSec=5
###启动Kubelet #3个节点都做 systemctl daemon-reload systemctl enable kubelet systemctl start kubelet systemctl status kubelet
###查看csr请求
kubectl get csr
###批准kubelet的TLS证书请求
kubectl get csr|grep 'Pending' | awk 'NR>0{print $1}'| xargs kubectl certificate approve
###查看节点状态
kubectl get node
###配置kube-proxy使用LVS [root@k8s-master ~]# yum install -y ipvsadm ipset conntrack [root@k8s-node-1 ~]# yum install -y ipvsadm ipset conntrack [root@k8s-node-2 ~]# yum install -y ipvsadm ipset conntrack ###创建 kube-proxy 证书请求 [root@k8s-master ~]# cd /usr/local/src/ssl/ [root@k8s-master ssl]# vim kube-proxy-csr.json { "CN": "system:kube-proxy", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "System" } ] } ###生成证书 [root@k8s-master ssl]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem -ca-key=/opt/kubernetes/ssl/ca-key.pem -config=/opt/kubernetes/ssl/ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy ###分发证书到所有节点 [root@k8s-master ssl]# cp kube-proxy*.pem /opt/kubernetes/ssl/ [root@k8s-master ssl]# scp kube-proxy*.pem 10.0.0.26:/opt/kubernetes/ssl/ [root@k8s-master ssl]# scp kube-proxy*.pem 10.0.0.27:/opt/kubernetes/ssl/ ###创建kube-proxy配置文件 #master节点做 [root@k8s-master ssl]# kubectl config set-cluster kubernetes --certificate-authority=/opt/kubernetes/ssl/ca.pem --embed-certs=true --server=https://10.0.0.25:6443 --kubeconfig=kube-proxy.kubeconfig [root@k8s-master ssl]# kubectl config set-credentials kube-proxy --client-certificate=/opt/kubernetes/ssl/kube-proxy.pem --client-key=/opt/kubernetes/ssl/kube-proxy-key.pem --embed-certs=true --kubeconfig=kube-proxy.kubeconfig [root@k8s-master ssl]# kubectl config set-context default --cluster=kubernetes --user=kube-proxy --kubeconfig=kube-proxy.kubeconfig [root@k8s-master ssl]# kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig [root@k8s-master ssl]# cp kube-proxy.kubeconfig /opt/kubernetes/cfg/ [root@k8s-master ssl]# scp kube-proxy.kubeconfig 10.0.0.26:/opt/kubernetes/cfg/ [root@k8s-master ssl]# scp kube-proxy.kubeconfig 10.0.0.27:/opt/kubernetes/cfg/ ###创建kube-proxy服务配置 #3个节点都创建 [root@k8s-master ssl]# mkdir /var/lib/kube-proxy [root@k8s-node-1 ~]# mkdir /var/lib/kube-proxy [root@k8s-node-2 ~]# mkdir /var/lib/kube-proxy ##master节点 [root@k8s-master ssl]# vim /usr/lib/systemd/system/kube-proxy.service [Unit] Description=Kubernetes Kube-Proxy Server Documentation=https://github.com/GoogleCloudPlatform/kubernetes After=network.target [Service] WorkingDirectory=/var/lib/kube-proxy ExecStart=/opt/kubernetes/bin/kube-proxy --bind-address=10.0.0.25 --hostname-override=10.0.0.25 --kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig --masquerade-all --feature-gates=SupportIPVSProxyMode=true --proxy-mode=ipvs --ipvs-min-sync-period=5s --ipvs-sync-period=5s --ipvs-scheduler=rr --logtostderr=true --v=2 --logtostderr=false --log-dir=/opt/kubernetes/log Restart=on-failure RestartSec=5 LimitNOFILE=65536 [Install] WantedBy=multi-user.target ##node1节点 [root@k8s-node-1 ~]# vim /usr/lib/systemd/system/kube-proxy.service [Unit] Description=Kubernetes Kube-Proxy Server Documentation=https://github.com/GoogleCloudPlatform/kubernetes After=network.target [Service] WorkingDirectory=/var/lib/kube-proxy ExecStart=/opt/kubernetes/bin/kube-proxy --bind-address=10.0.0.26 --hostname-override=10.0.0.26 --kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig --masquerade-all --feature-gates=SupportIPVSProxyMode=true --proxy-mode=ipvs --ipvs-min-sync-period=5s --ipvs-sync-period=5s --ipvs-scheduler=rr --logtostderr=true --v=2 --logtostderr=false --log-dir=/opt/kubernetes/log Restart=on-failure RestartSec=5 LimitNOFILE=65536 [Install] WantedBy=multi-user.target ##node2节点 [root@k8s-node-2 ~]# vim /usr/lib/systemd/system/kube-proxy.service [Unit] Description=Kubernetes Kube-Proxy Server Documentation=https://github.com/GoogleCloudPlatform/kubernetes After=network.target [Service] WorkingDirectory=/var/lib/kube-proxy ExecStart=/opt/kubernetes/bin/kube-proxy --bind-address=10.0.0.27 --hostname-override=10.0.0.27 --kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig --masquerade-all --feature-gates=SupportIPVSProxyMode=true --proxy-mode=ipvs --ipvs-min-sync-period=5s --ipvs-sync-period=5s --ipvs-scheduler=rr --logtostderr=true --v=2 --logtostderr=false --log-dir=/opt/kubernetes/log Restart=on-failure RestartSec=5 LimitNOFILE=65536 [Install] WantedBy=multi-user.target ###启动Kubernetes Proxy #3个节点都做 systemctl daemon-reload systemctl enable kube-proxy systemctl start kube-proxy systemctl status kube-proxy 检查LVS状态 ipvsadm -L -n kubectl get node
五:Flannel网络部署
1. 部署Flanne
###为Flannel生成证书 [root@k8s-master ~]# cd /usr/local/src/ssl/ [root@k8s-master ~]# vim flanneld-csr.json { "CN": "flanneld", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "BeiJing", "L": "BeiJing", "O": "k8s", "OU": "System" } ] } ###生成证书 [root@k8s-master ~]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem -ca-key=/opt/kubernetes/ssl/ca-key.pem -config=/opt/kubernetes/ssl/ca-config.json -profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld ###分发证书 [root@k8s-master ssl]# cp flanneld*.pem /opt/kubernetes/ssl/ [root@k8s-master ssl]# scp flanneld*.pem 10.0.0.26:/opt/kubernetes/ssl/ [root@k8s-master ssl]# scp flanneld*.pem 10.0.0.27:/opt/kubernetes/ssl/ ###下载Flannel软件包 [root@k8s-master ~]# cd /usr/local/src wget https://github.com/coreos/flannel/releases/download/v0.10.0/flannel-v0.10.0-linux-amd64.tar.gz [root@k8s-master src]# tar zxf flannel-v0.10.0-linux-amd64.tar.gz [root@k8s-master src]# cp flanneld mk-docker-opts.sh /opt/kubernetes/bin/ [root@k8s-master src]# scp flanneld mk-docker-opts.sh 10.0.0.26:/opt/kubernetes/bin/ [root@k8s-master src]# scp flanneld mk-docker-opts.sh 10.0.0.27:/opt/kubernetes/bin/ ###复制对应脚本到/opt/kubernetes/bin目录 [root@k8s-master ~]# cd /usr/local/src/kubernetes/cluster/centos/node/bin/ [root@k8s-master bin]# cp remove-docker0.sh /opt/kubernetes/bin/ [root@k8s-master bin]# scp remove-docker0.sh 10.0.0.26:/opt/kubernetes/bin/ [root@k8s-master bin]# scp remove-docker0.sh 10.0.0.27:/opt/kubernetes/bin/ ###配置Flannel [root@k8s-master ~]# vim /opt/kubernetes/cfg/flannel FLANNEL_ETCD="-etcd-endpoints=https://10.0.0.25:2379,https://10.0.0.26:2379,https://10.0.0.27:2379" FLANNEL_ETCD_KEY="-etcd-prefix=/kubernetes/network" FLANNEL_ETCD_CAFILE="--etcd-cafile=/opt/kubernetes/ssl/ca.pem" FLANNEL_ETCD_CERTFILE="--etcd-certfile=/opt/kubernetes/ssl/flanneld.pem" FLANNEL_ETCD_KEYFILE="--etcd-keyfile=/opt/kubernetes/ssl/flanneld-key.pem" ##复制配置到其它节点上 [root@k8s-master ~]# scp /opt/kubernetes/cfg/flannel 10.0.0.26:/opt/kubernetes/cfg/ [root@k8s-master ~]# scp /opt/kubernetes/cfg/flannel 10.0.0.27:/opt/kubernetes/cfg/ ###设置Flannel系统服务 [root@k8s-master ~]# vim /usr/lib/systemd/system/flannel.service [Unit] Description=Flanneld overlay address etcd agent After=network.target Before=docker.service [Service] EnvironmentFile=-/opt/kubernetes/cfg/flannel ExecStartPre=/opt/kubernetes/bin/remove-docker0.sh ExecStart=/opt/kubernetes/bin/flanneld ${FLANNEL_ETCD} ${FLANNEL_ETCD_KEY} ${FLANNEL_ETCD_CAFILE} ${FLANNEL_ETCD_CERTFILE} ${FLANNEL_ETCD_KEYFILE} ExecStartPost=/opt/kubernetes/bin/mk-docker-opts.sh -d /run/flannel/docker Type=notify [Install] WantedBy=multi-user.target RequiredBy=docker.service ##复制系统服务脚本到其它节点上 [root@k8s-master ~]# scp /usr/lib/systemd/system/flannel.service 10.0.0.26:/usr/lib/systemd/system/ [root@k8s-master ~]# scp /usr/lib/systemd/system/flannel.service 10.0.0.27:/usr/lib/systemd/system/ ###下载Flannel CNI集成插件 [root@k8s-master ~]# mkdir /opt/kubernetes/bin/cni #3个节点都要创建 [root@k8s-master ~]# cd /usr/local/src/ [root@k8s-master src]# wget https://github.com/containernetworking/plugins/releases/download/v0.7.1/cni-plugins-amd64-v0.7.1.tgz [root@k8s-master src]# tar zxf cni-plugins-amd64-v0.7.1.tgz -C /opt/kubernetes/bin/cni [root@k8s-master src]# scp -r /opt/kubernetes/bin/cni/* 10.0.0.26:/opt/kubernetes/bin/cni/ [root@k8s-master src]# scp -r /opt/kubernetes/bin/cni/* 10.0.0.27:/opt/kubernetes/bin/cni/ ###创建Etcd的key [root@k8s-master ~]# /opt/kubernetes/bin/etcdctl --ca-file /opt/kubernetes/ssl/ca.pem --cert-file /opt/kubernetes/ssl/flanneld.pem --key-file /opt/kubernetes/ssl/flanneld-key.pem --no-sync -C https://10.0.0.25:2379,https://10.0.0.26:2379,https://10.0.0.27:2379 mk /kubernetes/network/config '{ "Network": "10.2.0.0/16", "Backend": { "Type": "vxlan", "VNI": 1 }}' >/dev/null 2>&1 ###启动flannel #3个节点都做 chmod +x /opt/kubernetes/bin/* systemctl daemon-reload systemctl enable flannel systemctl start flannel systemctl status flannel ###配置Docker使用Flannel [root@k8s-master ~]# vim /usr/lib/systemd/system/docker.service #在Unit下面修改After和增加Requires [Unit] After=network-online.target firewalld.service flannel.service Requires=flannel.service #增加EnvironmentFile=-/run/flannel/docker和$DOCKER_OPTS [Service] EnvironmentFile=-/run/flannel/docker ExecStart=/usr/bin/dockerd $DOCKER_OPTS ###将配置复制到另外两个阶段 [root@k8s-master ~]# scp /usr/lib/systemd/system/docker.service 10.0.0.26:/usr/lib/systemd/system/ [root@k8s-master ~]# scp /usr/lib/systemd/system/docker.service 10.0.0.27:/usr/lib/systemd/system/ ###重启Docker #3个节点都做 systemctl daemon-reload systemctl restart docker
2. 创建一个K8S应用
###创建一个测试用的deployment [root@k8s-master ~]# kubectl run net-test --image=alpine --replicas=2 sleep 360000 ###查看获取IP情况 [root@k8s-master ~]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE net-test-5767cb94df-7kqhz 1/1 Running 0 30m 10.2.93.2 10.0.0.27 net-test-5767cb94df-xw6r9 1/1 Running 0 30m 10.2.58.2 10.0.0.25 ###测试联通性 [root@k8s-master ~]# ping 10.2.58.2 -c1
如下图所示说明成功:
###创建nginx的deployment.yaml文件
[root@k8s-master ~]# vim nginx-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: replicas: 3 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.10.3 ports: - containerPort: 80 ###创建deployment [root@k8s-master ~]# kubectl create -f nginx-deployment.yaml deployment.apps "nginx-deployment" created ###查看deployment [root@k8s-master ~]# kubectl get deployment NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE net-test 2 2 2 2 43m nginx-deployment 3 3 3 0 53s [root@k8s-master ~]# kubectl describe deployment nginx-deployment ###查看pod [root@k8s-master ~]# kubectl get pod NAME READY STATUS RESTARTS AGE net-test-5767cb94df-7kqhz 1/1 Running 0 45m net-test-5767cb94df-xw6r9 1/1 Running 0 45m nginx-deployment-75d56bb955-8tzh6 0/1 ContainerCreating 0 2m #ContainerCreating说明正在创建中 nginx-deployment-75d56bb955-bjmg7 0/1 ContainerCreating 0 2m #ContainerCreating说明正在创建中 nginx-deployment-75d56bb955-w4dks 0/1 ContainerCreating 0 2m #ContainerCreating说明正在创建中
[root@k8s-master ~]# kubectl describe pod nginx-deployment-75d56bb955-8tzh6|tail -2 Normal SuccessfulMountVolume 5m kubelet, 10.0.0.27 MountVolume.SetUp succeeded for volume "default-token-rpdp6" Normal Pulling 5m kubelet, 10.0.0.27 pulling image "nginx:1.10.3" #正在pull取nginx的镜像
[root@k8s-master ~]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE net-test-5767cb94df-7kqhz 1/1 Running 0 49m 10.2.93.2 10.0.0.27 net-test-5767cb94df-xw6r9 1/1 Running 0 49m 10.2.58.2 10.0.0.25 nginx-deployment-75d56bb955-8tzh6 0/1 ContainerCreating 0 6m <none> 10.0.0.27 nginx-deployment-75d56bb955-bjmg7 0/1 ContainerCreating 0 6m <none> 10.0.0.26 nginx-deployment-75d56bb955-w4dks 0/1 ContainerCreating 0 6m <none> 10.0.0.25
[root@k8s-master ~]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE net-test-5767cb94df-7kqhz 1/1 Running 0 1h 10.2.93.2 10.0.0.27 net-test-5767cb94df-xw6r9 1/1 Running 0 1h 10.2.58.2 10.0.0.25 nginx-deployment-75d56bb955-8tzh6 1/1 Running 0 21m 10.2.93.3 10.0.0.27 nginx-deployment-75d56bb955-bjmg7 1/1 Running 0 21m 10.2.83.2 10.0.0.26 nginx-deployment-75d56bb955-w4dks 1/1 Running 0 21m 10.2.58.3 10.0.0.25 ###测试连通性 [root@k8s-master ~]# curl -I 10.2.93.3 -s|awk 'NR==1{print$2}' 200 [root@k8s-master ~]# curl -I 10.2.83.2 -s|awk 'NR==1{print$2}' 200 [root@k8s-master ~]# curl -I 10.2.58.3 -s|awk 'NR==1{print$2}' 200 ###更新deployment #--record是记录日志,方便回滚 [root@k8s-master ~]# kubectl set image deployment/nginx-deployment nginx=nginx:1.12.2 --record deployment.apps "nginx-deployment" image updated ###查看更新后的deployment [root@k8s-master ~]# kubectl get deployment -o wide NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR net-test 2 2 2 2 2h net-test alpine run=net-test nginx-deployment 3 3 3 3 1h nginx nginx:1.12.2 app=nginx ###查看更新历史 [root@k8s-master ~]# kubectl rollout history deployment/nginx-deployment --revision=1 deployments "nginx-deployment" with revision #1 Pod Template: Labels: app=nginx pod-template-hash=3181266511 Containers: nginx: Image: nginx:1.10.3 Port: 80/TCP Host Port: 0/TCP Environment: <none> Mounts: <none> Volumes: <none> #查看具体某一个版本的更新历史 [root@k8s-master ~]# kubectl rollout history deployment/nginx-deployment --revision=1 deployments "nginx-deployment" with revision #1 Pod Template: Labels: app=nginx pod-template-hash=3181266511 Containers: nginx: Image: nginx:1.10.3 Port: 80/TCP Host Port: 0/TCP Environment: <none> Mounts: <none> Volumes: <none> ###快速回滚到上一个版本 [root@k8s-master ~]# kubectl rollout undo deployment/nginx-deployment deployment.apps "nginx-deployment" [root@k8s-master ~]# vim nginx-service.yaml kind: Service apiVersion: v1 metadata: name: nginx-service spec: selector: app: nginx ports: - protocol: TCP port: 80 targetPort: 80
###创建nginx的service.yaml文件 [root@k8s-master ~]# kubectl create -f nginx-service.yaml service "nginx-service" created
###查看service [root@k8s-master ~]# kubectl get service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.1.0.1 <none> 443/TCP 18h nginx-service ClusterIP 10.1.172.14 <none> 80/TCP 12s ###测试连通性 [root@k8s-master ~]# curl --head http://10.1.172.14 HTTP/1.1 200 OK Server: nginx/1.10.3 Date: Mon, 04 Jun 2018 04:58:47 GMT Content-Type: text/html Content-Length: 612 Last-Modified: Tue, 31 Jan 2017 15:01:11 GMT Connection: keep-alive ETag: "5890a6b7-264" Accept-Ranges: bytes
###查看虚IP [root@k8s-master ~]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 10.1.0.1:443 rr persistent 10800 -> 10.0.0.25:6443 Masq 1 0 0 TCP 10.1.172.14:80 rr -> 10.2.58.5:80 Masq 1 0 0 -> 10.2.83.4:80 Masq 1 0 0 -> 10.2.93.5:80 Masq 1 0 1 ###自动扩容到5个副本节点 [root@k8s-master ~]# kubectl scale deployment nginx-deployment --replicas 5 deployment.extensions "nginx-deployment" scaled [root@k8s-master ~]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE net-test-5767cb94df-7kqhz 1/1 Running 0 2h 10.2.93.2 10.0.0.27 net-test-5767cb94df-xw6r9 1/1 Running 0 2h 10.2.58.2 10.0.0.25 nginx-deployment-75d56bb955-6dfxr 1/1 Running 0 17s 10.2.83.5 10.0.0.26 nginx-deployment-75d56bb955-9jmch 1/1 Running 0 17s 10.2.93.6 10.0.0.27 nginx-deployment-75d56bb955-gssxl 1/1 Running 0 7m 10.2.58.5 10.0.0.25 nginx-deployment-75d56bb955-hkqdc 1/1 Running 0 7m 10.2.93.5 10.0.0.27 nginx-deployment-75d56bb955-s66c4 1/1 Running 0 7m 10.2.83.4 10.0.0.26 [root@k8s-master ~]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 10.1.0.1:443 rr persistent 10800 -> 10.0.0.25:6443 Masq 1 0 0 TCP 10.1.172.14:80 rr -> 10.2.58.5:80 Masq 1 0 0 -> 10.2.83.4:80 Masq 1 0 0 -> 10.2.83.5:80 Masq 1 0 0 -> 10.2.93.5:80 Masq 1 0 0 -> 10.2.93.6:80 Masq 1 0 0