zoukankan      html  css  js  c++  java
  • linux运维、架构之路-Kubernetes离线、二进制部署集群

    一、Kubernetes对应Docker的版本支持列表

    Kubernetes 1.9  <--Docker 1.11.2 to 1.13.1 and 17.03.x
    Kubernetes 1.8  <--Docker 1.11.2 to 1.13.1 and 17.03.x
    Kubernetes 1.7  <--Docker 1.10.3,  1.11.2,  1.12.6
    Kubernetes 1.6  <--Docker 1.10.3,  1.11.2,  1.12.6
    Kubernetes 1.5  <--Docker 1.10.3,  1.11.2,  1.12.3

    版本对应地址:

    https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.5.md#external-dependency-version-information
    https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.6.md#external-dependency-version-information
    https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.7.md#external-dependency-version-information
    https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.8.md#external-dependencies
    https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.9.md#external-dependencies
    

    二、Kubernetes集群部署架构图

     

    三、Kubernetes集群环境规划

    1、系统环境

    [root@k8s-master ~]# cat /etc/redhat-release 
    CentOS Linux release 7.2.1511 (Core) 
    [root@k8s-master ~]# uname -r
    3.10.0-327.el7.x86_64
    [root@k8s-master ~]# systemctl status firewalld.service 
    ● firewalld.service - firewalld - dynamic firewall daemon
       Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
       Active: inactive (dead)
    [root@k8s-master ~]# getenforce 
    Disabled

    2、服务器规划

    节点及功能

    主机名

    IP

    Master、etcd、registry

    K8s-master

    10.0.0.211

    Node1

    K8s-node-1

    10.0.0.212

    Node2

    K8s-node-2

    10.0.0.213

    3、统一hosts解析

    echo '
    10.0.0.211    k8s-master
    10.0.0.211    etcd
    10.0.0.211    registry
    10.0.0.212    k8s-node-1
    10.0.0.213    k8s-node-2' >> /etc/hosts

    4、下载Kubernetes(简称K8S)二进制文件

    https://github.com/kubernetes/kubernetes/releases  

    此处用的k8s版本为v:1.8.3

    https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.8.md#v183

     Docker版本为v17.03-ce

    etcd下载版本为v3.1.18

    https://github.com/coreos/etcd/releases/

    四、Kubernetes集群离线部署

    1、部署docker

    #上传docker离线包#
    yum install docker-ce-selinux-17.03.2.ce-1.el7.centos.noarch.rpm docker-ce-17.03.2.ce-1.el7.centos.x86_64.rpm -y
    #移除旧版本#
    yum remove docker 
                      docker-common 
                      docker-selinux 
                      docker-engine

    设置docker服务开机启动

    systemctl enable docker.service
    systemctl start docker.service 

    修改docker的镜像源为国内的daocloud

    curl -sSL https://get.daocloud.io/daotools/set_mirror.sh | sh -s http://a58c8480.m.daocloud.io

    2、部署etcd3

    CentOS7二进制安装组件注意事项:

    ①复制对应的二进制文件到/usr/bin目录下

    ②创建systemd service启动服务文件

    ③创建service 中对应的配置参数文件

    ④将该应用加入到开机自启

    #上传etcd-v3.1.18-linux-amd64.tar.gz#
    tar xf etcd-v3.1.18-linux-amd64.tar.gz
    cd etcd-v3.1.18-linux-amd64/
    mv etcd etcdctl /usr/bin/

    ①创建/etc/etcd/etcd.conf配置文件

    mkdir /etc/etcd/ -p
    mkdir /var/lib/etcd -p
    
    [root@k8s-master ~]# cat /etc/etcd/etcd.conf 
    ETCD_NAME=ETCD Server
    ETCD_DATA_DIR="/var/lib/etcd/"
    ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
    ETCD_ADVERTISE_CLIENT_URLS="http://10.0.0.211:2379"

    ②创建etcd.service启动服务文件

    [root@k8s-master ~]# cat /etc/systemd/system/etcd.service
    [Unit]
    Description=etcd.service
    [Service]
    Type=notify
    TimeoutStartSec=0
    Restart=always
    WorkingDirectory=/var/lib/etcd
    EnvironmentFile=-/etc/etcd/etcd.conf
    ExecStart=/usr/bin/etcd 
    [Install]
    WantedBy=multi-user.target
    说明:其中WorkingDirectory为etcd数据库目录,前面我们已经提前创建好了

    ③设置etcd开机启动

    systemctl daemon-reload
    systemctl enable etcd.service
    systemctl start etcd.service

    ④检验etcd是否安装成功

    [root@k8s-master ~]# etcdctl cluster-health
    member 8e9e05c52164694d is healthy: got healthy result from http://10.0.0.211:2379
    cluster is healthy

    3、部署k8s-Master节点组件

    #上传kubernetes-server-linux-amd64.tar.gz#
    tar xf kubernetes-server-linux-amd64.tar.gz
    mkdir -p /app/kubernetes/{bin,cfg}
    mv kubernetes/server/bin/{kube-apiserver,kube-scheduler,kube-controller-manager,kubectl} /app/kubernetes/bin

    ①配置kube-apiserver.service服务

    配置文件:vim /app/kubernetes/cfg/kube-apiserver

    # 启用日志标准错误
    KUBE_LOGTOSTDERR="--logtostderr=true"
    # 日志级别
    KUBE_LOG_LEVEL="--v=4"
    # Etcd服务地址
    KUBE_ETCD_SERVERS="--etcd-servers=http://10.0.0.211:2379"
    # API服务监听地址
    KUBE_API_ADDRESS="--insecure-bind-address=0.0.0.0"
    # API服务监听端口
    KUBE_API_PORT="--insecure-port=8080"
    # 对集群中成员提供API服务地址
    KUBE_ADVERTISE_ADDR="--advertise-address=10.0.0.211"
    # 允许容器请求特权模式,默认false
    KUBE_ALLOW_PRIV="--allow-privileged=false"
    # 集群分配的IP范围
    KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.10.10.0/24"

    启动服务systemd:vim /lib/systemd/system/kube-apiserver.service

    [Unit]
    Description=Kubernetes API Server
    Documentation=https://github.com/kubernetes/kubernetes
    [Service]
    EnvironmentFile=-/app/kubernetes/cfg/kube-apiserver
    #ExecStart=/app/kubernetes/bin/kube-apiserver ${KUBE_APISERVER_OPTS}
    ExecStart=/app/kubernetes/bin/kube-apiserver 
    ${KUBE_LOGTOSTDERR} 
    ${KUBE_LOG_LEVEL} 
    ${KUBE_ETCD_SERVERS} 
    ${KUBE_API_ADDRESS} 
    ${KUBE_API_PORT} 
    ${KUBE_ADVERTISE_ADDR} 
    ${KUBE_ALLOW_PRIV} 
    ${KUBE_SERVICE_ADDRESSES}
    Restart=on-failure
    [Install]
    WantedBy=multi-user.target

    启动服务,并设置开机自启

    systemctl daemon-reload 
    systemctl enable kube-apiserver.service
    systemctl start kube-apiserver.service

    ②配置kube-scheduler服务

    配置文件:

    vim /app/kubernetes/cfg/kube-scheduler
    KUBE_LOGTOSTDERR="--logtostderr=true"
    KUBE_LOG_LEVEL="--v=4"
    KUBE_MASTER="--master=10.0.0.211:8080"
    KUBE_LEADER_ELECT="--leader-elect"

    systemd服务文件:

    vim /lib/systemd/system/kube-scheduler.service
    [Unit]
    Description=Kubernetes Scheduler
    Documentation=https://github.com/kubernetes/kubernetes
    [Service]
    EnvironmentFile=-/app/kubernetes/cfg/kube-scheduler
    ExecStart=/app/kubernetes/bin/kube-scheduler 
    ${KUBE_LOGTOSTDERR} 
    ${KUBE_LOG_LEVEL} 
    ${KUBE_MASTER} 
    ${KUBE_LEADER_ELECT}
    Restart=on-failure
    [Install]
    WantedBy=multi-user.target

    启动服务并设置开机启动

    systemctl daemon-reload 
    systemctl enable kube-scheduler.service
    systemctl start kube-scheduler.service

    ③配置kube-controller-manger

    配置文件:

    cat /app/kubernetes/cfg/kube-controller-manager
    KUBE_LOGTOSTDERR="--logtostderr=true"
    KUBE_LOG_LEVEL="--v=4"
    KUBE_MASTER="--master=10.0.0.211:8080"

    systemd服务文件

    cat /lib/systemd/system/kube-controller-manager.service
    [Unit]
    Description=Kubernetes Controller Manager
    Documentation=https://github.com/kubernetes/kubernetes
    [Service]
    EnvironmentFile=-/app/kubernetes/cfg/kube-controller-manager
    ExecStart=/app/kubernetes/bin/kube-controller-manager 
    ${KUBE_LOGTOSTDERR} 
    ${KUBE_LOG_LEVEL} 
    ${KUBE_MASTER} 
    ${KUBE_LEADER_ELECT}
    Restart=on-failure
    [Install]
    WantedBy=multi-user.target

    启动服务并设置开机自启

    systemctl daemon-reload 
    systemctl enable kube-controller-manager.service
    systemctl start kube-controller-manager.service 

    至此Master节点组件就全部启动了,需要注意的是服务启动顺序有依赖,先启动etcd,再启动apiserver,其他组件无顺序要求

    查看Master节点组件运行进程

    [root@k8s-master ~]# ps -ef|grep kube
    root       6217      1  2 15:54 ?        00:00:19 /app/kubernetes/bin/kube-apiserver --logtostderr=true --v=4 --etcd-servers=http://10.0.0.211:2379 --insecure-bind-address=0.0.0.0 --insecure-port=8080 --advertise-address=10.0.0.211 --allow-privileged=false --service-cluster-ip-range=10.10.10.0/24
    root       6304      1  1 16:01 ?        00:00:04 /app/kubernetes/bin/kube-scheduler --logtostderr=true --v=4 --master=10.0.0.211:8080 --leader-elect
    root       6369      1  2 16:05 ?        00:00:02 /app/kubernetes/bin/kube-controller-manager --logtostderr=true --v=4 --master=10.0.0.211:8080
    root       6376   1926  0 16:07 pts/0    00:00:00 grep --color=auto kube

    验证Master节点功能

    [root@k8s-master ~]# kubectl get componentstatuses
    NAME                 STATUS    MESSAGE              ERROR
    etcd-0               Healthy   {"health": "true"}   
    controller-manager   Healthy   ok                   
    scheduler            Healthy   ok

    如果启动失败,查看日志

    [root@k8s-master ~]# journalctl -u kube-apiserver.service 
    -- Logs begin at 一 2018-09-03 14:57:42 CST, end at 一 2018-09-03 16:09:06 CST. --
    9月 03 15:54:00 k8s-master systemd[1]: Started Kubernetes API Server.
    9月 03 15:54:00 k8s-master systemd[1]: Starting Kubernetes API Server...
    9月 03 15:54:00 k8s-master kube-apiserver[6217]: I0903 15:54:00.924400    6217 flags.go:52] FLAG: --addre
    9月 03 15:54:00 k8s-master kube-apiserver[6217]: I0903 15:54:00.924847    6217 flags.go:52] FLAG: --admis
    9月 03 15:54:00 k8s-master kube-apiserver[6217]: I0903 15:54:00.924866    6217 flags.go:52] FLAG: --admis
    9月 03 15:54:00 k8s-master kube-apiserver[6217]: I0903 15:54:00.924873    6217 flags.go:52] FLAG: --adver
    9月 03 15:54:00 k8s-master kube-apiserver[6217]: I0903 15:54:00.924878    6217 flags.go:52] FLAG: --allow
    9月 03 15:54:00 k8s-master kube-apiserver[6217]: I0903 15:54:00.924884    6217 flags.go:52] FLAG: --alsol
    9月 03 15:54:00 k8s-master kube-apiserver[6217]: I0903 15:54:00.924904    6217 flags.go:52] FLAG: --anony
    9月 03 15:54:00 k8s-master kube-apiserver[6217]: I0903 15:54:00.924908    6217 flags.go:52] FLAG: --apise
    9月 03 15:54:00 k8s-master kube-apiserver[6217]: I0903 15:54:00.924914    6217 flags.go:52] FLAG: --audit

    配置环境变量

    echo "export PATH=$PATH:/app/kubernetes/bin" >> /etc/profile
    source /etc/profile

    4、部署K8s-Node节点组件

    #上传kubernetes-node-linux-amd64.tar.gz#
    tar xf kubernetes-node-linux-amd64.tar.gz
    mkdir -p /app/kubernetes/{bin,cfg}
    mv kubernetes/node/bin/{kubelet,kube-proxy} /app/kubernetes/bin/

    ①配置kubelet服务

    创建kubeconfig配置文件:kubeconfig文件用于kubelet连接master apiserver

    cat /app/kubernetes/cfg/kubelet.kubeconfig
    apiVersion: v1
    kind: Config
    clusters:
      - cluster:
          server: http://10.0.0.211:8080
        name: local
    contexts:
      - context:
          cluster: local
        name: local
    current-context: local

    创建kubelet配置文件

    cat /app/kubernetes/cfg/kubelet
    # 启用日志标准错误
    KUBE_LOGTOSTDERR="--logtostderr=true"
    # 日志级别
    KUBE_LOG_LEVEL="--v=4"
    # Kubelet服务IP地址
    NODE_ADDRESS="--address=10.0.0.212"
    # Kubelet服务端口
    NODE_PORT="--port=10250"
    # 自定义节点名称
    NODE_HOSTNAME="--hostname-override=10.0.0.212"
    # kubeconfig路径,指定连接API服务器
    KUBELET_KUBECONFIG="--kubeconfig=/app/kubernetes/cfg/kubelet.kubeconfig"
    # 允许容器请求特权模式,默认false
    KUBE_ALLOW_PRIV="--allow-privileged=false"
    # DNS信息
    KUBELET_DNS_IP="--cluster-dns=10.10.10.2"
    KUBELET_DNS_DOMAIN="--cluster-domain=cluster.local"
    # 禁用使用Swap
    KUBELET_SWAP="--fail-swap-on=false"

    systemd服务文件

    cat /lib/systemd/system/kubelet.service
    [Unit]
    Description=Kubernetes Kubelet
    After=docker.service
    Requires=docker.service
    [Service]
    EnvironmentFile=-/app/kubernetes/cfg/kubelet
    ExecStart=/app/kubernetes/bin/kubelet 
    ${KUBE_LOGTOSTDERR} 
    ${KUBE_LOG_LEVEL} 
    ${NODE_ADDRESS} 
    ${NODE_PORT} 
    ${NODE_HOSTNAME} 
    ${KUBELET_KUBECONFIG} 
    ${KUBE_ALLOW_PRIV} 
    ${KUBELET_DNS_IP} 
    ${KUBELET_DNS_DOMAIN} 
    ${KUBELET_SWAP}
    Restart=on-failure
    KillMode=process
    [Install]
    WantedBy=multi-user.target

    启动服务并设置开机自启

    systemctl daemon-reload 
    systemctl enable kubelet.service 
    systemctl start kubelet.service 

    ②配置kube-proxy服务

    配置文件

    mkdir /app/kubernetes/cfg/ -p
    
    cat /app/kubernetes/cfg/kube-proxy
    # 启用日志标准错误
    KUBE_LOGTOSTDERR="--logtostderr=true"
    # 日志级别
    KUBE_LOG_LEVEL="--v=4"
    # 自定义节点名称
    NODE_HOSTNAME="--hostname-override=10.0.0.212"
    # API服务地址
    KUBE_MASTER="--master=http://10.0.0.211:8080"

    systemd服务文件

    cat /lib/systemd/system/kube-proxy.service
    [Unit]
    Description=Kubernetes Proxy
    After=network.target
    [Service]
    EnvironmentFile=-/app/kubernetes/cfg/kube-proxy
    ExecStart=/app/kubernetes/bin/kube-proxy 
    ${KUBE_LOGTOSTDERR} 
    ${KUBE_LOG_LEVEL} 
    ${NODE_HOSTNAME} 
    ${KUBE_MASTER}
    Restart=on-failure
    [Install]
    WantedBy=multi-user.target

    启动服务并设置开机自启

    systemctl daemon-reload
    systemctl enable kube-proxy
    systemctl restart kube-proxy

    ③部署Flannel网络 

    下载安装

    wget https://github.com/coreos/flannel/releases/download/v0.9.1/flannel-v0.9.1-linux-amd64.tar.gz
    tar xf flannel-v0.9.1-linux-amd64.tar.gz
    cp flanneld mk-docker-opts.sh /usr/bin/
    mkdir -p /app/flannel/conf/

    配置内核转发

    cat <<EOF > /etc/sysctl.d/k8s.conf
    net.bridge.bridge-nf-call-ip6tables = 1
    net.bridge.bridge-nf-call-iptables = 1
    vm.swappiness=0
    EOF
    
    sysctl --system

    systemd服务文件

    cat /usr/lib/systemd/system/flanneld.service
    
    [Unit]
    Description=Flanneld overlay address etcd agent
    After=network.target
    After=network-online.target
    Wants=network-online.target
    After=etcd.service
    Before=docker.service
    
    [Service]
    Type=notify
    EnvironmentFile=/app/flannel/conf/flanneld
    EnvironmentFile=-/etc/sysconfig/docker-network
    ExecStart=/usr/bin/flanneld-start $FLANNEL_OPTIONS
    ExecStartPost=/usr/libexec/flannel/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
    Restart=on-failure
    
    [Install]
    WantedBy=multi-user.target
    WantedBy=docker.service

    配置文件:master、node都需要执行

    cat /app/flannel/conf/flanneld
    
    # Flanneld configuration options  
    # etcd url location.  Point this to the server where etcd runs
    FLANNEL_ETCD_ENDPOINTS="http://10.0.0.211:2379"
    # etcd config key.  This is the configuration key that flannel queries
    # For address range assignment
    FLANNEL_ETCD_KEY="/k8s/network"
    # Any additional options that you want to pass
    #FLANNEL_OPTIONS=""
    FLANNEL_OPTIONS="--logtostderr=false --log_dir=/var/log/k8s/flannel/ --etcd-endpoints=http://10.0.0.211:2379"

    启动服务并设置开机自启

    Master执行:

    etcdctl set /k8s/network/config '{ "Network": "172.16.0.0/16" }'
    systemctl daemon-reload 
    systemctl enable flanneld.service
    systemctl start flanneld.service

    验证服务:

    journalctl  -u flanneld |grep 'Lease acquired'
    9月 03 17:58:02 k8s-node-1 flanneld[9658]: I0903 17:58:02.862074    9658 manager.go:250] Lease acquired: 10.0.11.0/24
    9月 03 18:49:47 k8s-node-1 flanneld[11731]: I0903 18:49:47.882891   11731 manager.go:250] Lease acquire: 10.0.11.0/24

    ④启动Flannel之后,需要依次重启docker、kubernete

    Master节点执行:

    systemctl restart docker.service
    systemctl restart kube-apiserver.service
    systemctl restart kube-controller-manager.service
    systemctl restart kube-scheduler.service

    Node节点执行:

    systemctl restart docker.service
    systemctl restart kubelet.service
    systemctl restart kube-proxy.service

     Master节点查看IP信息

    Node节点查看IP信息

    小结:查看docker0和flannel0的网络设备,确保每个Node上的Docker0和flannel0在同一段内,并且不同节点的网段都被划分在172.16.0.0/16 的不同段内。如Master是172.16.88.0/16,Node1是172.16.65.0/16……

    成功最有效的方法就是向有经验的人学习!
  • 相关阅读:
    2021NUAA暑假集训 Day3 题解
    2021NUAA暑假集训 Day2 题解
    2021NUAA暑期模拟赛部分题解
    CodeForces 1038D Slime
    UVA 11149 Power of Matrix
    UVA 10655 Contemplation! Algebra
    UVA 10689 Yet another Number Sequence
    HDU 4549 M斐波那契数列
    HDU 4990 Reading comprehension
    CodeForces 450B Jzzhu and Sequences
  • 原文地址:https://www.cnblogs.com/yanxinjiang/p/9578352.html
Copyright © 2011-2022 走看看