zoukankan      html  css  js  c++  java
  • 【k8s实战一】Jenkins 部署应用到 Kubernetes

    【k8s实战一】Jenkins 部署应用到 Kubernetes

    01 本文主旨

    目标是演示整个Jenkins从源码构建镜像到部署镜像到Kubernetes集群过程。

    为了简化流程与容易重现文中效果,做出如下操作:

    • 使用 VBox 创建两台虚拟机
    • 使用 Minikube 初始化简单的k8s集群
    • 使用 GitHub 作为代码仓库,本文内容所有配置文件均会上传至此仓库 https://github.com/hellxz/cicd-demo.git

    02 CI/CD流程

    根据我画的这张图,Jenkins任务整体的流程分为五个步骤:

    • 获取源码
    • 构建制品
    • 构建镜像并推送
    • 发起部署请求
    • k8s调度pod,拉取镜像,部署完成

    03 准备虚拟机

    准备两个虚拟机,均安装好docker,参考Ubuntu 16.04及以上 安装/卸载 Docker-CE

    VM用途 系统版本 IP
    Jenkins Ubuntu 18.04.2 LTS 192.168.87.128
    k8s master Ubuntu 18.04.2 LTS 192.168.87.129

    设置Jenkins机器免密登录k8s master主机

    ssh-keygen -t rsa
    ssh-copy-id hellxz@192.168.87.129
    

    验证免密效果

    ssh hellxz@192.168.87.129
    exit
    

    04 部署registry

    部署registry之前,可以先将私有仓库配置到 /etc/docker/daemon.json,两台虚拟机均需要配置

    sudo mkdir -p /etc/docker
    sudo vim /etc/docker/daemon.json
    

    内容如下:

    {
        "insecure-registries": ["192.168.87.129:5000"]
    }
    

    接着在 192.168.87.129 虚拟机上部署registry,生产环境推荐使用Harbor

    docker run -d --name registry -p 5000:5000 registry 
    

    05 配置maven

    在jenkins虚拟机安装maven

    wget https://mirrors.aliyun.com/apache/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz
    tar zxvf apache-maven-3.6.3-bin.tar.gz
    mv apache-maven-3.6.3 maven-3.6.3
    

    修改 /etc/profile,添加 M2_HOME 环境变量映射到 maven 目录,重写 PATH 添加 maven 的 bin 目录,如图31、32行

    06 Tomcat部署Jenkins

    在jenkins虚拟机 192.168.87.128上执行命令下载 tomcat 与 jenkins.war

    #下载tomcat
    wget https://mirrors.tuna.tsinghua.edu.cn/apache/tomcat/tomcat-9/v9.0.41/bin/apache-tomcat-9.0.41.tar.gz
    #解压并重命名
    tar -zxf apache-tomcat-9.0.41.tar.gz
    mv apache-tomcat-9.0.41 /home/hellxz/jenkins
    #进入jenkins安装目录
    cd ~/jenkins/webapps
    #删除webapps下示例项目
    rm -rf *
    #下载最新版jenkins.war
    wget https://mirrors.ustc.edu.cn/jenkins/war/latest/jenkins.war
    #安装openjdk、git
    sudo apt-update
    sudo apt-get install -y default-jdk default-jdk-headless git
    

    设置Jenkins家目录,编辑 /etc/profile,添加 JENKINS_HOME 环境变量

    刷新配置

    source /etc/profile
    

    修改tomcat下 conf/context.xml,加大静态资源缓存,提前解决启动 Jenkins 时的报错

    <Resources cachingAllowed="true" cacheMaxSize="9999999" />
    

    启动Jenkins

    cd ~/jenkins/bin
    ./startup.sh
    

    访问 http://192.168.87.129:8080,安装推荐插件,安装加速可参考 Jenkins安装插件提速

    07 部署Kubernetes集群

    部署一个可以通过 kubectl 命令部署应用的k8s环境即可,部署k8s环境不是本文重点,以下为参考

    使用kubeadm部署Kubernetes集群,点击展开查看 参考我之前的博客吧,差异应该不大
    【k8s学习笔记】使用 kubeadm 部署 v1.18.5 版本 Kubernetes集群
    使用Minikube部署单节点测试k8s环境,点击展开查看
    
    #添加阿里apt kubernetes源
    sudo apt-get update && sudo apt-get install -y apt-transport-https
    curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo apt-key add - 
    sudo tee /etc/apt/sources.list.d/kubernetes.list <<-'EOF'
    deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
    EOF
    sudo apt-get update
    #安装kubectl
    sudo apt-get install -y kubectl
    #下载minikube二进制文件
    curl -Lo minikube https://kubernetes.oss-cn-hangzhou.aliyuncs.com/minikube/releases/v1.16.0/minikube-linux-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/
    #部署k8s环境,默认使用docker驱动。读者也可使用vbox
    minikube start -–insecure-registry=192.168.87.129:5000
    

    本文使用Minikube做测试,坑比较多,想趟坑可以先参考下 https://www.kubernetes.org.cn/6388.html

    关于Minikube的吐槽

    写文时,用的Win10创建Linux无桌面虚拟机,在虚拟机基础上装的minikube,minikube默认使用Docker在虚拟机中又创建了个Docker容器,里边才是k8s环境,所以想在虚拟机外访问套了两层的内网应用服务,可能还需要在虚拟机中加Nginx,通过它转发请求到虚拟机中的内网地址……虽然疯狂套娃挺坑的,但为了演示下效果还是可以的。 PS:在Linux/MacOS环境直接装Minikube,是可以通过浏览器直接看的。由于未直接在Win环境使用Docker Desktop,所以未确定Win下是否可用。

    08 准备Demo

    Demo分五部分:测试代码、Jenkins构建脚本、docker镜像构建脚本、k8s部署配置yaml、部署脚本,均处在同一仓库中

    仓库地址:https://github.com/hellxz/cicd-demo.git

    目录结构:

    demo

    使用start.spring.io生成的只添加spring-boot-starter-web依赖,在启动类上加了个 /test 的接口

    Dockerfile

    FROM openjdk:8-jdk
    #设置工作目录
    WORKDIR /root
    #复制制品jar到/root/app.jar位置
    ADD target/*.jar app.jar
    #JVM参数,给后期调优使用
    ENV JVM_OPTS=""
    #启动服务
    CMD java ${JVM_OPTS} 
        -Djava.security.egd=file:/dev/./urandom 
        -jar app.jar
    

    shell

    artifact2image.sh 制品转镜像Shell脚本,本文为了简单未安装Jenkins关于docker的插件

    #!/bin/bash
    # -- 构建镜像并推送私有仓库 --
    set -eu #如有报错或取不到变量情况停止执行
    
    #声名常量
    IMG_REGISTRY="192.168.87.129:5000" #镜像仓库
    IMG_NAME="cicd-demo"
    IMG_TAG=`date "+%Y%M%d_%H%M"`      #镜像标签,如 20201223_1351
    IMG_FULL_NAME="${IMG_REGISTRY}/${IMG_NAME}:${IMG_TAG}" #镜像上传与拉取的名称
    
    #构建镜像
    docker build -t ${IMG_FULL_NAME} .
    
    #推送镜像
    docker push ${IMG_FULL_NAME}
    
    #删除本地镜像
    docker rmi ${IMG_FULL_NAME}
    
    #修改deploy.yaml的镜像标签
    sed -i "s#{{IMAGE_NAME}}#${IMG_FULL_NAME}#g" deploy.yaml
    

    deploy2k8s.sh 部署镜像到k8s环境脚本

    #!/bin/bash
    # -- 部署image到k8s --
    # !注意:需要提前做ssh免密登录
    set -eu
    
    #定义常量
    PROJECT_NAME="cicd-demo"
    UPLOAD_DIR="/home/hellxz/apps/${PROJECT_NAME}"
    FILE_NAME="${UPLOAD_DIR}/deploy.yaml"
    SSH_USER="hellxz"
    SSH_IP="192.168.87.129"
    
    #首先删除待上传目录的同名文件
    ssh ${SSH_USER}@${SSH_IP} "rm -rf ${FILE_NAME}"
    
    #确保部署文件目录存在
    ssh ${SSH_USER}@${SSH_IP} "mkdir -p ${UPLOAD_DIR}"
    
    #远程复制部署文件
    scp -r deploy.yaml ${SSH_USER}@${SSH_IP}:${FILE_NAME}
    
    #远程执行部署命令
    ssh ${SSH_USER}@${SSH_IP} "kubectl apply -f ${FILE_NAME}"
    

    Jenkinsfile

    pipeline{
        agent any
        stages {
            //由于源码和Jenkinsfile处于同一仓库,在Jenkins项目执行时,会先将Jenkinsfile所在的仓库克隆下来,为了简单,这里就不重复添加拉取源码的操作了
            //其它情况,如使用多个仓库一些构建的,这种就需要额外添加拉取代码的stage了。
            stage('Build Artifact') {
                steps{
                    sh label:'maven building', script: 'mvn clean package -DskipTests'
                }
            }
            stage('Build Image'){
                steps{
                    sh label:'image building', script: '/bin/bash artifact2image.sh'
                }
            }
            stage('Deploy k8s'){
                steps{
                    sh label:'deploy image to k8s', script: '/bin/bash deploy2k8s.sh'
                }
            }
        }
        post {
            success{
                //成功清理工作空间,失败保留现场
                cleanWs()
            }
        }
    }
    
    

    deploy.yaml

    apiVersion: apps/v1   #api版本
    kind: Deployment      #对象类型
    metadata:             #元信息标识
      name: cicd-demo
      namespace: default
      labels:             #自定义Deployment绑定的标签
        app: cicd-demo
    spec:                 #期待运行状态
      replicas: 1         #部署实例数
      selector:
        matchLabels:
          app: cicd-demo
      template:           #pod
        metadata:
          labels:         #与spec.selector.matchLabels相同
            app: cicd-demo
        spec:             #pod容器期望的状态
          containers:
            - name: cicd-demo
              image: {{IMAGE_NAME}}
              imagePullPolicy: IfNotPresent
              env:
                - name: JVM_OPTS
                  value: "-Xms128m -Xmx256m"
              ports:
                - containerPort: 8080
    
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: cicd-demo-svc
      namespace: default
      labels:
        service: cicd-demo-svc
    spec:
      selector:            #匹配pod标签
        app: cicd-demo
      type: NodePort     #svc类型,ExternalName, ClusterIP, NodePort, and LoadBalancer
      ports:
        - name: cicd-demo-port
          protocol: TCP
          port: 8080
          nodePort: 30000
    
    

    09 创建Jenkins流水线项目

    访问 http://192.168.87.128:8080/jenkins 并登录,以下创建新流水线demo构建项目

    部署与验证

    执行构建demo程序

    日志输出:

    Started by user admin
    Obtained Jenkinsfile from git https://github.com/hellxz/cicd-demo.git
    Running in Durability level: MAX_SURVIVABILITY
    [Pipeline] Start of Pipeline
    [Pipeline] node
    Running on Jenkins in /home/hellxz/jenkins/home/workspace/demo
    [Pipeline] {
    [Pipeline] stage
    [Pipeline] { (Declarative: Checkout SCM)
    [Pipeline] checkout
    Selected Git installation does not exist. Using Default
    The recommended git tool is: NONE
    No credentials specified
    Cloning the remote Git repository
    Cloning repository https://github.com/hellxz/cicd-demo.git
     > git init /home/hellxz/jenkins/home/workspace/demo # timeout=10
    Fetching upstream changes from https://github.com/hellxz/cicd-demo.git
     > git --version # timeout=10
     > git --version # 'git version 2.17.1'
     > git fetch --tags --progress -- https://github.com/hellxz/cicd-demo.git +refs/heads/*:refs/remotes/origin/* # timeout=10
     > git config remote.origin.url https://github.com/hellxz/cicd-demo.git # timeout=10
     > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
    Avoid second fetch
     > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
    Checking out Revision 1fc086590e32c001c8b65a4c575f4aa1b5b20413 (refs/remotes/origin/master)
     > git config core.sparsecheckout # timeout=10
     > git checkout -f 1fc086590e32c001c8b65a4c575f4aa1b5b20413 # timeout=10
    Commit message: "update"
     > git rev-list --no-walk 1fc086590e32c001c8b65a4c575f4aa1b5b20413 # timeout=10
    [Pipeline] }
    [Pipeline] // stage
    [Pipeline] withEnv
    [Pipeline] {
    [Pipeline] stage
    [Pipeline] { (Build Artifact)
    [Pipeline] sh (maven building)
    + mvn clean package -DskipTests
    NOTE: Picked up JDK_JAVA_OPTIONS:  --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
    [INFO] Scanning for projects...
    
    [INFO] 
    [INFO] ----------------------< online.hellxz:cicd-demo >-----------------------
    [INFO] Building cicd-demo 0.0.1
    [INFO] --------------------------------[ jar ]---------------------------------
    [INFO] 
    [INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ cicd-demo ---
    [INFO] 
    [INFO] --- maven-resources-plugin:3.2.0:resources (default-resources) @ cicd-demo ---
    [INFO] Using 'UTF-8' encoding to copy filtered resources.
    [INFO] Using 'UTF-8' encoding to copy filtered properties files.
    [INFO] Copying 1 resource
    [INFO] Copying 0 resource
    [INFO] The encoding used to copy filtered properties files have not been set. This means that the same encoding will be used to copy filtered properties files as when copying other filtered resources. This might not be what you want! Run your build with --debug to see which files might be affected. Read more at https://maven.apache.org/plugins/maven-resources-plugin/examples/filtering-properties-files.html
    [INFO] 
    [INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ cicd-demo ---
    
    [INFO] Changes detected - recompiling the module!
    [INFO] Compiling 1 source file to /home/hellxz/jenkins/home/workspace/demo/target/classes
    
    [INFO] 
    [INFO] --- maven-resources-plugin:3.2.0:testResources (default-testResources) @ cicd-demo ---
    [INFO] Using 'UTF-8' encoding to copy filtered resources.
    [INFO] Using 'UTF-8' encoding to copy filtered properties files.
    [INFO] skip non existing resourceDirectory /home/hellxz/jenkins/home/workspace/demo/src/test/resources
    [INFO] 
    [INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ cicd-demo ---
    [INFO] No sources to compile
    [INFO] 
    [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ cicd-demo ---
    [INFO] Tests are skipped.
    [INFO] 
    [INFO] --- maven-jar-plugin:3.2.0:jar (default-jar) @ cicd-demo ---
    [INFO] Building jar: /home/hellxz/jenkins/home/workspace/demo/target/cicd-demo-0.0.1.jar
    [INFO] 
    [INFO] --- spring-boot-maven-plugin:2.4.1:repackage (repackage) @ cicd-demo ---
    [INFO] Replacing main artifact with repackaged archive
    [INFO] ------------------------------------------------------------------------
    [INFO] BUILD SUCCESS
    [INFO] ------------------------------------------------------------------------
    [INFO] Total time:  3.248 s
    [INFO] Finished at: 2020-12-23T11:51:20Z
    [INFO] ------------------------------------------------------------------------
    [Pipeline] }
    [Pipeline] // stage
    [Pipeline] stage
    [Pipeline] { (Build Image)
    
    [Pipeline] sh (image building)
    + /bin/bash artifact2image.sh
    Sending build context to Docker daemon  17.26MB
    
    Step 1/5 : FROM openjdk:8-jdk
     ---> 89f100fa8f9f
    Step 2/5 : WORKDIR /root
     ---> Using cache
     ---> 17670bfa01f1
    Step 3/5 : ADD target/*.jar app.jar
    
     ---> b7e4292e4f80
    Step 4/5 : ENV JVM_OPTS=""
     ---> Running in 0be8c16c9c8a
    Removing intermediate container 0be8c16c9c8a
     ---> 3ef66b100082
    Step 5/5 : CMD java -Djava.security.egd=file:/dev/./urandom     ${JVM_OPTS}     -jar app.jar
     ---> Running in c8209a642a2d
    Removing intermediate container c8209a642a2d
     ---> bc7859f9b4fe
    Successfully built bc7859f9b4fe
    Successfully tagged 192.168.87.129:5000/cicd-demo:20205123_1151
    
    The push refers to repository [192.168.87.129:5000/cicd-demo]
    44cbb90cc2c3: Preparing
    f85e383859a1: Preparing
    ffb4778f8a52: Preparing
    e528f2c31deb: Preparing
    c5f4367d4a59: Preparing
    ceecb62b2fcc: Preparing
    193bc1d68b80: Preparing
    f0e10b20de19: Preparing
    ceecb62b2fcc: Waiting
    193bc1d68b80: Waiting
    f0e10b20de19: Waiting
    ffb4778f8a52: Layer already exists
    e528f2c31deb: Layer already exists
    f85e383859a1: Layer already exists
    193bc1d68b80: Layer already exists
    c5f4367d4a59: Layer already exists
    ceecb62b2fcc: Layer already exists
    f0e10b20de19: Layer already exists
    44cbb90cc2c3: Pushed
    
    20205123_1151: digest: sha256:24976370741b8e20a7e8c7d18e13f4b72bb492f9c77908927d3b17b8e1ce75d4 size: 2006
    Untagged: 192.168.87.129:5000/cicd-demo:20205123_1151
    Untagged: 192.168.87.129:5000/cicd-demo@sha256:24976370741b8e20a7e8c7d18e13f4b72bb492f9c77908927d3b17b8e1ce75d4
    Deleted: sha256:bc7859f9b4fefc5fdcb39bab4c77dff94980d70fcd21b0b3755f317f517e4291
    Deleted: sha256:3ef66b100082a9d6949802c53fe57442d0e156216f0ba7a544b4e44d5a80e7a4
    Deleted: sha256:b7e4292e4f807a13f800b5acc8adb0a84926e5ce5c4cc850a2769503558efcb9
    Deleted: sha256:2afe950a3c18304e26f7d6b3f3bc48d9de631eb8827203bfa273d361647d0951
    [Pipeline] }
    [Pipeline] // stage
    [Pipeline] stage
    [Pipeline] { (Deploy k8s)
    [Pipeline] sh (deploy image to k8s)
    + /bin/bash deploy2k8s.sh
    
    deployment.apps/cicd-demo created
    service/cicd-demo-svc created
    [Pipeline] }
    [Pipeline] // stage
    [Pipeline] stage
    [Pipeline] { (Declarative: Post Actions)
    
    [Pipeline] cleanWs
    [WS-CLEANUP] Deleting project workspace...
    [WS-CLEANUP] Deferred wipeout is used...
    [WS-CLEANUP] done
    [Pipeline] }
    [Pipeline] // stage
    [Pipeline] }
    [Pipeline] // withEnv
    [Pipeline] }
    [Pipeline] // node
    [Pipeline] End of Pipeline
    Finished: SUCCESS
    
    

    可以看到以下输出,说明程序已部署到k8s集群中了

    deployment.apps/cicd-demo created
    service/cicd-demo-svc created
    

    ssh登录k8s的机器验证

    总结

    简单总结下,Jenkins与Kubernetes的集成,更多地是Jenkins如何控制Kubernetes部署的流程,文中例子是通过 构建并上传镜像ssh发送配置和命令 实现的。

    本文是参考慕课网上的教程,快速扫了下视频实践所得,是了解其思路后自行实现的。

    实际操作上才发现,脑子说会了,然后双手却没有跟上,还是要动手啊!

    操作了一天半左右,最开始还想着用vagrant初始化VM,由于工作上有个虚拟机是VBox5的,升级新版不能用。搞了半天放弃了Vagrant(VBox6.1升级了命令参数不兼容旧版,导致Vagrant为兼容它把几乎所有版本Vagrant都改了……)。

    文中难免因为安装部分的描述而过于冗长,感谢大家能看到这里,最后再次感谢慕课网的鹿哥

    参考

    https://www.imooc.com/learn/1112
    https://cloud.tencent.com/developer/article/1394657
    https://www.jianshu.com/p/3f13850d66ea
    https://www.cnblogs.com/hellxz/p/use-kubeadm-init-kubernetes-cluster.html
    https://www.cnblogs.com/hellxz/p/how-kubernetes-deploy-application.html
    https://developer.aliyun.com/article/221687?spm=a2c6h.12873639.0.0.755c6235EZuEht
    https://minikube.sigs.k8s.io/docs/start
    https://www.cnblogs.com/hellxz/p/install_jenkins.html
    https://www.cnblogs.com/hellxz/p/jenkins_install_plugins_faster.html

  • 相关阅读:
    Halcon二维仿射变换实例探究
    redis主从+哨兵实战
    单点安装redis+哨兵
    一个学习技术不错的网站
    reset master 不能乱用呀
    MySQL 5.7基于GTID的主从复制实践
    『浅入深出』MySQL 中事务的实现
    使用二进制包安装mysql
    jenkins+gitlab
    mysql的Innodb存储引擎提一嘴
  • 原文地址:https://www.cnblogs.com/hellxz/p/easyway_jenkins_integration_k8s.html
Copyright © 2011-2022 走看看