前言
最近在学习如何使用jenkins结合阿里云k8s自动化集成和部署springcloud微服务和前端next.js项目,现在记录下来分享给大家,文章有什么不妥的地方,希望大佬们批评指正。
DevOps CI/CD 流程
面向对象
-
熟练使用vue或者react
-
熟练使用SpringCloud微服务
-
熟练使用docker容器
-
熟练使用jenkins自动化运维工具
-
熟练使用k8s(deployment、service、ingress)
准备工作
1、购买阿里云ACK集群(或者自行搭建)
我购买的是阿里云ACK托管版,创建集群地址
注意:创建ACK集群不收费,收费的是里面的NAT网关、SLB负载均衡、ECS服务器等
2、安装gitlab
这里我之前写了一篇阿里云ECS搭建gitlab,安装gitlab地址
3、安装jenkins
这里我之前写了一篇阿里云ECS搭建jenkins,安装jenkins地址
4、安装docker
使用官网推荐yum 安装比较方便,安装如下:
sudo yum install -y yum-utils \
device-mapper-persistent-data \
lvm2
sudo yum-config-manager \
--add-repo \
https://download.docker.com/linux/centos/docker-ce.repo
sudo yum install docker-ce docker-ce-cli containerd.io
系统架构
Next.js 前端开发
Next.js项目:
服务:react+next.js
服务端口:3000,
K8S:Deployment+Server+Ingress
Pod名:demo-webapp
Dockerfile
FROM node:12
# 设置工作路径。所有的路径都会关联WORKDIR
WORKDIR /usr/src/app
# 安装依赖
COPY package*.json ./
RUN npm install
# 拷贝源文件
COPY . .
# 构建应用
RUN npm run build
# 运行应用
CMD [ "npm", "start" ]
SpringCloud微服务开发
1、服务发现:SpringCloud Eureka
微服务名:demo-eureka-server
微服务端口:8761,
K8S:Deployment+Service+Ingress(因为要通过网址查看服务注册情况,所以要添加ingress)
Pod名:demo-eureka-server
2、服务配置:SpringCloud Config
微服务名:demo-config-server
微服务端口:8888,
K8S:Deployment+Service
Pod名:demo-config-server
3、服务认证和授权:SpringSecurity + Oauth2
微服务名:demo-auth-service
微服务端口:8901,
K8S:Deployment+Service
Pod名:demo-auth-service
4、服务网关:SpringCloud Zuul
微服务名:demo-auth-service
微服务端口:5555,
K8S:Deployment+Service+Ingress(服务网关是所有服务对外唯一入口,所以要配置ingress)
Pod名:demo-auth-service
5、编写Dockerfile、K8S yaml
为了方便起见,上面的微服务名和Pod名都设置相同名称,具体的服务开发过程这里就暂时忽略,后面有时间再写篇搭建企业级微服务的文章吧,相关微服务的目录结构如下所示:
注意:不同服务之间Dockerfile和k8s yaml文件大同小异,我们以Eureka为例:
Eureka Dockerfile:
FROM openjdk:8-jdk-alpine MAINTAINER "zhangwei"<zhangwei900808@126.com> RUN mkdir -p /usr/local/configsvr ARG JAR_FILE ADD ${JAR_FILE} /usr/local/configsvr/app.jar ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar","/usr/local/configsvr/app.jar"] EXPOSE 8888
Eureka K8S Yaml:
apiVersion: extensions/v1beta1 kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/configuration-snippet: | rewrite ^/eureka/css/(.*)$ /eureka/eureka/css/$1 redirect; rewrite ^/eureka/js/(.*)$ /eureka/eureka/js/$1 redirect; nginx.ingress.kubernetes.io/force-ssl-redirect: 'true' nginx.ingress.kubernetes.io/rewrite-target: /$2 nginx.ingress.kubernetes.io/service-weight: '' generation: 4 name: <podname>-ingress namespace: default spec: rules: - host: baidu.com http: paths: - backend: serviceName: <podname>-svc servicePort: 8761 path: /eureka(/|$)(.*) pathType: ImplementationSpecific --- apiVersion: v1 kind: Service metadata: name: <podname>-svc namespace: default spec: externalTrafficPolicy: Local ports: - nodePort: 31061 port: 8761 protocol: TCP targetPort: 8761 selector: app: <podname> sessionAffinity: None type: NodePort --- apiVersion: apps/v1 kind: Deployment metadata: annotations: deployment.kubernetes.io/revision: '1' generation: 1 labels: app: <podname> name: <podname> namespace: default spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: <podname> strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: labels: app: <podname> spec: containers: - env: - name: LANG value: C.UTF-8 - name: JAVA_HOME value: /usr/lib/jvm/java-1.8-openjdk image: <imagename> imagePullPolicy: IfNotPresent name: <podname> ports: - containerPort: 8761 protocol: TCP resources: requests: cpu: 250m memory: 512Mi terminationMessagePath: /dev/termination-log terminationMessagePolicy: File dnsPolicy: ClusterFirst restartPolicy: Always schedulerName: default-scheduler securityContext: {} terminationGracePeriodSeconds: 30
注意:不同的策服务bootstrap.yml和application.yml文件还是有所区别,比如下面这些:
demo-auth-service、demo-zuul-server 配置bootstrap.yml文件添加SpringCloud Config地址:
spring: application: name: authservice cloud: config: enabled: true # config的Server名 uri: http://demo-config-server-svc:8888
demo-auth-service、demo-zuul-server、demo-config-server 配置application.yml文件添加Eureka地址:
eureka: instance: preferIpAddress: true hostname: demo-eureka-server-svc client: register-with-eureka: true fetch-registry: true service-url: # eureka的Server名 defaultZone: http://demo-eureka-server-svc:8761/eureka/
Git工作流
按照Git工作流,一般我们可以分为:开发环境(develop)、测试环境(release)、预生产环境(uat)和生产环境(prop),对应的分支分别是:dev、test、release、master,因此不同阶段提交的代码所属分支也不相同,且dockerfile、jenkins pipline、k8s yaml文件也不相同,这个要注意下。
Jenkins DevOps CI/CD
1、视图规范
jenkins可以按照git工作流添加:测试视图、预生产视图、生产视图,如下所示:
2、创建流水线任务
我们先创建流水线任务并编写pipline script脚本来创建CI/CD流水线步骤
3、编写前端的pipline script
先编写环境变量
environment { GIT_REPOSITORY="前端代码仓库地址" K8S_YAML="k8s yaml文件所在目录" DOCKER_USERNAME="docker 仓库用户名" DOCKER_PWD="docker仓库密码" ALIYUN_DOCKER_HOST = '阿里云docker仓库域名' ALIYUN_DOCKER_NAMESPACE="阿里云docker仓库命名空间" ALIYUN_DOCKER_REPOSITORY_NAME="阿里云docker仓库命名空间下的仓库名" }
步骤一:克隆代码
stage("Clone") { steps { echo "1.Clone Stage" // 删除文件夹 deleteDir() // 测试分支,jenkins-gitlab-ssh-hash是ssh密钥,替换成自己的就好 git branch: 'test', credentialsId: 'jenkins-gitlab-ssh-hash', url: "${GIT_REPOSITORY}" script { // 获取git提交的hash值做为docker镜像tag GIT_TAG = sh(returnStdout: true, script: 'git rev-parse --short HEAD').trim() // 组装成完整地址 DOCKER_REPOSITORY = "${ALIYUN_DOCKER_HOST}/${ALIYUN_DOCKER_NAMESPACE}/${ALIYUN_DOCKER_REPOSITORY_NAME}" DOCKER_REPOSITORY_TAG = "${DOCKER_REPOSITORY}:${GIT_TAG}" } } }
步骤二:代码测试
stage("Test") { steps { echo "2.Test Stage" } }
步骤三:制作docker镜像
stage("Build") { steps { echo "3.Build Docker Image Stage" sh "docker build -t ${DOCKER_REPOSITORY_TAG} -f docker/Dockerfile ." } }
步骤四:推送docker镜像
stage("Push") { steps { echo "4.Push Docker Image Stage" //推送Docker镜像,username 跟 password 为 阿里云容器镜像服务的账号密码 sh "docker login --username=${DOCKER_USERNAME} --password=${DOCKER_PWD} ${ALIYUN_DOCKER_HOST}" // 开始推送镜像到阿里云docker镜像仓库 sh "docker push ${DOCKER_REPOSITORY_TAG}" // 删除jenkins生成的image sh ''' docker images | grep seaurl | awk '{print $3}' | xargs docker rmi -f ''' } }
步骤五:k8s部署docker镜像
stage("Deploy") { steps { echo "5.发布镜像" // 使用sed替换k8s yaml文件中的<imagename>和<podname> sh "sed -i 's#<imagename>#${DOCKER_REPOSITORY_TAG}#g;s#<podname>#${POD_NAME}#g' ${K8S_YAML}" // 执行应用k8s yaml sh "kubectl apply -f ${K8S_YAML}" } }
sed 语法大家可以自行百度,这里我使用的是#分隔,而没有使用/分隔,原因是分隔的字符中包含/,所以不能再用。
首先,我们来看看前端k8s yaml文件
apiVersion: extensions/v1beta1 kind: Ingress metadata: annotations: cert-manager.io/cluster-issuer: letsencrypt-prod-http01 kubernetes.io/ingress.class: nginx nginx.ingress.kubernetes.io/force-ssl-redirect: 'true' nginx.ingress.kubernetes.io/service-weight: '' generation: 3 name: <podname>-ingress namespace: default spec: rules: - host: baidu.com http: paths: - backend: serviceName: <podname>-svc servicePort: 3000 path: / pathType: ImplementationSpecific tls: - hosts: - baidu.com secretName: <podname>-ingress --- apiVersion: v1 kind: Service metadata: name: <podname>-svc namespace: default spec: ports: - port: 3000 protocol: TCP targetPort: 3000 selector: app: <podname> sessionAffinity: None type: ClusterIP --- apiVersion: apps/v1 kind: Deployment metadata: annotations: deployment.kubernetes.io/revision: '1' generation: 1 labels: app: <podname> name: <podname> namespace: default spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: <podname> strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: labels: app:<podname> spec: containers: - image: <imagename> imagePullPolicy: IfNotPresent name: <podname> resources: requests: cpu: 250m memory: 512Mi
上面的yaml文件中包含:deployment、service和ingress,ingress中配置了tls,因此可以使用https来访问域名,并且配置了nginx.ingress.kubernetes.io/force-ssl-redirect: 'true'所以可以自动将http 跳转到https,其中里面的<podname>和<imagename>是要被sed替换成真实名称和地址。
完整的pipline script脚本:
pipeline { agent any environment { GIT_REPOSITORY="前端代码仓库地址" K8S_YAML="k8s yaml文件所在目录" DOCKER_USERNAME="docker 仓库用户名" DOCKER_PWD="docker仓库密码" ALIYUN_DOCKER_HOST = '阿里云docker仓库域名' ALIYUN_DOCKER_NAMESPACE="阿里云docker仓库命名空间" ALIYUN_DOCKER_REPOSITORY_NAME="阿里云docker仓库命名空间下的仓库名" } stages { stage("Clone") { steps { echo "1.Clone Stage" // 删除文件夹 deleteDir() // 测试分支,jenkins-gitlab-ssh-hash是ssh密钥,替换成自己的就好 git branch: 'test', credentialsId: 'jenkins-gitlab-ssh-hash', url: "${GIT_REPOSITORY}" script { // 获取git提交的hash值做为docker镜像tag GIT_TAG = sh(returnStdout: true, script: 'git rev-parse --short HEAD').trim() // 组装成完整地址 DOCKER_REPOSITORY = "${ALIYUN_DOCKER_HOST}/${ALIYUN_DOCKER_NAMESPACE}/${ALIYUN_DOCKER_REPOSITORY_NAME}" DOCKER_REPOSITORY_TAG = "${DOCKER_REPOSITORY}:${GIT_TAG}" } } } stage("Test") { steps { echo "2.Test Stage" } } stage("Build") { steps { echo "3.Build Docker Image Stage" sh "docker build -t ${DOCKER_REPOSITORY_TAG} -f docker/Dockerfile ." } } stage("Push") { steps { echo "4.Push Docker Image Stage" //推送Docker镜像,username 跟 password 为 阿里云容器镜像服务的账号密码 sh "docker login --username=${DOCKER_USERNAME} --password=${DOCKER_PWD} ${ALIYUN_DOCKER_HOST}" // 开始推送镜像到阿里云docker镜像仓库 sh "docker push ${DOCKER_REPOSITORY_TAG}" // 删除jenkins生成的image sh ''' docker images | grep seaurl | awk '{print $3}' | xargs docker rmi -f ''' } } stage("Deploy") { steps { echo "5.发布镜像" // 使用sed替换k8s yaml文件中的<imagename>和<podname> sh "sed -i 's#<imagename>#${DOCKER_REPOSITORY_TAG}#g;s#<podname>#${POD_NAME}#g' ${K8S_YAML}" // 执行应用k8s yaml sh "kubectl apply -f ${K8S_YAML}" } } } }
点击立即构建,如下所示:
4、编写微服务的pipline script
先编写环境变量
environment { GIT_REPOSITORY="代码仓库地址" MODULE_NAME="maven 模块名" POD_NAME="k8s pod name" K8S_YAML="${MODULE_NAME}/src/main/k8s/eurekasvr.yaml" DOCKER_USERNAME="docker 仓库用户名" DOCKER_PWD="docker仓库密码" ALIYUN_DOCKER_HOST = 阿里云docker仓库域名' ALIYUN_DOCKER_NAMESPACE="阿里云docker仓库命名空间" ALIYUN_DOCKER_REPOSITORY_NAME="阿里云docker仓库命名空间下的仓库名" }
步骤一:克隆代码
stage("Clone") { steps { echo "1.Clone Stage" // 删除文件夹 deleteDir() // 测试分支,jenkins-gitlab-ssh-hash是ssh密钥,替换成自己的就好 git branch: 'test', credentialsId: 'jenkins-gitlab-ssh-hash', url: "${GIT_REPOSITORY}" script { // 获取git提交的hash值做为docker镜像tag GIT_TAG = sh(returnStdout: true, script: 'git rev-parse --short HEAD').trim() // 组装成完整地址 DOCKER_REPOSITORY = "${ALIYUN_DOCKER_HOST}/${ALIYUN_DOCKER_NAMESPACE}/${ALIYUN_DOCKER_REPOSITORY_NAME}" DOCKER_REPOSITORY_TAG = "${DOCKER_REPOSITORY}:${GIT_TAG}" } } }
步骤二:代码测试
stage("Test") { steps { echo "2.Test Stage" } }
步骤三:制作docker镜像
stage("Build") { steps { echo "3.Build Server" sh "mvn -e -U -pl ${MODULE_NAME} -am clean package -Dmaven.test.skip=true dockerfile:build -Ddockerfile.tag=${GIT_TAG} -Ddockerfile.repository=${DOCKER_REPOSITORY}" } }
这里说明一下,因为我们使用的是maven 多模块所以打包编译的时候要分模块来打包,所以要使用-pl指定模块名,然后我们pom.xml中使用了dockerfile-maven-plugin,如下所示:
<build> <plugins> <plugin> <groupId>com.spotify</groupId> <artifactId>dockerfile-maven-plugin</artifactId> <version>1.4.10</version> <configuration> <!-- 指定dockerfile所在目录--> <dockerfile>src/main/docker/Dockerfile</dockerfile> <buildArgs> <!--提供参数向Dockerfile传递--> <JAR_FILE>target/${project.build.finalName}.jar</JAR_FILE> </buildArgs> </configuration> </plugin> <plugin> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-maven-plugin</artifactId> </plugin> </plugins> </build>
步骤四:推送docker镜像
stage("Push") { steps { echo "4.Push Docker Image Stage" //推送Docker镜像,username 跟 password 为 阿里云容器镜像服务的账号密码 sh "docker login --username=${DOCKER_USERNAME} --password=${DOCKER_PWD} ${ALIYUN_DOCKER_HOST}" // 开始推送镜像到阿里云docker镜像仓库 sh "docker push ${DOCKER_REPOSITORY_TAG}" // 删除jenkins生成的image sh ''' docker images | grep seaurl | awk '{print $3}' | xargs docker rmi -f ''' } }
步骤五:k8s部署docker镜像
stage("Deploy") { steps { echo "5.发布镜像" // 使用sed替换k8s yaml文件中的<imagename>和<podname> sh "sed -i 's#<imagename>#${DOCKER_REPOSITORY_TAG}#g;s#<podname>#${POD_NAME}#g' ${K8S_YAML}" // 执行应用k8s yaml sh "kubectl apply -f ${K8S_YAML}" } }
sed 语法大家可以自行百度,这里我使用的是#分隔,而没有使用/分隔,原因是分隔的字符中包含/,所以不能再用。
首先,我们来看看Eureka k8s yaml文件
apiVersion: extensions/v1beta1 kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/configuration-snippet: | rewrite ^/eureka/css/(.*)$ /eureka/eureka/css/$1 redirect; rewrite ^/eureka/js/(.*)$ /eureka/eureka/js/$1 redirect; nginx.ingress.kubernetes.io/force-ssl-redirect: 'true' nginx.ingress.kubernetes.io/rewrite-target: /$2 nginx.ingress.kubernetes.io/service-weight: '' generation: 4 name: <podname>-ingress namespace: default spec: rules: - host: baidu.com http: paths: - backend: serviceName: <podname>-svc servicePort: 8761 path: /eureka(/|$)(.*) pathType: ImplementationSpecific --- apiVersion: v1 kind: Service metadata: name: <podname>-svc namespace: default spec: externalTrafficPolicy: Local ports: - nodePort: 31061 port: 8761 protocol: TCP targetPort: 8761 selector: app: <podname> sessionAffinity: None type: NodePort --- apiVersion: apps/v1 kind: Deployment metadata: annotations: deployment.kubernetes.io/revision: '1' generation: 1 labels: app: <podname> name: <podname> namespace: default spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: <podname> strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: labels: app: <podname> spec: containers: - env: - name: LANG value: C.UTF-8 - name: JAVA_HOME value: /usr/lib/jvm/java-1.8-openjdk image: <imagename> imagePullPolicy: IfNotPresent name: <podname> ports: - containerPort: 8761 protocol: TCP resources: requests: cpu: 250m memory: 512Mi terminationMessagePath: /dev/termination-log terminationMessagePolicy: File dnsPolicy: ClusterFirst restartPolicy: Always schedulerName: default-scheduler securityContext: {} terminationGracePeriodSeconds: 30
上面的yaml文件中包含:deployment、service和ingress,ingress中配置了tls,因此可以使用https来访问域名,并且配置了nginx.ingress.kubernetes.io/force-ssl-redirect: 'true'所以可以自动将http 跳转到https,其中里面的<podname>和<imagename>是要被sed替换成真实名称和地址。
完整的pipline script脚本:
pipeline { agent any environment { GIT_REPOSITORY="代码仓库地址" MODULE_NAME="maven 模块名" POD_NAME="k8s pod name" K8S_YAML="${MODULE_NAME}/src/main/k8s/eurekasvr.yaml" DOCKER_USERNAME="docker 仓库用户名" DOCKER_PWD="docker仓库密码" ALIYUN_DOCKER_HOST = 阿里云docker仓库域名' ALIYUN_DOCKER_NAMESPACE="阿里云docker仓库命名空间" ALIYUN_DOCKER_REPOSITORY_NAME="阿里云docker仓库命名空间下的仓库名" } stages { stage("Clone") { steps { echo "1.Clone Stage" // 删除文件夹 deleteDir() git branch: 'test',credentialsId: '1297dda3-e592-4e70-8fb0-087a26c08db0', url: "${GIT_REPOSITORY}" script { // 获取git代码tag为docker仓库tag // GIT_TAG = sh(returnStdout: true,script: 'git describe --tags --always').trim() // 获取git提交hash做为docker仓库tag GIT_TAG = sh(returnStdout: true, script: 'git rev-parse --short HEAD').trim() DOCKER_REPOSITORY = "${ALIYUN_DOCKER_HOST}/${ALIYUN_DOCKER_NAMESPACE}/${ALIYUN_DOCKER_REPOSITORY_NAME}" // docker 阿里镜像仓库 DOCKER_REPOSITORY_TAG = "${DOCKER_REPOSITORY}:${GIT_TAG}" } } } stage("Test") { steps { echo "2.Test Stage" } } stage("Build") { steps { echo "3.Build Server" sh "mvn -e -U -pl ${MODULE_NAME} -am clean package -Dmaven.test.skip=true dockerfile:build -Ddockerfile.tag=${GIT_TAG} -Ddockerfile.repository=${DOCKER_REPOSITORY}" } } stage("Push") { steps { echo "4.Push Docker Image Stage" //推送Docker镜像,username 跟 password 为 阿里云容器镜像服务的账号密码 sh "docker login --username=${DOCKER_USERNAME} --password=${DOCKER_PWD} ${ALIYUN_DOCKER_HOST}" // 开始推送镜像到阿里云docker镜像仓库 sh "docker push ${DOCKER_REPOSITORY_TAG}" // 删除jenkins生成的image sh ''' docker images | grep seaurl | awk '{print $3}' | xargs docker rmi -f ''' } } stage("Deploy") { steps { echo "5.发布镜像" sh "sed -i 's#<dockerrepository>#${DOCKER_REPOSITORY_TAG}#g;s#<podname>#${POD_NAME}#g' ${K8S_YAML}" sh "kubectl apply -f ${K8S_YAML}" } } } }
点击立即构建,如下所示:
查看k8s是否成功
可以通过命令
kubectl get deploy
kubectl get pod
kubectl get svc
kubectl get ingress
还可以查看pod日志,对其成功或失败进行分析:
kubectl logs podname
访问Eureka
postman访问微服务接口地址
可以通过postman接口访问对外的zuul地址,查看是否可以认证通过:
浏览器访问Web页面地址
可以通过next.js的部署地址,查看是否部署成功:
总结
1、jenkins pipline script脚本和Dockerfile写法网上各式各样,大家找到一个符合标准的就好
2、kubectl apply -f 的作用是:如果没有创建deployment就会创建,否则就更新
3、微服务k8s yaml文件中Service要设置type: NodePort否则,微服务间不能通信
4、jenkins 编译微服务的maven模块要用到-pl 和 dockerfile-maven-plugin
5、微服务如果要用k8s ingress要设置https,以及http自动跳转到https
引用
JenkinsPipeline部署一个Kubernetes 应用
采用jenkins pipeline实现自动构建并部署至k8s
Configuring-CI-CD-on-Kubernetes-with-Jenkins
spring-k8s
jenkins pipeline 自动构建并部署至k8s
微服务实战(一)基于OAUTH2.0统一认证授权的微服务基础架构
使用cert-manager申请免费的HTTPS证书
SPRINGBOOT利用SPRING.PROFILES.ACTIVE=@SPRING.ACTIVE@不同环境下灵活切换配置文件
https://kuboard.cn/learning/k8s-practice/ocp/eureka-server.html#%E6%9F%A5%E7%9C%8B%E9%83%A8%E7%BD%B2%E7%BB%93%E6%9E%9C
kubernetes部署微服务spring cloud的简单例子
k8s-nginx-ingress eureka二级路径转发的问题
转自:https://segmentfault.com/a/1190000040119257?utm_source=tag-newest