交付dubbo微服务到k8s (二）

1. 交付dubbo服务提供者

1.1.1 在master机器上安装helm

在master机器 192.168.31.37 上安装helm3.1

wget https://get.helm.sh/helm-v3.1.0-linux-amd64.tar.gz
tar zxvf helm-v3.0.0-linux-amd64.tar.gz 
mv linux-amd64/helm /usr/bin/

1.1.2 配置helm chart

在 master 机器 192.168.31.37上创建helm chart

mkdir /root/helm
helm create dubbo-demo-service

• values.yaml

image:
  repository: harbor.od.com/app/dubbo-demo-service 
  tag: master_20200814_0116

• templates/dp.yaml

kind: Deployment
apiVersion: apps/v1
metadata:
  name: dubbo-demo-service
  namespace: app
  labels:
    name: dubbo-demo-service
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dubbo-demo-service
  template:
    metadata:
      labels:
        app: dubbo-demo-service
    spec:
      containers:
      - name: dubbo-demo-service
        image: {{ .Values.image.repository }}:{{ .Values.image.tag }} 
        ports:
        - containerPort: 20880
          protocol: TCP
        env:
        - name: JAR_BALL
          value: dubbo-server.jar
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext:
        runAsUser: 0
      schedulerName: default-scheduler

1.1.3 将helm 的chart拷贝到 192.168.31.200:/data/jenkins/jenkins_home/helm

在 192.168.31.200:/data/jenkins/jenkins_home下创建helm目录， 将chart从192.168.31.37拷贝到这个目录下

将 master下的 ~/.kube/config 拷贝到helm目录下

tree helm/

1.2 配置参数

1.2.1 设置变量 app_name和image_name

• app_name: app名字

变量名: app_name
变量的默认值: dubbo-demo-service

• image_name: 镜像名字

变量名: image_name	
变量的默认值: app/dubbo-demo-service

1.2.2 设置变量 git_repo 和 git_version

• git_repo: 代码的git地址

变量名: git_repo
变量的默认值: https://gitee.com/chen1219_1/dubbo-demo-service.git

• git_version: 拉取代码哪个分支

变量名: git_version
变量的默认值: master

1.2.3 设置变量 add_tag 和 mvn_dir

• add_tag: 打包代码后的镜像的tag

变量名: add_tag
变量的默认值: 20200814_0116

• mvn_dir: maven打包时候的执行目录

变量名: mvn_dir
变量的默认值: ./

1.2.4 设置变量 target_dir 和 mvn_cmd

• target_dir: maven打包完成后生成的jar包所在的目录， 一般是在执行目录下生成target目录， jar包在target目录下

变量名: target_dir
变量的默认值: ./dubbo-server/target

• mvn_cmd: maven打包时候执行的命令

变量名: mvn_cmd
变量的默认值: mvn clean package -Dmaven.test.skip=true

1.2.5 设置变量 base_image 和 maven

• base_image： 构建镜像时候所用的底包是哪个镜像

变量名: base_image
变量的选项:
base/jre8:8u112
base/jre7:7u80

• maven: 构建镜像时候使用的maven版本

  这里的变量的选项要和在 192.168.31.200:/data/jenkins/jenkins_home下的maven的目录是一样的， 在打包时候直接是使用这个绝对路径来调用mvn命令进行打包的

变量名: maven
变量的选项:
3.6.3-8u242
3.6.1-8u232

1.3 配置流水线

pipeline {
    agent any

    stages {
        stage('pull') {  //get project code from repo
            steps {
                sh "git clone ${params.git_repo} ${params.app_name}/${env.BUILD_NUMBER} && cd ${params.app_name}/${env.BUILD_NUMBER} && git checkout ${params.git_version}"	
            }
        }
		
		stage('build') {  //exec mvn cmd
            steps {
                sh "cd ${params.app_name}/${env.BUILD_NUMBER} && /var/jenkins_home/maven-${params.maven}/bin/${params.mvn_cmd}"	
				sh "cd ${params.app_name}/${env.BUILD_NUMBER} && cd ${params.target_dir} && mkdir project_dir && mv *.jar ./project_dir"
				writeFile file: "${params.app_name}/${env.BUILD_NUMBER}/Dockerfile", text: """FROM harbor.od.com/${params.base_image}
				ADD ${params.target_dir}/project_dir /opt/project_dir"""
                sh "cd ${params.app_name}/${env.BUILD_NUMBER} && docker build --no-cache . -t harbor.od.com/${params.image_name}:${params.git_version}_${params.add_tag}  && docker push harbor.od.com/${params.image_name}:${params.git_version}_${params.add_tag}"	
            }
        }
		
		
		stage('deploy') {  //build image and push to registry
            steps {
				sh "helm --kubeconfig /var/jenkins_home/helm/kube_config upgrade dubbo-demo-service -n app --set image.tag=${params.git_version}_${params.add_tag} --install /var/jenkins_home/helm/dubbo-demo-service"
            }
        }
		
    }
}

1.4 配置jenkins参数的值

1.5 新建harbor项目目录

1.6 查看流水线运行状态

1.7 查看harbor仓库

1.8 查看k8s 中是否自动部署成功

1.9 在zookeeper上查看是否注册成功

/opt/zookeeper-3.4.14/bin/zkCli.sh
ls /

2. dubbo-monitor工具

源码： https://github.com/Jeromefromcn/dubbo-monitor

mkdir /root/dubbo-monitor
cd /root/dubbo-monitor
git clone https://gitee.com/chen1219/dubbo-monitor.git

2.1 修改如下配置

192.168.31.37 上操作

/root/dubbo-monitor/dubbo-monitor/dubbo-monitor-simple/conf/dubbo_origin.properties

dubbo.application.name=dubbo-monitor
dubbo.application.owner=Java_winner
dubbo.registry.address=zookeeper://zk1.od.com:2181?backup=zk2.od.com:2181,zk3.od.com:2181
dubbo.protocol.port=20880
dubbo.jetty.port=8080
dubbo.jetty.directory=/dubbo-monitor-simple/monitor
dubbo.charts.directory=/dubbo-monitor-simple/charts
dubbo.statistics.directory=/dubbo-monitor-simple/statistics

说明 ： dubbo.registry.address / dubbo.protocol.port 参数 参考代码https://gitee.com/chen1219_1/dashboard/projects/chen1219_1/dubbo-demo-service/blob/master/dubbo-server/src/main/java/config.properties

/root/dubbo-monitor/dubbo-monitor/dubbo-monitor-simple/bin/start.sh

#!/bin/bash
sed -e "s/{ZOOKEEPER_ADDRESS}/$ZOOKEEPER_ADDRESS/g" /dubbo-monitor-simple/conf/dubbo_origin.properties > /dubbo-monitor-simple/conf/dubbo.properties
cd `dirname $0`
BIN_DIR=`pwd`
cd ..
DEPLOY_DIR=`pwd`
CONF_DIR=$DEPLOY_DIR/conf

SERVER_NAME=`sed '/dubbo.application.name/!d;s/.*=//' conf/dubbo.properties | tr -d '
'`
SERVER_PROTOCOL=`sed '/dubbo.protocol.name/!d;s/.*=//' conf/dubbo.properties | tr -d '
'`
SERVER_PORT=`sed '/dubbo.protocol.port/!d;s/.*=//' conf/dubbo.properties | tr -d '
'`
LOGS_FILE=`sed '/dubbo.log4j.file/!d;s/.*=//' conf/dubbo.properties | tr -d '
'`

if [ -z "$SERVER_NAME" ]; then
    SERVER_NAME=`hostname`
fi

PIDS=`ps -f | grep java | grep "$CONF_DIR" |awk '{print $2}'`
if [ -n "$PIDS" ]; then
    echo "ERROR: The $SERVER_NAME already started!"
    echo "PID: $PIDS"
    exit 1
fi

if [ -n "$SERVER_PORT" ]; then
    SERVER_PORT_COUNT=`netstat -tln | grep $SERVER_PORT | wc -l`
    if [ $SERVER_PORT_COUNT -gt 0 ]; then
        echo "ERROR: The $SERVER_NAME port $SERVER_PORT already used!"
        exit 1
    fi
fi

LOGS_DIR=""
if [ -n "$LOGS_FILE" ]; then
    LOGS_DIR=`dirname $LOGS_FILE`
else
    LOGS_DIR=$DEPLOY_DIR/logs
fi
if [ ! -d $LOGS_DIR ]; then
    mkdir $LOGS_DIR
fi
STDOUT_FILE=$LOGS_DIR/stdout.log

LIB_DIR=$DEPLOY_DIR/lib
LIB_JARS=`ls $LIB_DIR|grep .jar|awk '{print "'$LIB_DIR'/"$0}'|tr "
" ":"`

JAVA_OPTS=" -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true "
JAVA_DEBUG_OPTS=""
if [ "$1" = "debug" ]; then
    JAVA_DEBUG_OPTS=" -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=n "
fi
JAVA_JMX_OPTS=""
if [ "$1" = "jmx" ]; then
    JAVA_JMX_OPTS=" -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false "
fi
JAVA_MEM_OPTS=""
BITS=`java -version 2>&1 | grep -i 64-bit`
if [ -n "$BITS" ]; then
    JAVA_MEM_OPTS=" -server -Xmx128m -Xms128m -Xmn32m -XX:PermSize=16m -Xss256k -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:LargePageSizeInBytes=128m -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 "
else
    JAVA_MEM_OPTS=" -server -Xms128m -Xmx128m -XX:PermSize=16m -XX:SurvivorRatio=2 -XX:+UseParallelGC "
fi

echo -e "Starting the $SERVER_NAME ...c"
exec java $JAVA_OPTS $JAVA_MEM_OPTS $JAVA_DEBUG_OPTS $JAVA_JMX_OPTS -classpath $CONF_DIR:$LIB_JARS com.alibaba.dubbo.container.Main > $STDOUT_FILE 2>&1

scp -r /root/dubbo-monitor/dubbo-monitor/ 192.168.31.200:/data/dockerfile/

2.2 执行Dockerfile

192.168.31.200 上操作

cd /data/dockerfile/dubbo-monitor/
docker build . --no-cache -t harbor.od.com/infra/dubbo-monitor:latest
docker push harbor.od.com/infra/dubbo-monitor:latest

2.3 准备证书

mkdir https
cd https
openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=dubbo/0=dubbo"
kubectl create secret tls tls-secret --key tls.key --cert tls.crt

2.4 准备资源

• dubbo-monitor-deployment.yaml

kind: Deployment
apiVersion: apps/v1
metadata:
  name: dubbo-monitor
  namespace: infra
  labels:
    name: dubbo-monitor
spec:
  replicas: 1
  selector: 
    matchLabels:
      name: dubbo-monitor
  template:
    metadata:
      labels:
        app: dubbo-monitor
        name: dubbo-monitor
    spec:
      containers:
      - name: dubbo-monitor
        image: harbor.od.com/infra/dubbo-monitor:latest
        ports:
        - containerPort: 8080
          protocol: TCP
        - containerPort: 20880
          protocol: TCP
      imagePullSecrets: 
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext:
        runAsUser: 0

• dubbo-monitor-svc.yaml

kind: Service
apiVersion: v1
metadata:
  name: dubbo-monitor
  namespace: infra
spec:
  ports:
  - protocol: TCP
    port: 8080
    port: 8080
  selector:
    app: dubbo-monitor

• dubbo-monitor-ingress.yaml

kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  name: dubbo-monitor
  namespace: infra
  annotations:
    kubernetes.io/ingress.class: intranet4public
spec:
  tls:
    - hosts: 
      - dubbo-monitor.od.com
      secretName: tls-secret
  rules:
  - host: dubbo-monitor.od.com
    http:
      paths:
      - path: /
        backend:
          serviceName: dubbo-monitor
          servicePort: 8080

• dubbo-monitor-configmap.yaml

kind: ConfigMap
apiVersion: v1
metadata:
  name: dubbo-monitor-cm
  namespace: infra
data:
  dubbo.properties: |
     dubbo.container=log4j,spring,registry,jetty
     dubbo.application.name=dubbo-monitor
     dubbo.application.owner=alibaba
     dubbo.registry.address=zookeeper://zk1.od.com:2181?backup=zk2.od.com:2181,zk3.od.com:2181
     dubbo.protocol.port=20880
     dubbo.jetty.port=8080
     dubbo.jetty.directory=/dubbo-monitor-simple/monitor
     dubbo.charts.directory=/dubbo-monitor-simple/charts
     dubbo.statistics.directory=/dubbo-monitor-simple/statistics
     dubbo.log4j.file=logs/dubbo-monitor-simple.log
     dubbo.log4j.level=WARN 

2.5 部署资源

kubectl apply -f dubbo-monitor-deployment.yaml
kubectl apply -f dubbo-monitor-svc.yaml
kubectl apply -f dubbo-monitor-ingress.yaml
kubectl apply -f dubbo-monitor-configmap.yaml

2.6 解析ip地址并访问

C:WindowsSystem32driversetchosts

192.168.31.40 dubbo-monitor.od.com

浏览器输入https://dubbo-monitor.od.com/

3. 交付dubbo服务消费者

3.1 配置helm chart

helm create dubbo-demo-service

• values.yaml

image:
  repository: harbor.od.com/app/dubbo-demo-consumer 
  tag: master_20200815_2242

• template/dp.yaml

kind: Deployment
apiVersion: apps/v1
metadata:
  name: dubbo-demo-consumer
  namespace: app
  labels:
    name: dubbo-demo-consumer
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dubbo-demo-consumer
  template:
    metadata:
      labels:
        app: dubbo-demo-consumer
    spec:
      containers:
      - name: dubbo-demo-consumer
        image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
        ports:
        - containerPort: 8080
          protocol: TCP
        - containerPort: 20880
          protocol: TCP
        env:
        - name: JAR_BALL
          value: dubbo-client.jar
        imagePullPolicy: IfNotPresent
      imagePullSecrets:
      - name: harbor
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext:
        runAsUser: 0
      schedulerName: default-scheduler

• templates/svc.yaml

kind: Service
apiVersion: v1
metadata:
  name: dubbo-demo-consumer
  namespace: app
spec:
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080
  selector:
    app: dubbo-demo-consumer

• templates/ingress.yaml

kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  name: dubbo-demo-consumer
  namespace: app
  annotations:
    kubernetes.io/ingress.class: intranet4public
spec:
  rules:
  - host: demo.od.com
    http:
      paths:
      - path: /
        backend:
          serviceName: dubbo-demo-consumer
          servicePort: 8080

将helm 的chart拷贝到 `192.168.31.200:/data/jenkins/jenkins_home/helm

scp -r /root/helm/dubbo-demo-consumer/ hdss200:/data/jenkins/jenkins_home/helm/

3.2 配置参数

变量名和dubbo-demo的一样， 变量的默认值有所改变

app_name: dubbo-demo-consumer
image_name: app/dubbo-demo-consumer
git_repo: git@gitee.com:chen1219_1/dubbo-demo-web.git
git_version: master
add_tag: 20200815_2242
mvn_dir: ./
target_dir: ./dubbo-client/target
mvn_cmd: mvn clean package -Dmaven.test.skip=true
base_image: 
	base/jre8:8u112
	base/jre7:7u80
maven: 
	3.6.3-8u242
	3.6.1-8u232

3.3 配置编译和部署命令

pipeline {
    agent any

    stages {
        stage('pull') {  //get project code from repo
            steps {
                sh "git clone ${params.git_repo} ${params.app_name}/${env.BUILD_NUMBER} && cd ${params.app_name}/${env.BUILD_NUMBER} && git checkout ${params.git_version}"	
            }
        }
		
		stage('build') {  //exec mvn cmd
            steps {
                sh "cd ${params.app_name}/${env.BUILD_NUMBER} && /var/jenkins_home/maven-${params.maven}/bin/${params.mvn_cmd}"	
				sh "cd ${params.app_name}/${env.BUILD_NUMBER} && cd ${params.target_dir} && mkdir project_dir && mv *.jar ./project_dir"
				writeFile file: "${params.app_name}/${env.BUILD_NUMBER}/Dockerfile", text: """FROM harbor.od.com/${params.base_image}
				ADD ${params.target_dir}/project_dir /opt/project_dir"""
                sh "cd ${params.app_name}/${env.BUILD_NUMBER} && docker build --no-cache . -t harbor.od.com/${params.image_name}:${params.git_version}_${params.add_tag}  && docker push harbor.od.com/${params.image_name}:${params.git_version}_${params.add_tag}"	
            }
        }
		
		
 		stage('deploy') {  //build image and push to registry
             steps {
 				sh "helm --kubeconfig /var/jenkins_home/helm/kube_config upgrade dubbo-demo-consumer -n app --set image.tag=${params.git_version}_${params.add_tag} --install /var/jenkins_home/helm/dubbo-demo-consumer"
             }
         }
		
    }
}

3.4 执行流水线

3.5 解析ip地址并访问

C:WindowsSystem32driversetchosts

192.168.31.40 dubbo-monitor.od.com

浏览器输入http://demo.od.com/

直接访问是访问失败的， 浏览器中需要传参数 http://demo.od.com/hello

• hello 后面的输出取决于name变量的赋值， 比如说 http://demo.od.com/hello?name=world