交付dubbo微服务到k8s (一)

1. dubbo微服务架构

微服务提供者集群： 暴露服务的服务提供方

微服务消费者集群： 调用远程服务的服务消费方

注册中心： 微服务提供者集群注册， 微服务消费者集群订阅， 微服务消费者集群通过RPC协议调用微服务提供者集群

Monitor: 统计服务的调用次数和调用时间的监控中心

部署的基本流程为 git推送代码到jenkins, jenkins编译代码并打包镜像推送到镜像仓库， 然后通过kubectl或者helm操作k8s的YAML文件部署服务

用户访问的基本流程为用户访问ingress, ingress代理微服务消费者集群， ingress得到响应后， 再将数据返回给用户

2. 部署zookeeper集群

• Zookeeper是Dubbo微服务集群的注册中心
• zookeeper高可用机制和etcd集群一致
• 需要jdk环境

2.1 基础架构

主机名	角色	ip
hdss200.host.com	harbor,zk1	192.168.31.200
hdss12.host.com	zk2	192.168.31.12
hdss11.host.com	dns, zk3	192.168.31.11

2.2 安装jdk1.8

​ jdk下载地址： https://www.oracle.com/java/technologies/javase/javase-jdk8-downloads.html

• 解压文件到指定目录

mkdir -p /usr/java
tar -xf jdk-8u261-linux-x64.tar.gz -C /usr/java/
cd /usr/java/

• 加入环境变量

vim /etc/profile

export JAVA_HOME=/usr/java/jdk1.8.0_261
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/lib/tools.jar

source /etc/profile

• 查看java版本

java -version

2.3 安装zookeeper

​ zookeeper 下载地址： https://archive.apache.org/dist/zookeeper/

2.3.1 解压，配置

• 解压文件到指定位置

wget https://archive.apache.org/dist/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz
tar xf zookeeper-3.4.14.tar.gz -C /opt/

• 创建zookeeper 数据目录 和 日志目录

cd /opt/
mkdir -p /data/zookeeper/data /data/zookeeper/logs

• 编写配置文件

cp /opt/zookeeper-3.4.14/conf/zoo_sample.cfg /opt/zookeeper-3.4.14/conf/zoo.cfg
vim /opt/zookeeper-3.4.14/conf/zoo.cfg

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper/data
dataLogDir=/data/zookeeper/logs
clientPort=2181
server.1=zk1.od.com:2888:3888
server.2=zk2.od.com:2888:3888
server.3=zk3.od.com:2888:3888

2.3.2 myid

`192.168.31.200

echo 1 > /data/zookeeper/data/myid

192.168.31.12

echo 2 > /data/zookeeper/data/myid

192.168.31.11

echo 3 > /data/zookeeper/data/myid

2.3.3 做dns解析

vim /var/named/od.com.zone

$ORIGIN od.com.
$TTL 600    ; 10 minutes
@   IN SOA  dns.od.com.  dnsadmin.od.com.  (
                 2020072401   ; serial
                 10800      ; refresh (3 hours)
                 900        ; retry (15 minutes)
                 604800     ; expire (1 week)
                 86400      ; minimun (1 day)
                 )
            NS  dns.od.com.
$TTL 60 ; 1 minute
dns                A      192.168.31.11
harbor             A      192.168.31.200
zk1             A         192.168.31.200
zk2             A         192.168.31.12
zk3             A         192.168.31.11

systemctl restart named

• 检查dns

2.3.4 依次启动zookeeper

• 启动服务

/opt/zookeeper-3.4.14/bin/zkServer.sh start

• 检查端口

netstat -ntulp|grep 2181

• 检查zookeeper集群的主从状态

/opt/zookeeper-3.4.14/bin/zkServer.sh status

3. 部署jenkins

官网： https://www.jenkins.io/download/

dockerhub： https://hub.docker.com/r/jenkins/jenkins

3.1 准备镜像

docker pull jenkins/jenkins:2.190.3-centos
docker tag jenkins/jenkins:2.190.3-centos harbor.od.com/public/jenkins:2.190.3-centos
docker push harbor.od.com/public/jenkins:2.190.3-centos

建议使用 2.190.3版本， 2.235.3，2.235.4版本， 使用jenkins打包成的镜像， 会将底包里面的脚本修改， 导致服务起不来

3.2 自定义Dockerfile

• 创建目录

mkdir -p /data/dockerfile/jenkins

• 切换目录

cd /data/dockerfile/jenkins

• 创建密钥对

ssh-keygen -t rsa -b 2048 -C '1392263019@qq.com' -N "" -f /root/.ssh/id_rsa

• 准备dockerfile文件

cd /data/dockerfile/jenkins
cp /root/.docker/config.json .
cp /root/.ssh/id_rsa .
wget https://get.helm.sh/helm-v3.1.0-linux-amd64.tar.gz
tar -xf helm-v3.1.0-linux-amd64.tar.gz

vim Dockerfile

FROM harbor.od.com/public/jenkins:2.235.4-lts 
USER root
RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && echo 'Asia/Shanghai' >/etc/timezone
ADD linux-amd64/helm /usr/bin
ADD id_rsa /root/.ssh/id_rsa
ADD config.json /root/.docker/config.json
ADD get-docker.sh /get-docker.sh
RUN  cat /etc/os-release
RUN echo "   StrictHostKeyChecking no" >> /etc/ssh/ssh_config && /get-docker.sh --mirror Aliyun 

3.3 制作自定义镜像

docker build --no-cache . -t harbor.od.com/infra/jenkins4helm:2.190.3-centos

• 测试下镜像

  需要先把公钥放到gitee上

docker run --rm harbor.od.com/infra/jenkins4helm:2.190.3-centos ssh -T git@gitee.com

3.4 创建infra仓库

3.5 创建kubernetes 命名空间以及私有仓库infra的secret

kubectl create ns infra
kubectl create secret docker-registry harbor --docker-server=harbor.od.com --docker-username=admin --docker-password=Harbor12345 -n infra

3.6 推送镜像

docker push harbor.od.com/infra/jenkins4helm:2.190.3-centos

3.7 准备nfs共享存储

所有运算节点安装nfs-utils, 并启动服务

yum -y install nfs-utils

• 配置nfs服务

192.168.31.200

echo '/data/jenkins 192.168.31.0/24(rw,no_root_squash)' >> /etc/exports

• 启动nfs服务

systemctl start rpcbind  &&systemctl enable rpcbind
systemctl start nfs && systemctl enable nfs

• 查看nfs服务

exportfs -v

• 创建目录jenkins_home

mkdir /data/jenkins/jenkins_home/

3.8 准备资源

• dp.yaml

kind: Deployment
apiVersion: apps/v1
metadata:
  name: jenkins
  namespace: infra
  labels:
    name: jenkins
spec:
  replicas: 1
  selector:
    matchLabels:
      name: jenkins
  template:
    metadata:
      labels:
        app: jenkins
        name: jenkins
    spec:
      volumes:
      - name: data
        nfs:
          server: hdss200
          path: /data/jenkins/jenkins_home
      - name: docker
        hostPath:
           path: /run/docker.sock
      imagePullSecrets:
      - name: harbor
      containers:
      - name: jenkins
        image: harbor.od.com/infra/jenkins:2.235.3 
        securityContext:
          runAsUser: 0
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
          protocol: TCP
        env:
        - name: JAVA_OPTS
          value: -Xmx512m -Xms512m -Dhudson.security.csrf.GlobalCrumbIssuerConfiguration.DISABLE_CSRF_PROTECTION=true
        volumeMounts:
        - name: data
          mountPath: /var/jenkins_home
        - name: docker
          mountPath: /run/docker.sock

• svc.yaml

kind: Service
apiVersion: v1
metadata:
  name: jenkins
  namespace: infra
spec:
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
  selector:
    app: jenkins

• ingress.yaml

kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  name: jenkins
  namespace: infra
  annotations:
   kubernetes.io/ingress.class: intranet4public 
spec:
  rules:
  - host: jenkins.od.com
    http:
      paths:
      - path: /
        backend:
          serviceName: jenkins
          servicePort: 80

3.9 部署资源

kubectl apply -f dp.yaml
kubectl apply -f svc.yaml
kubectl apply -f ingress.yaml

• 查看资源

kubectl get all -n infra
kubectl get ing -n infra

3.10 解析域名

将域名 jenkins.od.com 设定了 kubernetes.io/ingress.class: intranet4public , 指定了ingress为 设定了参数为 intranet4public 的ingress, 所以在本机的C:WindowsSystem32driversetchosts将域名解析为 192.168.31.41 jenkins.od.com

3.11 浏览器访问

浏览器输入 http://jenkins.od.com/

3.12 初始配置jenkins

• 输入初始密码

cat secrets/initialAdminPassword 

• 将所有的插件都取消勾选

3.13 创建管理者用户

用户名 admin

密码 admin123

3.14 允许匿名用户有读权限

3.15 配置国内镜像源

192.168.31.200

cd /data/jenkins/jenkins_home/updates/
sed -i 's/updates.jenkins-ci.org/download/mirrors.tuna.tsinghua.edu.cn/jenkins/g' default.json
sed -i 's/www.google.com/www.baidu.com/g' default.json

执行完重启pod

3.16 安装插件 Blue Ocean

Manage Jenkins - Manage Plugins

3.17 验证jenkins pod

• 进入pod, 选择在pod中运行

• 查看用户， 时间， docker容器， 是否能登入harbor仓库, 是否能通过ssh连接gitee

4. maven安装

下载地址： https://archive.apache.org/dist/maven/maven-3/3.6.3/binaries/

maven官网： http://maven.apache.org/index.html

192.168.31.200

4.1 下载

wget https://archive.apache.org/dist/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz
tar -xf apache-maven-3.6.3-bin.tar.gz -C /data/jenkins/jenkins_home/
mv /data/jenkins/jenkins_home/apache-maven-3.6.3/ /data/jenkins/jenkins_home/maven-3.6.3-8u242

4.2 设置国内镜像源

/data/jenkins/jenkins_home/maven-3.6.3-8u242/conf/settings.xml

<mirror>
    <id>aliyunmaven</id>
    <mirrorOf>*</mirrorOf>
    <name>阿里云公共仓库</name>
    <url>https://maven.aliyun.com/repository/public</url>
</mirror>

5. 制作dubbo微服务的底包镜像

192.168.31.200

5.1 准备dockerfile用到的文件

• 拉取基础镜像 jre8:8u112

mkdir -p cd /data/dockerfile/jre8
docker pull docker.io/stanleyws/jre8:8u112
docker tag stanleyws/jre8:8u112 harbor.od.com/public/jre:8u112
docker push harbor.od.com/public/jre:8u112
cd /data/dockerfile/jre8

• jmx_javaagent-0.3.1.jar

wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.3.1/jmx_prometheus_javaagent-0.3.1.jar -O jmx_javaagent-0.3.1.jar

• config.yml

---
rules:
  - pattern: '.*'

• entrypoint.sh

#!/bin/bash
M_OPTS="-Duser.timezone=Asia/Shanghai -javaagent:/opt/prom/jmx_javaagent-0.3.1.jar=$(hostname -i):$(M_PORT:-"12346"):/opt/prom/config.yml"
C_OPTS=$(C_OPTS)
JAR_BALL=$(JAR_BALL)
exec java -jar $(M_OPTS) $(C_OPTS) $(JAR_BALL)

chmod +x entrypoint.sh

• 目录结构

/data/dockerfile/jre8/
├── config.yml
├── Dockerfile
├── entrypoint.sh
└── jmx_javaagent-0.3.1.jar

5.2 编写dockerfile

/data/dockerfile/jre8/Dockerfile

FROM harbor.od.com/public/jre:8u112
RUN /bin/cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && echo 'Asia/Shanghai >/etc/timezone'
ADD  config.yml /opt/prom/config.yml
ADD jmx_javaagent-0.3.1.jar /opt/prom
WORKDIR /opt/project_dir
ADD entrypoint.sh /entrypoint.sh
CMD ["/entrypoint.sh"]

5.3 制作镜像

docker build . -t harbor.od.com/base/jre8:8u112