KubeSphere：harbor+gitlab+nexus+springboot流水线部署

预备环境：kubesphere-v3.0，harbor，gitlab，nexus（这些环境这里不介绍安装，我这里harbor，gitlab，nexus都是单独安装的，没有部署在k8s集群中）

且kubesphere已经创建好企业空间和DevOps工程。

环境介绍

ip	端口	角色	软件
192.168.1.36		k8s-master	k8s-1.18相关，kubesphere-v.3.0
192.168.1.37		k8s-node1	k8s-1.18相关，kubesphere-v.3.0
192.168.1.38		k8s-node2	k8s-1.18相关，kubesphere-v.3.0
192.168.1.50	82	代码仓库	gitlab-12.9.9
192.168.1.52	85	镜像仓库	harbor-1.9.2
192.168.1.48	8081	maven仓库	nexus3.30

创建凭证

创建Harbor凭证

创建Gitlab凭证

创建kubeconfig凭证

导入项目

在github中fork下来kubesphere官方提供的demo：https://github.com/kubesphere/devops-java-sample

在github上生成个人token

在自己的gitlab中导入github的token,然后填入自己的token

导入刚才fork的项目到gitlab

导入后，我们只需要master分支就行了。其他的分支全部删除。

准备工作

配置流水线

创建流水线：我这里图省事，直接流水线取名叫dev

流水线配置如下：

上传java镜像到harbor

docker pull java:openjdk-8-jre-alpine
docker login -u admin -p Harbor12345 192.168.1.52:85
docker tag java:openjdk-8-jre-alpine 192.168.1.52:85/library/java:openjdk-8-jre-alpine
docker push 192.168.1.52:85/library/java:openjdk-8-jre-alpine

修改Dockerfile

修改Dockerfile-on-prem

FROM 192.168.1.52:85/library/java:openjdk-8-jre-alpine
WORKDIR /home
COPY target/*.jar /home
ENTRYPOINT java -jar *.jar

设置命名空间

修改文件：deploy/dev/devops-sample.yaml和deploy/dev/devops-sample-svc.yaml，把所有命名空间名称改为dev

修改文件：deploy/prod/devops-sample.yaml和deploy/prod/devops-sample-svc.yaml，把所有命名空间名称改为dev-prod

kubectl create namespace dev
kubectl create namespace dev-prod

修改Jenkinsfile

pipeline {
  agent {
    node {
      label 'maven'
    }
  }

    parameters {
        string(name:'TAG_NAME',defaultValue: '',description:'')
    }

    environment {
        HARBOR_CREDENTIAL_ID = 'harbor-id'
        GITLAB_CREDENTIAL_ID = 'gitlab-id'
        KUBECONFIG_CREDENTIAL_ID = 'demo-kubeconfig'
        REGISTRY = '192.168.1.52:85'
        HARBOR_NAMESPACE = 'library'
        GITLAB_ACCOUNT = 'root'
        APP_NAME = 'devops-java-sample'
    }

    stages {
        stage ('checkout scm') {
            steps {
                checkout(scm)
            }
        }

        stage ('unit test') {
            steps {
                container ('maven') {
                    sh 'mvn clean -gs `pwd`/configuration/settings.xml test'
                }
            }
        }

        stage ('build & push') {
            steps {
                container ('maven') {
                    sh 'mvn -Dmaven.test.skip=true -gs `pwd`/configuration/settings.xml clean package'
                    sh 'docker build -f Dockerfile-on-prem -t $REGISTRY/$HARBOR_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER .'
                    withCredentials([usernamePassword(passwordVariable : 'DOCKER_PASSWORD' ,usernameVariable : 'DOCKER_USERNAME' ,credentialsId : "$HARBOR_CREDENTIAL_ID" ,)]) {
                        sh 'echo "$DOCKER_PASSWORD" | docker login $REGISTRY -u "$DOCKER_USERNAME" --password-stdin'
                        sh 'docker push  $REGISTRY/$HARBOR_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER'
                    }
                }
            }
        }

        stage('push latest'){
           when{
             branch 'master'
           }
           steps{
                container ('maven') {
                  sh 'docker tag  $REGISTRY/$HARBOR_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER $REGISTRY/$HARBOR_NAMESPACE/$APP_NAME:latest '
                  sh 'docker push  $REGISTRY/$HARBOR_NAMESPACE/$APP_NAME:latest '
                }
           }
        }

        stage('deploy to dev') {
          when{
            branch 'master'
          }
          steps {
            input(id: 'deploy-to-dev', message: 'deploy to dev?')
            kubernetesDeploy(configs: 'deploy/dev/**', enableConfigSubstitution: true, kubeconfigId: "$KUBECONFIG_CREDENTIAL_ID")
          }
        }
        stage('push with tag'){
          when{
            expression{
              return params.TAG_NAME =~ /v.*/
            }
          }
          steps {
              container ('maven') {
                input(id: 'release-image-with-tag', message: 'release image with tag?')
                  withCredentials([usernamePassword(credentialsId: "$GITLAB_CREDENTIAL_ID", passwordVariable: 'GIT_PASSWORD', usernameVariable: 'GIT_USERNAME')]) {
                    sh 'git config --global user.email "kubesphere@yunify.com" '
                    sh 'git config --global user.name "kubesphere" '
                    sh 'git tag -a $TAG_NAME -m "$TAG_NAME" '
                    sh 'git push http://$GIT_USERNAME:$GIT_PASSWORD@192.168.1.50:82/$GITLAB_ACCOUNT/devops-java-sample.git --tags --ipv4'
                  }
                sh 'docker tag  $REGISTRY/$HARBOR_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER $REGISTRY/$HARBOR_NAMESPACE/$APP_NAME:$TAG_NAME '
                sh 'docker push  $REGISTRY/$HARBOR_NAMESPACE/$APP_NAME:$TAG_NAME '
          }
          }
        }
        stage('deploy to production') {
          when{
            expression{
              return params.TAG_NAME =~ /v.*/
            }
          }
          steps {
            input(id: 'deploy-to-production', message: 'deploy to production?')
            kubernetesDeploy(configs: 'deploy/prod/**', enableConfigSubstitution: true, kubeconfigId: "$KUBECONFIG_CREDENTIAL_ID")
          }
        }
    }
}

修改maven配置文件

configuration/settings.xml

<?xml version="1.0" encoding="UTF-8"?>
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
  <localRepository>${PWD}/artifacts/m2</localRepository>
  <pluginGroups>
  </pluginGroups>
  <proxies>
  </proxies>
  <servers>
    <server>
      <id>nexus</id>
      <username>admin</username>
      <password>1234</password>
    </server>
  </servers>
  <mirrors>
    <mirror>
      <id>nexus</id>
      <name>nexus repository</name>
      <url>http://192.168.1.48:8081/repository/group_repo/</url>
      <mirrorOf>central</mirrorOf>
    </mirror>
  </mirrors>
  <profiles>
    <profile>
      <id>jdk-1.8</id>
      <activation>
        <activeByDefault>true</activeByDefault>
        <jdk>1.8</jdk>
      </activation>
      <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <maven.compiler.compilerVersion>1.8</maven.compiler.compilerVersion>
      </properties>
    </profile>
  </profiles>
</settings>

修改pom文件

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <!-- Spring Boot 启动父依赖 -->
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.1.13.RELEASE</version>
    </parent>
    <packaging>jar</packaging>
    <groupId>io.kubesphere.devops</groupId>
     <artifactId>devops-sample</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>devops-sample :: HelloWorld Demo</name>
      <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
      </properties>
        <dependencies>
            <dependency>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-starter-web</artifactId>
            </dependency>
            <dependency>
                <groupId>junit</groupId>
                <artifactId>junit</artifactId>
                <version>4.13.1</version>
            </dependency>
        </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.jacoco</groupId>
                <artifactId>jacoco-maven-plugin</artifactId>
                <version>0.8.2</version>
                <configuration>
                <append>true</append>
                </configuration>
                <executions>
                <execution>
                    <id>agent-for-ut</id>
                    <goals>
                    <goal>prepare-agent</goal>
                    </goals>
                </execution>
                <execution>
                    <id>agent-for-it</id>
                    <goals>
                    <goal>prepare-agent-integration</goal>
                    </goals>
                </execution>
                <execution>
                    <id>jacoco-site</id>
                    <phase>verify</phase>
                    <goals>
                    <goal>report</goal>
                    </goals>
                </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <configuration>
                    <fork>true</fork>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.sonarsource.scanner.maven</groupId>
                <artifactId>sonar-maven-plugin</artifactId>
                <version>3.6.0.1398</version>
            </plugin>
        </plugins>
    </build>
</project>

测试

注意这里的tag_name不能和gitlab中的tag一样，否则会报错。

点击继续：

等待一段时间

kubectl get svc -n dev-prod
kubectl get svc -n dev

kubectl get pods -n dev-prod
kubectl get pods -n dev

访问30861和30961端口（部署成功）