基于k8s的jenkins安装

一、集群安装Jenkins

1、准备jenkins镜像文件

docker pull jenkins/jenkins:2.204.1
docker tag a3f949e5ebfd harbor地址/infra/jenkins:v2.204.1
docker push harbor地址/infra/jenkins:v2.204.1

2、YAML资源文件

创建一个文件夹 在这个文件下创建3个 yaml 文件 pvc.yaml rbac.yaml jenkins.yaml

Jenkins master的安装，需要将数据持久化。可以利用NFS作为存储资源，创建PVC对象来挂载。PV/PVC配置文件pvc.yaml如下：

apiVersion: v1
kind: PersistentVolume
metadata:
  name: opspv
spec:
  capacity:
    storage: 20Gi
  accessModes:
  - ReadWriteMany
  persistentVolumeReclaimPolicy: Delete
  nfs:
    server: 172.29.207.78          #注意：此处为NFS服务器的地址
    path: /data/k8s

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: opspvc
  namespace: kube-ops
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 20Gi

同时，对于即将创建的Jenkins master资源对象，需要授予其一些权限，比如增删改查等。相应的配置文件rbac.yaml如下：

apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins
  namespace: kube-ops

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: jenkins
rules:
  - apiGroups: ["extensions", "apps"]
    resources: ["deployments"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get","list","watch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get"]

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: jenkins
  namespace: kube-ops
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: jenkins
subjects:
  - kind: ServiceAccount
    name: jenkins
    namespace: kube-ops

然后，基于jenkins/jenkins:lts 镜像创建jenkins master镜像，配置文件jenkins.yaml 如下：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins
  namespace: kube-ops
spec:
  selector:
    matchLabels:
      app: jenkins
  template:
    metadata:
      labels:
        app: jenkins
    spec:
      terminationGracePeriodSeconds: 10
      serviceAccount: jenkins
      containers:
      - name: jenkins
        image: harbor.zzx/infra/jenkins:v2.204.1
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
          name: web
          protocol: TCP
        - containerPort: 50000
          name: agent
          protocol: TCP
        resources:
          limits:
            cpu: 1000m
            memory: 1Gi
          requests:
            cpu: 500m
            memory: 512Mi
        livenessProbe:
          httpGet:
            path: /login
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 5
          failureThreshold: 12
        readinessProbe:
          httpGet:
            path: /login
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 5
          failureThreshold: 12
        volumeMounts:
        - name: jenkinshome
          subPath: jenkins
          mountPath: /var/jenkins_home
        env:
        - name: LIMITS_MEMORY
          valueFrom:
            resourceFieldRef:
              resource: limits.memory
              divisor: 1Mi
        - name: JAVA_OPTS
          value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -Duser.timezone=Asia/Shanghai
      imagePullSecrets:
      - name: docker-harbor-registry
      securityContext:
        fsGroup: 1000
      volumes:
      - name: jenkinshome
        persistentVolumeClaim:
          claimName: opspvc

---
apiVersion: v1
kind: Service
metadata:
  name: jenkins
  namespace: kube-ops
  labels:
    app: jenkins
spec:
  selector:
    app: jenkins
  type: NodePort
  ports:
  - name: web
    port: 8080
    targetPort: web
    nodePort: 30080    #注意：外部访问端口
  - name: agent
    port: 50000
    targetPort: agent

最后，在目录内执行命令如下：

kubectl create namespace kube-ops
kubectl create -f ./

此时，我们通过命令kubectl -n kube-ops get pod可以查看到jenkins已成功创建。

二、jenkins动态配置slave

1、初始化jenkins配置

浏览器打开masterIP:30080

其中的管理员密码，我们既可以进入容器内对应的目录查看，也可以在nfs server服务器上执行命令：
cat /data/k8s/jenkins/secrets/initialAdminPassword 来查看；然后选择安装推荐的插件，如下：

然后添加管理员账户即可进入Jenkins界面。

2、配置jenkins slave

(1) 安装Kubernetes插件

进入 Manage Jenkins—>Manage Plugins—>可选插件(Available)—>Kubernetes plugin勾选，直接安装即可。

(2) 配置Kubernetes插件

点击Manage Jenkins—>Configure System—>云—>新增一个云—>Kubernetes，如下：

然后配置如下：
先注意 名称默认为kubernetes，然后 Kubernetes地址 填写https://kubernetes.default.svc.cluster.local，命名空间为kube-ops；接着点击右边的 连接测试 按钮，如果显示Connection test successful，表示Jenkins可以和Kubernetes集群正常通信了。最后，在Jenkins地址，填入：http://服务名.kube-ops.svc.cluster.local:8080，如下所示：

(3) 创建Pipeline动态构建测试

创建一个类型为Pipeline类型Job命名为test-slave，然后在Pipeline脚本填写下面一个简单的测试脚本

def label = "jenkins-slave-${UUID.randomUUID().toString()}"
podTemplate(label: label, cloud: 'kubernetes') {
    node(label) {
        stage('Run shell') {
            sh 'sleep 10s'
            sh 'echo hello world.'
        }
    }
}

(4) 点击构建流水线

可以看到在jenkins的namespaces下自动创建了对应的agent pod 相当于就是一个jenkins 的node 当任务执行完成这个pod会自动退出这个pod默认会去pull一个jenkins/jnlp-slave:x.xx-xx-alpine的镜像

[root@hdss7-21 ~]# kubectl get pods -n k8s-ops
NAME                                                             READY   STATUS    RESTARTS   AGE
jenkins-77b9c47874-qjgfd                                         1/1     Running   1          13h
jenkins-slave-c07daa7b-31ef-41ea-825e-05c9c721edad-sb7h6-lpgwv   1/1     Running   0          18s