helm v3 调试 k8s elasticsearch集群(5)

• 链接：helm v3部署es集群

上次我们已经通过helm部署了es的release，es集群是起来了，功能也正常，这次我们模拟测试，简单验证es的功能与性能。

测试主要分：

• 节点宕机
• 存储不足
• 待完善

本次测试，将master设replicas为3，data设置为3，client设置为2，因为只需调试时用kibana，设置1即可(可不用设置)。

调整es-client节点

# 停掉原有client的release
helm uninstall es-c -nelk
# 编辑副本为2
vim es-client.yaml
---
replicas: 1 -> replicas: 2
---
# 重新install release
helm install es-client -f es-client.yaml elastic/elasticsearch -nelk  --version 6.8.18 --debug > client.log

删除原来的的pvc,pv

# 删除data/master节点的pvc
kubectl delete  persistentvolumeclaim/helm-data-helm-data-0 -nelk
kubectl delete persistentvolumeclaim/helm-master-helm-master-0 -nelk

# 删除data/master节点的pv
kubectl delete pv/tets-pv -nelk
kubectl delete pv/tets-pv-d -nelk

新建pv，pvc，由于测试的master数量为3，data数量为3，所以应该设置6份pv， storageclass不变。

local-pv-d-0.yaml
local-pv-d-1.yaml
local-pv-d-2.yaml
local-pv-m-0.yaml
local-pv-m-1.yaml
local-pv-m-2.yaml

由于master节点主要用于负责集群间的管理工作，存储可以设置小些，data 节点，负责存储数据, 设置大些；client节点，负责代理 ElasticSearch Cluster 集群，负载均衡，不用设置存储卷。

master节点

# local-pv-m-0.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: test-pv-m-0
spec:
  capacity:
    storage: 10Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local: # local类型
    path: /mnt/m  # 节点上的具体路径
  nodeAffinity: # 这里就设置了节点亲和
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - node1 # 这里我们使用node01节点，该节点有/data/vol1路径


---
# local-pv-m-1.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: test-pv-m-1
spec:
  capacity:
    storage: 10Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local: # local类型
    path: /mnt/m  # 节点上的具体路径
  nodeAffinity: # 这里就设置了节点亲和
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - node2 # 这里我们使用node01节点，该节点有/data/vol1路径


---
# local-pvc-m-2.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: test-pv-m-2
spec:
  capacity:
    storage: 10Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local: # local类型
    path: /mnt/m  # 节点上的具体路径
  nodeAffinity: # 这里就设置了节点亲和
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - node3 # 这里我们使用node01节点，该节点有/data/vol1路径

---
# local-pvc-d-0.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: test-pv-d-0
spec:
  capacity:
    storage: 10Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local: # local类型
    path: /mnt/data  # 节点上的具体路径
  nodeAffinity: # 这里就设置了节点亲和
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - node1 # 这里我们使用node01节点，该节点有/data/vol1路径

---
# local-pvc-d-1.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: test-pv-d-1
spec:
  capacity:
    storage: 10Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local: # local类型
    path: /mnt/data  # 节点上的具体路径
  nodeAffinity: # 这里就设置了节点亲和
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - node2 # 这里我们使用node01节点，该节点有/data/vol1路径

---
# local-pvc-d-2.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: test-pv-d-2
spec:
  capacity:
    storage: 10Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local: # local类型
    path: /mnt/data  # 节点上的具体路径
  nodeAffinity: # 这里就设置了节点亲和
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - node3 # 这里我们使用node01节点，该节点有/data/vol1路径

创建6个pv，建议按照下列顺序创建，否则创建一堆pv，pv处于游离状态，在install release的时候，bound的pv可能乱序，即master和data乱序

# 先创建master的pv, 然后install release
kubectl apply -f  local-pv-m-0.yaml -nelk
kubectl apply -f  local-pv-m-1.yaml -nelk
kubectl apply -f  local-pv-m-2.yaml -nelk
# master install release后，再创建data的pv，然后install data的release
kubectl apply -f  local-pv-d-0.yaml -nelk
kubectl apply -f  local-pv-d-1.yaml -nelk
kubectl apply -f  local-pv-d-2.yaml -nelk

查看pv创建状态：
kubectl get pv,pvc -nelk

可以看到pv全部处于bound状态。

下面更改es-master.yaml/es-data.yaml，修改对应的存储及replicas 1->3:

# es-master.yaml
---
# 集群名称
clusterName: "helm"
# 节点所属群组
nodeGroup: "master"
# Master 节点的服务地址，这里是Master,不需要
masterService: ""
# 节点类型：
roles:
  master: "true"
  ingest: "false"
  data: "false"
# 节点数量，做为 Master 节点，数量必须是 node >=3 and node mod 2 == 1
replicas: 3 # master eligible nodes，master的候选节点为3
minimumMasterNodes: 2 # 为防止脑裂，建议设置： 候选主节点数/2+1，即 3/2 + 1 = 2
esMajorVersion: ""
esConfig:
  # elasticsearch.yml 的配置，主要是数据传输和监控的开关及证书配置
  elasticsearch.yml: |
    xpack:
      security:
        enabled: true
        transport:
          ssl:
            enabled: true
            verification_mode: certificate
            keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
            truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
      monitoring:
        collection:
          enabled: true
# 设置 ES 集群的 elastic 账号密码为变量
extraEnvs:
  - name: ELASTIC_USERNAME
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: username
  - name: ELASTIC_PASSWORD
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: password
envFrom: []
# 挂载证书位置
secretMounts:
  - name: elastic-certificates
    secretName: elastic-certificates
    path: /usr/share/elasticsearch/config/certs
# 镜像拉取来源，我对镜像做了一些简单的修改，故放置于自建的 harbor 里。
image: "docker.elastic.co/elasticsearch/elasticsearch"
imageTag: "6.8.18"
imagePullPolicy: "IfNotPresent"
imagePullSecrets:
  - name: registry-secret
podAnnotations: {}
labels: {}
# ES 的 JVM 内存
esJavaOpts: "-Xmx512m -Xms512m" # 机器内存小时，不要设置太大，另外master节点功能主要是协调器，如索引的创建、集群管理等
# ES 运行所需的资源
resources:
  requests:
    cpu: "500m"
    memory: "1Gi"
  limits:
    cpu: "500m"
    memory: "1Gi"
initResources: {}
sidecarResources: {}
# ES 的服务 IP,如果没有设置这个，服务有可能无法启动。
networkHost: "0.0.0.0"
# ES 的存储配置
volumeClaimTemplate:
  storageClassName: "local-storage" # 本地存储
  accessModes: [ "ReadWriteOnce" ]
  resources:
    requests:
      storage: 10Gi
# PVC 开关
persistence:
  enabled: true
  labels:
    enabled: false
  annotations: {}
# rbac 暂未详细研究
rbac:
  create: false
  serviceAccountAnnotations: {}
  serviceAccountName: ""
# 镜像部署选择节点
# nodeSelector:
#   kubernetes.io/hostname: node1
# 容忍污点，如果 K8S 集群节点较少，需要在 Master 节点部署，需要使用此项
tolerations:
  - operator: "Exists"


-------------es-data
# es-data.yaml
---
# 集群名称，必须和 Master 节点的集群名称保持一致
clusterName: "helm"
# 节点类型
nodeGroup: "data"
# Master 节点服务名称
masterService: "helm-master"
# 节点权限，为 True 的是提供相关服务，Data 节点不需要 Master 权限
roles:
  master: "false"
  ingest: "true"
  data: "true"
# 节点数量
replicas: 3 # 作为数据节点，设置多些，一般根据主分片及副本数确定max的扩展节点数，max = num_shards + num_replicas * num_shards
esMajorVersion: "6"
esConfig:
# elasticsearch.yml 配置，同 Master 节点配置
  elasticsearch.yml: |
    xpack:
      security:
        enabled: true
        transport:
          ssl:
            enabled: true
            verification_mode: certificate
            keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
            truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
      monitoring:
        collection:
          enabled: true
extraEnvs:
# 同 Master 节点配置
  - name: ELASTIC_USERNAME
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: username
  - name: ELASTIC_PASSWORD
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: password
envFrom: []
secretMounts:
# 证书挂载，同 Master 节点配置
  - name: elastic-certificates
    secretName: elastic-certificates
    path: /usr/share/elasticsearch/config/certs
image: "docker.elastic.co/elasticsearch/elasticsearch"
imageTag: "6.8.18"
imagePullPolicy: "IfNotPresent"
imagePullSecrets:
  - name: registry-secret
podAnnotations: {}
labels: {}
# ES节点的 JVM 内存分配，根据实际情况进行增加
esJavaOpts: "-Xmx512m -Xms512m"

# ES 运行所需的资源
resources:
  requests:
    cpu: "1000m"
    memory: "1Gi"
  limits:
    cpu: "1000m"
    memory: "1Gi"
initResources: {}
sidecarResources: {}
# ES 的服务 IP,如果没有设置这个，服务有可能无法启动。
networkHost: "0.0.0.0"

# ES 数据存储
volumeClaimTemplate:
  storageClassName: "local-storage"
  accessModes: [ "ReadWriteOnce" ]
  resources:
    requests:
      storage: 10Gi

# PVC 开关
persistence:
  enabled: true
  labels:
    enabled: false
  annotations: {}

# rbac 暂未详细研究
rbac:
  create: false
  serviceAccountAnnotations: {}
  serviceAccountName: ""
# 镜像部署选择节点
# nodeSelector:
#   elk-rolse: data
# 容忍污点，如果 K8S 集群节点较少，需要在 Master 节点部署，需要使用此项
tolerations:
  - operator: "Exists"

helm安装master/data的release：

helm install es-m -nelk -f es-master.yaml elastic/elasticsearch --version 6.8.18 --debug > master.log
helm install es-d -nelk -f es-data.yaml elastic/elasticsearch --version 6.8.18 --debug > data.log

查看po的运行状况
等待一会，待集群自动发现后，全部处于ready状态：

查看svc，pv，pvc的状态
可以看到svc全部成功，pv全部bound, 随着es-roleX.yaml的release的创建，pvc自动创建，也全部bound状态

es的discovery.zen使用单播unicast机制来发现处于同一集群名下的es实例，cluster.name，即通过绑定集群主节点的headless的svc来发现同一集群下的机器，具体见如下：

root@node91:20211104 10:08:10:es-values# grep -C 5  headless master.log
---
# Source: elasticsearch/templates/service.yaml
kind: Service
apiVersion: v1
metadata:
  name: helm-master-headless
  labels:
    heritage: "Helm"
    release: "es-m"
    chart: "elasticsearch"
    app: "helm-master"
--
    chart: "elasticsearch"
    app: "helm-master"
  annotations:
    esMajorVersion: "6"
spec:
  serviceName: helm-master-headless
  selector:
    matchLabels:
      app: "helm-master"
  replicas: 3
  podManagementPolicy: Parallel
--
              fieldRef:
                fieldPath: metadata.name
          - name: discovery.zen.minimum_master_nodes
            value: "2"
          - name: discovery.zen.ping.unicast.hosts
            value: "helm-master-headless"
          - name: cluster.name
            value: "helm"
          - name: network.host
            value: "0.0.0.0"
          - name: ES_JAVA_OPTS
root@node91:20211104 10:08:17:es-values# grep -C 5  headless data.log
---
# Source: elasticsearch/templates/service.yaml
kind: Service
apiVersion: v1
metadata:
  name: helm-data-headless
  labels:
    heritage: "Helm"
    release: "es-d"
    chart: "elasticsearch"
    app: "helm-data"
--
    chart: "elasticsearch"
    app: "helm-data"
  annotations:
    esMajorVersion: "6"
spec:
  serviceName: helm-data-headless
  selector:
    matchLabels:
      app: "helm-data"
  replicas: 3
  podManagementPolicy: Parallel
--
          - name: node.name
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: discovery.zen.ping.unicast.hosts
            value: "helm-master-headless"
          - name: cluster.name
            value: "helm"
          - name: network.host
            value: "0.0.0.0"
          - name: ES_JAVA_OPTS
root@node91:20211104 10:08:40:es-values# grep -C 5  headless client.log
---
# Source: elasticsearch/templates/service.yaml
kind: Service
apiVersion: v1
metadata:
  name: helm-client-headless
  labels:
    heritage: "Helm"
    release: "es-c"
    chart: "elasticsearch"
    app: "helm-client"
--
    chart: "elasticsearch"
    app: "helm-client"
  annotations:
    esMajorVersion: "6"
spec:
  serviceName: helm-client-headless
  selector:
    matchLabels:
      app: "helm-client"
  replicas: 2
  podManagementPolicy: Parallel
--
          - name: node.name
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: discovery.zen.ping.unicast.hosts
            value: "helm-master-headless" 
          - name: cluster.name
            value: "helm"
          - name: network.host
            value: "0.0.0.0"
          - name: ES_JAVA_OPTS

查看集群健康状况：
任意节点下curl, 因为es是去中心化的(peer to peer, p2p)，访问任意一个节点，返回结果都是一样的

# curl -u elastic:elastic123456 http://172.22.118.91:30200/_cluster/health?pretty
---
{
  "cluster_name" : "helm",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 8, # 3-master，3-data，2-client
  "number_of_data_nodes" : 3, # 3-data
  "active_primary_shards" : 12,
  "active_shards" : 24,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}
---

在kibana上查看节点的情况：

可看到master-2节点是集群的主节点。

进一步查看索引indices的状态：

新建一批index，设置其shards和replicas的数目，由于data节点测试的节点数为3个，建议主分片的shards数量也设置为3个，另外副本分片replicas的数量设置大于等于1，也不要过大，否则导致副本分片没有分配到节点上，总之考虑高可用的情况下，保证分片都可被分配到节点上，应该设置每个节点都有主分片，副本分片最好大于等于1，性能随着副本数增加成开口下下抛物线型。

health-tips:

green: 分片都分配到节点上，集群可用
yellow: 有分片没有分配到节点上，但可以使用，分片数量设置不合理
red: 集群不可用

遇到问题

1.helm install release时，etcdserver request请求超时
报错：

• https://github.com/helm/helm/issues/7992
• https://blog.csdn.net/textdemo123/article/details/96460680

2.重启install release集群时，需要清理挂载path