k8s 压测工具之perf-test

1、安装go环境

下载go

配置环境变量

export GO_HOME=/usr/local/go
export PATH=$GO_HOME/bin:$PATH

2、下载perftest

https://github.com/kubernetes/perf-tests/

3、解压后进入clusterloader2目录

cd perf-tests-master/clusterloader2/

4、运行时可以直接go运行，或者编译后

go run cmd/clusterloader.go --kubeconfig=kubeConfig.yaml --testconfig=config.yaml

或先编译，编译会生成clusterloader

./run-e2e.sh --testconfig=config.yaml
./clusterloader --kubeconfig=kubeConfig.yaml --testconfig=config.yaml

5、配置待测试文件

pod形式

1、pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: {{.Name}}
#  name: nginx
  labels:
    group: {{.Group}}
spec:
  containers:
  - image: nginx:1.12
    name: nginx

2、测试文件 config-pod.yaml

{{$namespaces := 1}}
{{$podPerNamespace := 2}}
{{$totalPods := 2}}  #pod数量

name: pod-test
automanagedNamespaces: {{$namespaces}}
tuningSets:
- name: Sequence
  parallelismLimitedLoad:
    parallelismLimit: 1
steps:
# start measurements
- measurements:
  - Identifier: PodStartupLatency
    Method: PodStartupLatency
    Params:
      action: start
      labelSelector: group = latency
- measurements:
  - Identifier: SchedulingMetrics
    Method: SchedulingMetrics
    Params:
      action: reset
      labelSelector: group = latency
- measurements:
  - Identifier: SchedulingThroughput
    Method: SchedulingThroughput
    Params:
      action: start
# create pods
- phases:
  - namespaceRange:
      min: 1
      max: {{$namespaces}}
    replicasPerNamespace: {{$podPerNamespace}}
    tuningSet: Sequence
    objectBundle:
    - basename: latency-pod
      objectTemplatePath: pod.yaml
      templateFillMap:
        Group: latency
        AppName: latency-pod
- measurements:
  - Identifier: SchedulingThroughput
    Method: SchedulingThroughput
    Params:
      action: gather
- measurements:
  - Identifier: WaitForRunningPods
    Method: WaitForRunningPods
    Params:
      desiredPodCount: {{$totalPods}}
      labelSelector: group = latency
      timeout: 15m
 gather measurements
- measurements:
  - Identifier: PodStartupLatency
    Method: PodStartupLatency
    Params:
      action: gather
- measurements:
  - Identifier: SchedulingMetrics
    Method: SchedulingMetrics
    Params:
      action: gather

deploy形式

1、deploy文件 deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{.Name}}
  labels:
    group: {{.Group}}
spec:
  replicas: {{.Replicas}}
  selector:
    matchLabels:
      name: {{.Name}}
  template:
    metadata:
      labels:
        name: {{.Name}}
        group: {{.Group}}
    spec:
      containers:
      - name: {{.Name}}
        image: nginx:1.12

2、config-deployment-old.yaml

{{$namespaces := 1}}
{{$deploymentPerNamespace := 200}}

name: deployment-test
automanagedNamespaces: {{$namespaces}}
tuningSets:
- name: Sequence
  parallelismLimitedLoad:
    parallelismLimit: 1
steps:
# start measurements
- measurements:
  - Identifier: WaitForRunningDeployments
    Method: WaitForControlledPodsRunning
    Params:
      action: start
      apiVersion: apps/v1
      kind: Deployment
      labelSelector: group = latency
      operationTimeout: 15m
# create deployment
- phases:
  - namespaceRange:
      min: 1
      max: {{$namespaces}}
    replicasPerNamespace: {{$deploymentPerNamespace}}
    tuningSet: Sequence
    objectBundle:
    - basename: latency-deployment
      objectTemplatePath: deployment.yaml
      templateFillMap:
        Replicas: 3
        Group: latency
- measurements:
  - Identifier: WaitForRunningDeployments
    Method: WaitForControlledPodsRunning
    Params:
      action: gather
# create deployment
- phases:
  - namespaceRange:
      min: 1
      max: {{$namespaces}}
    replicasPerNamespace: {{$deploymentPerNamespace}}
    tuningSet: Sequence
    objectBundle:
    - basename: latency-deployment
      objectTemplatePath: deployment.yaml
      templateFillMap:
        Replicas: 5
        Group: latency
- measurements:
  - Identifier: WaitForRunningDeployments
    Method: WaitForControlledPodsRunning
    Params:
      action: gather

- phases:
  - namespaceRange:
      min: 1
      max: {{$namespaces}}
    replicasPerNamespace: {{$deploymentPerNamespace}}
    tuningSet: Sequence
    objectBundle:
    - basename: latency-deployment
      objectTemplatePath: deployment.yaml
      templateFillMap:
        Replicas: 0
        Group: latency
- measurements:
  - Identifier: WaitForRunningDeployments
    Method: WaitForControlledPodsRunning
    Params:
      action: gather

6、完整测试命令

样例：

./clusterloader --kubeconfig=/root/.kube/config  --alsologtostderr --provider=local --masterip=<集群masterIP> --master-internal-ip=<集群masterIP> --mastername=<集群master hostname> --report-dir=<测试结果生成目录> --testconfig=<测试文件>

实际命令

#测试pod
./clusterloader --kubeconfig=/root/.kube/config  --alsologtostderr --provider=local --masterip=192.168.7.7 --master-internal-ip=192.168.7.7 --mastername=master --report-dir=result --testconfig=config-pod.yaml

#测试deployment
./clusterloader --kubeconfig=/root/.kube/config  --alsologtostderr --provider=local --masterip=192.168.7.7 --master-internal-ip=192.168.7.7 --mastername=master --report-dir=result --testconfig=config-deployment.yaml

7、结果展示