使用Kubernetes、K3s和Traefik2进行本地开发

作者简介

Vyacheslav，拥有运维和项目管理经验的软件工程师

这篇文章将承接我此前搭建的本地Docker开发环境，具体步骤已经放在在以下网址：

https://github.com/Voronenko/traefik2-compose-template

除了经典的docker化的项目之外，我还有其他的Kubernetes项目。尽管Kubernetes已经成为容器编排的事实标准，但是不得不承认Kubernetes是一个既消耗资源又消耗金钱的平台。由于我并不经常需要外部集群，因此我使用轻量级K3s发行版来进行Kubernetes本地开发。

K3s是为IoT和边缘计算而构建的经过认证的Kubernetes发行版之一，还能够按产品规模部署到VM。

我使用K3s的方式是这样的：在我的工作笔记本上本地安装K3s，尽管有时我需要在本地部署较重的测试工作负载，为此，我准备了两个神器——两个运行ESXi的外部Intel NUCs。

默认情况下，K3s安装Traefik 1.x作为ingress，如果你对此十分满意，那么无需往下继续阅读了。

在我的场景中，我同时会牵涉到好几个项目，特别是经典的docker和docker swarm，因此我经常遇到在独立模式下部署Traefik的情况。

因此，本文其余部分将深入介绍如何将外部traefik2配置为K3s集群的ingress。

安装Kubernetes K3s系列集群

你可以按照常规方式使用命令curl -sfL https://get.k3s.io | sh -安装K3s，或者你可以使用轻量实用程序k3sup安装（https://github.com/alexellis/k3sup）。具体步骤在之前的文章介绍过。

与我们的设置不同的是，我们使用命令--no-deploy traefik专门安装了不带traefik组件的K3s。

export CLUSTER_MASTER=192.168.3.100
export CLUSTER_DEPLOY_USER=slavko
k3sup install --ip $CLUSTER_MASTER --user $CLUSTER_DEPLOY_USER --k3s-extra-args '--no-deploy traefik'

执行后，你将获得使用kubectl所需的连接详细信息。安装K3s后，你可以快速检查是否可以看到节点。

# Test your cluster with - export path to k3s cluster kubeconfig:
export KUBECONFIG=/home/slavko/kubeconfig
kubectl get node -o wide

注：这里没有固定的安装模式，你甚至可以使用docker-compose自行启动它。

server:
  image: rancher/k3s:v0.8.0
  command: server --disable-agent --no-deploy traefik
  environment:
    - K3S_CLUSTER_SECRET=somethingtotallyrandom
    - K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml
    - K3S_KUBECONFIG_MODE=666
  volumes:
    # k3s will generate a kubeconfig.yaml in this directory. This volume is mounted
    # on your host, so you can then 'export KUBECONFIG=/somewhere/on/your/host/out/kubeconfig.yaml',
    # in order for your kubectl commands to work.
    - /somewhere/on/your/host/out:/output
    # This directory is where you put all the (yaml) configuration files of
    # the Kubernetes resources.
    - /somewhere/on/your/host/in:/var/lib/rancher/k3s/server/manifests
  ports:
    - 6443:6443

node:
  image: rancher/k3s:v0.8.0
  privileged: true
  links:
    - server
  environment:
    - K3S_URL=https://server:6443
    - K3S_CLUSTER_SECRET=somethingtotallyrandom
  volumes:
    # this is where you would place a alternative traefik image (saved as a .tar file with
    # 'docker save'), if you want to use it, instead of the traefik:v2.0 image.
    - /sowewhere/on/your/host/custom-image:/var/lib/rancher/k3s/agent/images

配置Traefik 2，与Kubernetes一起使用

在文章开头提到的链接中，我已经在我的系统中安装了Traefik 2，并根据该链接内容，服务于一些需求。现在是时候配置Traefik 2 Kubernetes后端了。

Traefik 2使用CRD（自定义资源定义）来完成这一点。定义的最新示例可以在以下链接中找到，但这些示例仅适用于Traefik 2也作为Kubernetes工作负载的一部分执行的情况：

https://docs.traefik.io/reference/dynamic-configuration/kubernetes-crd/

对于外部Traefik 2，我们仅需要以下描述的定义子集。

我们引入一系列自定义资源定义，以允许我们来描述我们的Kubernetes服务将会如何暴露到外部，traefik-crd.yaml：

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutes.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRoute
    plural: ingressroutes
    singular: ingressroute
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: ingressroutetcps.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: IngressRouteTCP
    plural: ingressroutetcps
    singular: ingressroutetcp
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: middlewares.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: Middleware
    plural: middlewares
    singular: middleware
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: tlsoptions.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TLSOption
    plural: tlsoptions
    singular: tlsoption
  scope: Namespaced

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: traefikservices.traefik.containo.us

spec:
  group: traefik.containo.us
  version: v1alpha1
  names:
    kind: TraefikService
    plural: traefikservices
    singular: traefikservice
  scope: Namespaced  

同时，我们需要集群角色traefik-ingress-controller，以提供对服务、端点和secret的只读访问权限以及自定义的traefik.containo.us组，traefik-clusterrole.yaml：

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller

rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - traefik.containo.us
    resources:
      - middlewares
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - traefik.containo.us
    resources:
      - ingressroutes
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - traefik.containo.us
    resources:
      - ingressroutetcps
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - traefik.containo.us
    resources:
      - tlsoptions
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - traefik.containo.us
    resources:
      - traefikservices
    verbs:
      - get
      - list
      - watch

最后，我们需要系统服务账号traefik-ingress-controller与之前创建的集群角色traefik-ingress-controller相关联。

---
kind: ServiceAccount
apiVersion: v1
metadata:
  namespace: kube-system
  name: traefik-ingress-controller

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller

roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
  - kind: ServiceAccount
    name: traefik-ingress-controller
    namespace: kube-system

我们应用以上资源之后：

apply:
  kubectl apply -f traefik-crd.yaml
  kubectl apply -f traefik-clusterrole.yaml
  kubectl apply -f traefik-service-account.yaml

我们已经准备好开始调整Traefik 2

将Traefik 2指向K3s集群

根据Traefik文档的建议，当Traefik部署到Kubernetes中时，它将读取环境变量KUBERNETES_SERVICE_HOST和KUBERNETES_SERVICE_PORT或KUBECONFIG来构造端点。

在/var/run/secrets/kubernetes.io/serviceaccount/token中查找访问token，而SSL CA证书将在/var/run/secrets/kubernetes.io/serviceaccount/ca.crt.中查找。当部署到Kubernetes内部时，两者都会自动提供挂载。

当无法找到环境变量时，Traefik会尝试使用external-cluster客户端连接到Kubernetes API server。这一情况下，需要设置endpoint。具体来说，可以将其设置为kubectl代理使用的URL，以使用相关的kubeconfig授予的身份验证和授权连接到Kubernetes集群。

Traefik 2可以使用任何受支持的配置类型来静态配置-toml、yaml或命令行交换。

[providers.kubernetesCRD]
  endpoint = "http://localhost:8080"
  token = "mytoken"

providers:
  kubernetesCRD:
    endpoint = "http://localhost:8080"
    token = "mytoken"
    # ...

--providers.kubernetescrd.endpoint=http://localhost:8080 
--providers.kubernetescrd.token=mytoken

第一次运行时，如果你在外部有Traefik，很有可能没有traefik-ingress-controller访问token来指定mytoken。那么，你需要执行以下命令：

# Check all possible clusters, as your .KUBECONFIG may have multiple contexts:
kubectl config view -o jsonpath='{"Cluster name	Server
"}{range .clusters[*]}{.name}{"	"}{.cluster.server}{"
"}{end}'

# Output kind of
# Alias tip: k config view -o jsonpath='{"Cluster name	Server
"}{range .clusters[*]}{.name}{"	"}{.cluster.server}{"
"}{end}'
# Cluster name  Server
# default  https://127.0.0.1:6443

# You are interested in: "default", if you did not name it differently

# Select name of cluster you want to interact with from above output:
export CLUSTER_NAME="default"

# Point to the API server referring the cluster name
export APISERVER=$(kubectl config view -o jsonpath="{.clusters[?(@.name=="$CLUSTER_NAME")].cluster.server}")
# usually https://127.0.0.1:6443

# Gets the token value
export TOKEN=$(kubectl get secrets -o jsonpath="{.items[?(@.metadata.annotations['kubernetes.io/service-account.name']=='traefik-ingress-controller')].data.token}" --namespace kube-system|base64 --decode)

# Explore the API with TOKEN

如果成功了，你应该收到以下响应：

{
  "kind": "APIVersions",
  "versions": [
    "v1"
  ],
  "serverAddressByClientCIDRs": [
    {
      "clientCIDR": "0.0.0.0/0",
      "serverAddress": "192.168.3.100:6443"
    }
  ]

以及一些事实，如token：

eyJhbGciOiJSUzI1NiIsImtpZCI6IjBUeTQyNm5nakVWbW5PaTRRbDhucGlPeWhlTHhxTXZjUDJsRmNacURjVnMifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJ0cmFlZmlrLWluZ3Jlc3MtY29udHJvbGxlci10b2tlbi12emM3diIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJ0cmFlZmlrLWluZ3Jlc3MtY29udHJvbGxlciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImQ5NTc3ZTkxLTdlNjQtNGMwNi1iZDgyLWNkZTk0OWM4MTI1MSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTp0cmFlZmlrLWluZ3Jlc3MtY29udHJvbGxlciJ9.Mk8EBS4soO8uX-uSnV3o4qZKR6Iw6bgeSmPhHbJ2fjuqFgLnLh4ggxa-N9AqmCsEWiYjSi5oKAu986UEC-_kGQh3xaCYsUwlkM8147fsnwCbomSeGIct14JztVL9F8JwoDH6T0BOEjn-J9uY8-fUKYL_Y7uTrilhFapuILPsj_bFfgIeOOapRD0XshKBQV9Qzg8URxyQyfzl68ilm1Q13h3jLj8CFE2RlgEUFk8TqYH4T4fhfpvV-gNdmKJGODsJDI1hOuWUtBaH_ce9w6woC9K88O3FLKVi7fbvlDFrFoJ2iVZbrRALPjoFN92VA7a6R3pXUbKebTI3aUJiXyfXRQ

根据上次响应的API server的外部地址：https://192.168.3.100:6443

同样，提供的token中没有任何特殊之处：这是JWT的token，你可以使用https://jwt.io/#debugger-io，检查它的内容。

{
  "alg": "RS256",
  "kid": "0Ty426ngjEVmnOi4Ql8npiOyheLxqMvcP2lFcZqDcVs"
}
{
  "iss": "kubernetes/serviceaccount",
  "kubernetes.io/serviceaccount/namespace": "kube-system",
  "kubernetes.io/serviceaccount/secret.name": "traefik-ingress-controller-token-vzc7v",
  "kubernetes.io/serviceaccount/service-account.name": "traefik-ingress-controller",
  "kubernetes.io/serviceaccount/service-account.uid": "d9577e91-7e64-4c06-bd82-cde949c81251",
  "sub": "system:serviceaccount:kube-system:traefik-ingress-controller"
}

正确的配置非常重要，因此请确保对APISERVER的两个调用均返回合理的响应。

export APISERVER=YOURAPISERVER
export TOKEN=YOURTOKEN

curl -X GET $APISERVER/api --header "Authorization: Bearer $TOKEN" --insecure

curl -X GET $APISERVER/api/v1/endpoints --header "Authorization: Bearer $TOKEN" --insecure

创建其他访问token

控制器循环确保每个服务账户都有一个带有API token的secret，可以像我们之前那样被发现。

此外，你还可以为一个服务账户创建额外的token，创建一个ServiceAccountToken类型的secret，并为服务账户添加一个注释，控制器会用生成的token来更新它。

---
apiVersion: v1
kind: Secret
namespace: kube-system
metadata:
  name: traefik-manual-token
  annotations:
    kubernetes.io/service-account.name: traefik-ingress-controller
type: kubernetes.io/service-account-token

# Any tokens for non-existent service accounts will be cleaned up by the token controller.

# kubectl describe secrets/traefik-manual-token

用以下命令创建：

kubectl create -f ./traefik-service-account-secret.yaml
kubectl describe secret traefik-manual-token

删除/无效：

kubectl delete secret traefik-manual-token

对外部traefik 2的更改构成定义

我们需要在文章开头给出的链接中获得的traefik2配置进行哪些更改？

https://github.com/Voronenko/traefik2-compose-template

a） 我们在新文件夹kubernetes_data中存储ca.crt文件，该文件用于验证对Kubernetes授权的调用。这是可以在kubeconfig文件的clusters-> cluster-> certificate-authority-data下找到的证书。

该volume将映射在/var/run/secrets/kubernetes.io/serviceaccount下以获取官方Traefik 2镜像

volumes:
    ...
      - ./kubernetes_data:/var/run/secrets/kubernetes.io/serviceaccount

b） 调整Traefik 2 kubernetescrd后端以提供3个参数：endpoint、证书路径和token。请注意，作为外部Traefik作为docker容器，你需要指定正确的endpoint地址，并确保以安全的方式进行。

  - "--providers.kubernetescrd=true"
      - "--providers.kubernetescrd.endpoint=https://192.168.3.100:6443"
      - "--providers.kubernetescrd.certauthfilepath=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
      - "--providers.kubernetescrd.token=YOURTOKENWITHOUTANYQUOTES

如果你都执行正确了，那么你现在应该在Traefik UI上看到了一些希望。如果你没有看到traefik，或者在运行Traefik时有问题，你可以查看之后的故障排除部分。

现在是时候通过Trafik 2暴露一些Kubernetes服务了，以确保Traefik 2能够作为ingress工作。让我们来看经典案例whoami服务，whoami-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: whoami

spec:
  ports:
    - protocol: TCP
      name: web
      port: 80
  selector:
    app: whoami

---
kind: Deployment
apiVersion: apps/v1
metadata:
  namespace: default
  name: whoami
  labels:
    app: whoami

spec:
  replicas: 2
  selector:
    matchLabels:
      app: whoami
  template:
    metadata:
      labels:
        app: whoami
    spec:
      containers:
        - name: whoami
          image: containous/whoami
          ports:
            - name: web
              containerPort: 80

并且以http或https的方式暴露它，whoami.k.voronenko.net全限定域名下的whoami-ingress-route.yaml。

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: ingressroute-notls
  namespace: default
spec:
  entryPoints:
    - web
  routes:
    - match: Host(`whoami.k.voronenko.net`)
      kind: Rule
      services:
        - name: whoami
          port: 80

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: ingressroute-tls
  namespace: default
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`whoami.k.voronenko.net`)
      kind: Rule
      services:
        - name: whoami
          port: 80
  tls:
    certResolver: default

然后应用它：

kubectl apply -f whoami-service.yaml
  kubectl apply -f whoami-ingress-route.yaml

应用后，你应该会在Traefik dashboard上看到一些希望，即KubernetesCRD后端。

正如你所看到的，Traefik已经检测到我们的K3s Kubernetes集群上运行的新工作负载，而且它与我们在同一个盒子上的经典Docker工作负载（如portainer）很好地共存。

让我们检查一下Traefik 2是否将Traefik路由到了我们的Kubernetes工作负载：如你所见，你可以在http和https endpoint上成功地接触到whoami工作负载，浏览器接受你的证书为可信任的“绿标签”。

我们的目标达到了！我们在本地笔记本上配置了Traefik 2。Traefik 2将你的docker或Kubernetes工作流暴露在http或https endpoint上。带可选的 letsencrypt 的 Traefik 2 将负责 https。

故障排查

正如你所知，在配置过程可能存在多个问题，你可以考虑使用一些分析工具，如：

https://github.com/Voronenko/dotfiles/blob/master/Makefile#L185

我特别建议：

a） VMWare octant：这是一个基于Web的功能强大的Kubernetes dashboard，你可以在上面使用你的kubeconfig

b） Rakess：这是一个独立工具也是一个kubectl插件，用于显示Kubernetes服务器资源的访问矩阵（https://github.com/corneliusweig/rakkess）

检查系统账户的凭据

rakkess --sa kube-system:traefik-ingress-controller

c） kubectl

检查哪些角色与服务账户相关联

kubectl get clusterrolebindings -o json | jq -r '
  .items[] |
  select(
    .subjects // [] | .[] |
    [.kind,.namespace,.name] == ["ServiceAccount","kube-system","traefik-ingress-controller"]
  ) |
  .metadata.name'

d） Traefik 文档：例如kubernetescrd后端提供了更多配置开关的方式。

--providers.kubernetescrd  (Default: "false")
        Enable Kubernetes backend with default settings.
    --providers.kubernetescrd.certauthfilepath  (Default: "")
        Kubernetes certificate authority file path (not needed for in-cluster client).
    --providers.kubernetescrd.disablepasshostheaders  (Default: "false")
        Kubernetes disable PassHost Headers.
    --providers.kubernetescrd.endpoint  (Default: "")
        Kubernetes server endpoint (required for external cluster client).
    --providers.kubernetescrd.ingressclass  (Default: "")
        Value of kubernetes.io/ingress.class annotation to watch for.
    --providers.kubernetescrd.labelselector  (Default: "")
        Kubernetes label selector to use.
    --providers.kubernetescrd.namespaces  (Default: "")
        Kubernetes namespaces.
    --providers.kubernetescrd.throttleduration  (Default: "0")
        Ingress refresh throttle duration
    --providers.kubernetescrd.token  (Default: "")
        Kubernetes bearer token (not needed for in-cluster client).
    --providers.kubernetesingress  (Default: "false")
        Enable Kubernetes backend with default settings.
    --providers.kubernetesingress.certauthfilepath  (Default: "")
        Kubernetes certificate authority file path (not needed for in-cluster client).
    --providers.kubernetesingress.disablepasshostheaders  (Default: "false")
        Kubernetes disable PassHost Headers.
    --providers.kubernetesingress.endpoint  (Default: "")
        Kubernetes server endpoint (required for external cluster client).
    --providers.kubernetesingress.ingressclass  (Default: "")
        Value of kubernetes.io/ingress.class annotation to watch for.
    --providers.kubernetesingress.ingressendpoint.hostname  (Default: "")
        Hostname used for Kubernetes Ingress endpoints.
    --providers.kubernetesingress.ingressendpoint.ip  (Default: "")
        IP used for Kubernetes Ingress endpoints.
    --providers.kubernetesingress.ingressendpoint.publishedservice  (Default: "")
        Published Kubernetes Service to copy status from.
    --providers.kubernetesingress.labelselector  (Default: "")
        Kubernetes Ingress label selector to use.
    --providers.kubernetesingress.namespaces  (Default: "")
        Kubernetes namespaces.
    --providers.kubernetesingress.throttleduration  (Default: "0")
        Ingress refresh throttle duration
    --providers.kubernetesingress.token  (Default: "")
        Kubernetes bearer token (not needed for in-cluster client).

e） 确保Traefik有足够的权限可以访问apiserver endpoint

如果你希望Traefik为你查询信息：通过在配置中放置一些错误的apiserver地址，可以查看访问的endpoint和查询顺序。有了这些知识和你的Traefik Kubernetes token，你就可以使用Traefik凭证检查这些endpoint是否可以访问。

traefik_1    | E0421 12:30:12.624877       1 reflector.go:125] pkg/mod/k8s.io/client-go@v0.0.0-20190718183610-8e956561bbf5/tools/cache/reflector.go:98: Failed to list *v1.Endpoints: Get https://192.168.3.101:6443/api/v1/endpoints?limit=500&resourceVersion=0:
traefik_1    | E0421 12:30:12.625341       1 reflector.go:125] pkg/mod/k8s.io/client-go@v0.0.0-20190718183610-8e956561bbf5/tools/cache/reflector.go:98: Failed to list *v1.Service: Get https://192.168.3.101:6443/api/v1/services?limit=500&resourceVersion=0:
traefik_1    | E0421 12:30:12.625395       1 reflector.go:125] pkg/mod/k8s.io/client-go@v0.0.0-20190718183610-8e956561bbf5/tools/cache/reflector.go:98: Failed to list *v1beta1.Ingress: Get https://192.168.3.101:6443/apis/extensions/v1beta1/ingresses?limit=500&resourceVersion=0:
traefik_1    | E0421 12:30:12.625449       1 reflector.go:125] pkg/mod/k8s.io/client-go@v0.0.0-20190718183610-8e956561bbf5/tools/cache/reflector.go:98: Failed to list *v1alpha1.Middleware: Get https://192.168.3.101:6443/apis/traefik.containo.us/v1alpha1/middlewares?limit=500&resourceVersion=0:
traefik_1    | E0421 12:30:12.625492       1 reflector.go:125] pkg/mod/k8s.io/client-go@v0.0.0-20190718183610-8e956561bbf5/tools/cache/reflector.go:98: Failed to list *v1alpha1.IngressRoute: Get https://192.168.3.101:6443/apis/traefik.containo.us/v1alpha1/ingressroutes?limit=500&resourceVersion=0:
traefik_1    | E0421 12:30:12.625531       1 reflector.go:125] pkg/mod/k8s.io/client-go@v0.0.0-20190718183610-8e956561bbf5/tools/cache/reflector.go:98: Failed to list *v1alpha1.TraefikService: Get https://192.168.3.101:6443/apis/traefik.containo.us/v1alpha1/traefikservices?limit=500&resourceVersion=0:
traefik_1    | E0421 12:30:12.625572       1 reflector.go:125] pkg/mod/k8s.io/client-go@v0.0.0-20190718183610-8e956561bbf5/tools/cache/reflector.go:98: Failed to list *v1alpha1.TLSOption: Get https://192.168.3.101:6443/apis/traefik.containo.us/v1alpha1/tlsoptions?limit=500&resourceVersion=0:
traefik_1    | E0421 12:30:12.625610       1 reflector.go:125] pkg/mod/k8s.io/client-go@v0.0.0-20190718183610-8e956561bbf5/tools/cache/reflector.go:98: Failed to list *v1alpha1.IngressRouteTCP: Get https://192.168.3.101:6443/apis/traefik.containo.us/v1alpha1/ingressroutetcps?limit=500&resourceVersion=0:

f） 记录K3s本身

安装脚本将自动检测你的操作系统是使用systemd还是openrc并启动服务。使用openrc运行时，将在/var/log/k3s.log中创建日志。使用systemd运行时，将在/var/log/syslog中创建日志，并使用journalctl -u k3s查看。

在那里，你可能会得到一些提示，例如：

кві 21 15:42:44 u18d k3s[612]: E0421 15:42:44.936960     612 authentication.go:104] Unable to authenticate the request due to an error: invalid bearer token

这将为你提供有关K8s Traefik起初使用时出现问题的线索，Enjoy your journey！

相关代码你可以在以下链接中找到：

https://github.com/Voronenko/k3s-mini