k3s

　　看同事写的k3s的总结，还以为看错了，不应该是k8s么。查了下，还真的是有k3s的。   

        我对k8s也是了解一些主要组件的作用，更多的是用kubectl定位问题。干技术的就是不仅工作上的知识要学习，也要扩展知识面的宽度。

　　

　　下面的转自：https://www.jianshu.com/p/dbc8d9a8374e

　　

　　k3s是rancher®出品的一个简化、轻量的k8s，本篇博客记录k3s的安装及踩的部分坑。
　　从名字上也能看出，k3s比k8s少了些东西，详情可见其官网k3s.io，本地试验可参考官网离线安装教程

 

k3s官网

安装步骤

准备工作

　　首先去其github的releases页下载主可执行文件k3s、离线安装包k3s-airgap-images-amd64.tar和安装脚本
　　我用的是v1.18.6+k3s1版本，其于2020年7月16日发布。
　　增加可执行文件和脚本的可执行权限

wget https://get.k3s.io -O install-k3s.sh
chmod +x install-k3s.sh

　　需要有/usr/local/bin/k3s，可考虑软连接

sudo ln -s /home/dev/program/k3s /usr/local/bin/k3s

　　复制tar文件到/var/lib/rancher/k3s/agent/images

sudo mkdir -p /var/lib/rancher/k3s/agent/images
sudo cp k3s-airgap-images-amd64.tar /var/lib/rancher/k3s/agent/images

定制一些变量

　　先设置变量如下：

export INSTALL_K3S_SKIP_DOWNLOAD=true
export INSTALL_K3S_EXEC="--docker --write-kubeconfig ~/.kube/config --write-kubeconfig-mode 666"

　　逐个解释一下：

1. INSTALL_K3S_SKIP_DOWNLOAD=true效果为不去下载k3s可执行文件
2. INSTALL_K3S_EXEC="（略）"效果为启动k3s服务时使用的额外参数
3. --docker效果为使用docker而不是默认的containerd
4. --write-kubeconfig-mode 666效果为将配置文件权限改为非所有者也可读可写，进而使kubectl命令无需root或sudo
5. --write-kubeconfig ~/.kube/config效果为将配置文件写到k8s默认会用的位置，而不是k3s默认的位置/etc/rancher/k3s/k3s.yaml。后者会导致istio、helm需要额外设置或无法运行。

　　官网教程-安装选项中还有其他可用的选项

执行安装脚本

$ ./install-k3s.sh
[INFO]  Skipping k3s download and verify
[INFO]  Skipping /usr/local/bin/kubectl symlink to k3s, already exists
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s

　　执行k3s命令看看效果

$ k3s
NAME:
   k3s - Kubernetes, but small and simple

USAGE:
   k3s [global options] command [command options] [arguments...]

VERSION:
   v0.9.1 (755bd1c6)

COMMANDS:
   server   Run management server
   agent    Run node agent
   kubectl  Run kubectl
   crictl   Run crictl
   ctr      Run ctr
   help, h  Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --debug        Turn on debug logs
   --help, -h     show help
   --version, -v  print the version

　　还有k3s kubectl和kubectl

$ k3s kubectl get all --all-namespaces
NAMESPACE     NAME                             READY   STATUS    RESTARTS   AGE
kube-system   pod/coredns-66f496764-mkwjv      1/1     Running   0          5m9s
kube-system   pod/helm-install-traefik-t4xlj   1/1     Running   0          5m8s


NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-system   service/kube-dns     ClusterIP   10.43.0.10   <none>        53/UDP,53/TCP,9153/TCP   5m27s
default       service/kubernetes   ClusterIP   10.43.0.1    <none>        443/TCP                  5m25s


NAMESPACE     NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/coredns   1/1     1            1           5m27s

NAMESPACE     NAME                                DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/coredns-66f496764   1         1         1       5m9s



NAMESPACE     NAME                             COMPLETIONS   DURATION   AGE
kube-system   job.batch/helm-install-traefik   0/1           5m8s       5m25s

访问kubernetes服务

　　由于k3s默认没有提供dashboard作为web ui，先访问k8s的rest

NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes   ClusterIP   10.43.0.1    <none>        443/TCP

　　会要求输入用户名密码，在~/.kube/config中有访问其的用户名密码，内容类似如下：

users:
- name: default
  user:
    password: ec2fb0ab4401d7f2525d480fd08e908d
    username: admin

　　文件位置默认为/etc/rancher/k3s/k3s.yaml，但是前述步骤中通过--write-kubeconfig ~/.kube/config修改
　　认证似乎是www basic（对k8s还没了解到这种程度，此处存疑）
　　也可kubectl version或随便kubectl run测试一下

若干问题

如何卸载

　　见install.sh的回显，其中有uninstall-script：

[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh

离线安装包

　　如果没有复制k3s-airgap-images-amd64.tar，会卡着

$ k3s kubectl get all --all-namespaces
NAMESPACE     NAME                             READY   STATUS              RESTARTS   AGE
kube-system   pod/helm-install-traefik-t4xlj   0/1     ContainerCreating   0          4m42s
kube-system   pod/coredns-66f496764-mkwjv      0/1     ContainerCreating   0          4m43s


NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-system   service/kube-dns     ClusterIP   10.43.0.10   <none>        53/UDP,53/TCP,9153/TCP   5m1s
default       service/kubernetes   ClusterIP   10.43.0.1    <none>        443/TCP                  4m59s


NAMESPACE     NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/coredns   0/1     1            0           5m1s

NAMESPACE     NAME                                DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/coredns-66f496764   1         1         0       4m43s



NAMESPACE     NAME                             COMPLETIONS   DURATION   AGE
kube-system   job.batch/helm-install-traefik   0/1           4m42s      4m59s

复制后，安装过程继续

拉不下镜像

　　可能因为啥拉镜像失败，可通过kubectl describe pod coredns-57d8bbb86-mndrr -n kube-system查看events：

Events:
  Type     Reason                  Age               From                      Message
  ----     ------                  ----              ----                      -------
  Warning  FailedScheduling        <unknown>         default-scheduler         0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.
  Normal   Scheduled               <unknown>         default-scheduler         Successfully assigned kube-system/coredns-57d8bbb86-mndrr to dk-aspire-5943g
  Warning  FailedCreatePodSandBox  3s (x4 over 89s)  kubelet, dk-aspire-5943g  Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.1": Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

　　可见是由于拉不下镜像k8s.gcr.io/pause:3.1，于是从阿里云拉下镜像，再tag

$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1
3.1: Pulling from google_containers/pause
cf9202429979: Pull complete 
Digest: sha256:759c3f0f6493093a9043cc813092290af69029699ade0e3dbe024e968fcb7cca
Status: Downloaded newer image for registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1
registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1
$ docker images
REPOSITORY                                                  TAG                 IMAGE ID            CREATED             SIZE
registry.cn-hangzhou.aliyuncs.com/google_containers/pause   3.1                 da86e6ba6ca1        22 months ago       742kB
$ docker tag da86e6ba6ca1 k8s.gcr.io/pause:3.1
$ docker images
REPOSITORY                                                  TAG                 IMAGE ID            CREATED             SIZE
k8s.gcr.io/pause                                            3.1                 da86e6ba6ca1        22 months ago       742kB
registry.cn-hangzhou.aliyuncs.com/google_containers/pause   3.1                 da86e6ba6ca1        22 months ago       742kB

kubectl需要root权限

　　前已述及，在安装前设置若干变量，其中有针对这个问题的

$ kubectl get all
WARN[2019-10-20T22:58:52.068331383+08:00] Unable to read /etc/rancher/k3s/k3s.yaml, please start server with --write-kubeconfig-mode to modify kube config permissions 
error: Error loading config file "/etc/rancher/k3s/k3s.yaml": open /etc/rancher/k3s/k3s.yaml: permission denied

　　/etc/rancher/k3s/k3s.yaml的默认权限为-rw-------即600，所有者root root
　　根据提示，在启动时需要带有--write-kubeconfig-mode *新权限*，经试验，666可以起到让kubectl无需root权限的效果
　　此外，v1.17.0+k3s.1的文档中提到一个选项：

   --rootless                                 (experimental) Run rootless

　　但是试验不成功，service k3s启动失败
　　定制环境变量如下：

export INSTALL_K3S_SKIP_DOWNLOAD=true
export INSTALL_K3S_EXEC="--docker --write-kubeconfig ~/.kube/config --write-kubeconfig-mode 666"

　　启动失败日志片段如下：

$ ./install-k3s.sh 
（略）
[INFO]  systemd: Starting k3s
Job for k3s.service failed because the control process exited with error code.
See "systemctl status k3s.service" and "journalctl -xe" for details.
$ journalctl -xe
（略）
Jan 09 15:31:40 dk-mi13 k3s[4490]: time="2020-01-09T15:31:40.488024565+08:00" level=fatal msg="resolving : determining current user: $HOME is not defined"
Jan 09 15:31:40 dk-mi13 systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- An ExecStart= process belonging to unit k3s.service has exited.
-- 
-- The process' exit code is 'exited' and its exit status is 1.
Jan 09 15:31:40 dk-mi13 systemd[1]: k3s.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- The unit k3s.service has entered the 'failed' state with result 'exit-code'.
Jan 09 15:31:40 dk-mi13 systemd[1]: Failed to start Lightweight Kubernetes.
-- Subject: A start job for unit k3s.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- A start job for unit k3s.service has finished with a failure.
-- 
-- The job identifier is 16669 and the job result is failed.

//TODO
　　这个问题暂未解决，报错中似乎比较关键的

level=fatal msg="resolving : determining current user: $HOME is not defined"

　　也有问题，无论是自己的账户还是su用root账户，查看$HOME变量均可得到值

$ echo $HOME
/home/dk
$ env | grep HOME
HOME=/home/dk
（略）
$ su
Password: 
# echo $HOME
/root
# env | grep HOME
HOME=/root
（略）

KUBECONFIG位置

　　配置文件默认位置给其他处带来一些不便，例如使用helm需要如下额外参数以指定配置文件的位置

--kubeconfig /etc/rancher/k3s/k3s.yaml

　　想改为更为通用的~/.kube/config，使用参数--write-kubeconfig ~/.kube/config
　　此外，在v1.17.0+k3s.1版本中，使用kubectl -v 6可见其对配置文件的处理：

$ kubectl get all -v 6
I0109 11:27:11.815808   20876 loader.go:375] Config loaded from file:  /etc/rancher/k3s/k3s.yaml

　　依然读取/etc/rancher/k3s/k3s.yaml，但这个文件实际上链接到了~/.kube/config：

$ ll /etc/rancher/k3s/k3s.yaml 
lrwxrwxrwx 1 root root 21 Jan  9 15:48 /etc/rancher/k3s/k3s.yaml -> /home/dk/.kube/config
$ ll ~/.kube/config
-rw-rw-rw- 1 root root 1052 Jan  9 15:48 /home/dk/.kube/config

kubectl get all 耗时长

　　用v1.17.0+k3s.1执行kubectl get all耗时较长（v1.18.6+k3s1中问题依旧），但是kubectl get pod等查看一种资源的命令耗时并不较长，增加-v 6查看更详细日志：

$ kubectl get all -v 6
（略）
I0109 11:27:11.824426   20876 round_trippers.go:443] GET https://127.0.0.1:6443/api?timeout=32s 200 OK in 8 milliseconds
I0109 11:27:11.824977   20876 round_trippers.go:443] GET https://127.0.0.1:6443/apis?timeout=32s 200 OK in 0 milliseconds
I0109 11:27:11.825346   20876 cached_discovery.go:130] failed to write cache to /home/dk/.kube/cache/discovery/127.0.0.1_6443/servergroups.json due to mkdir /home/dk/.kube/cache: permission denied
I0109 11:27:11.828528   20876 round_trippers.go:443] GET https://127.0.0.1:6443/api/v1?timeout=32s 200 OK in 2 milliseconds
I0109 11:27:11.829574   20876 cached_discovery.go:87] failed to write cache to /home/dk/.kube/cache/discovery/127.0.0.1_6443/v1/serverresources.json due to mkdir /home/dk/.kube/cache: permission denied
（略）

　　可知，原因是向~/.kube/cache文件夹下写时无权限，处理大量错误耗费了时间。默认无此文件夹，上层.kube文件夹所有者root root，权限755

$ ll ~ | grep .kube
drwxr-xr-x  2 root root  4096 Jan  9 11:30  .kube/

　　若使用sudo kubectl get all没有此耗时问题。
　　修正方法，将此文件夹权限改为其他用户可写；或者新建cache和http-cache两文件夹，并更改所有者为当前用户。后一种方法例：

$ sudo mkdir cache http-cache
$ sudo chown dk:dk cache http-cache

　　至此，解决了kubectl get all等命令耗时太长问题

作者：dracula337435
链接：https://www.jianshu.com/p/dbc8d9a8374e
来源：简书
著作权归作者所有。商业转载请联系作者获得授权，非商业转载请注明出处。