deepin 安装 microk8s 1.17 踩坑记录

microk8s 1.17

环境: Debian 系 Deepin 15.11 桌面系统, ubuntu 理论上可参照

安装参考链接: http://www.imooc.com/article/291860

安装

sudo apt update
sudo apt install snapd snap
export $PATH=PATH:/snap/bin >> ~/.zshrc && source ~/.zshrc
sudo snap install microk8s --classic
sudo microk8s.status --wait-ready

## status 输出
microk8s is running
addons:
cilium: disabled
dashboard: enabled
dns: enabled
fluentd: disabled
gpu: disabled
helm: disabled
ingress: disabled
istio: disabled
jaeger: disabled
juju: disabled
knative: disabled
kubeflow: disabled
linkerd: disabled
metallb: disabled
metrics-server: disabled
prometheus: disabled
rbac: disabled
registry: disabled
storage: disabled

监控 pods 状态

watch microk8s.kubectl get all --all-namespaces

这是问题解决后的状态, STATUS 都是 Running

NAMESPACE     NAME                                                  READY   STATUS    RESTARTS   AGE
kube-system   pod/coredns-9b8997588-hlqxz                           1/1     Running   54         4h38m
kube-system   pod/dashboard-metrics-scraper-687667bb6c-7f79n        0/1     Pending   0          6m50s
kube-system   pod/dashboard-metrics-scraper-687667bb6c-r8tgq        0/1     Evicted   0          37m
kube-system   pod/heapster-v1.5.2-5c58f64f8b-lj2nf                  4/4     Running   0          37m
kube-system   pod/kubernetes-dashboard-5c848cc544-47fqk             0/1     Evicted   0          6m53s
kube-system   pod/kubernetes-dashboard-5c848cc544-4zdgs             0/1     Evicted   0          6m52s
kube-system   pod/kubernetes-dashboard-5c848cc544-7mhmj             0/1     Evicted   0          6m52s
kube-system   pod/kubernetes-dashboard-5c848cc544-7xwfw             0/1     Pending   0          6m50s
kube-system   pod/kubernetes-dashboard-5c848cc544-c7t4v             0/1     Evicted   0          6m51s
kube-system   pod/kubernetes-dashboard-5c848cc544-kfnds             0/1     Evicted   0          6m53s
kube-system   pod/kubernetes-dashboard-5c848cc544-l8r6s             0/1     Evicted   0          6m54s
kube-system   pod/kubernetes-dashboard-5c848cc544-ms8gg             0/1     Evicted   0          6m54s
kube-system   pod/kubernetes-dashboard-5c848cc544-ngvlc             0/1     Evicted   0          6m54s
kube-system   pod/kubernetes-dashboard-5c848cc544-p7xqc             0/1     Evicted   0          6m54s
kube-system   pod/kubernetes-dashboard-5c848cc544-wlw5m             0/1     Evicted   0          37m
kube-system   pod/monitoring-influxdb-grafana-v4-6d599df6bf-nvr62   2/2     Running   0          37m

NAMESPACE     NAME                                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes                  ClusterIP   10.152.183.1     <none>        443/TCP                  5h34m
kube-system   service/dashboard-metrics-scraper   ClusterIP   10.152.183.61    <none>        8000/TCP                 37m
kube-system   service/heapster                    ClusterIP   10.152.183.168   <none>        80/TCP                   37m
kube-system   service/kube-dns                    ClusterIP   10.152.183.10    <none>        53/UDP,53/TCP,9153/TCP   4h38m
kube-system   service/kubernetes-dashboard        ClusterIP   10.152.183.29    <none>        443/TCP                  37m
kube-system   service/monitoring-grafana          ClusterIP   10.152.183.195   <none>        80/TCP                   37m
kube-system   service/monitoring-influxdb         ClusterIP   10.152.183.212   <none>        8083/TCP,8086/TCP        37m

NAMESPACE     NAME                                             READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/coredns                          1/1     1            1           4h38m
kube-system   deployment.apps/dashboard-metrics-scraper        0/1     1            0           37m
kube-system   deployment.apps/heapster-v1.5.2                  1/1     1            1           37m
kube-system   deployment.apps/kubernetes-dashboard             0/1     1            0           37m
kube-system   deployment.apps/monitoring-influxdb-grafana-v4   1/1     1            1           37m

NAMESPACE     NAME                                                        DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/coredns-9b8997588                           1         1         1       4h38m
kube-system   replicaset.apps/dashboard-metrics-scraper-687667bb6c        1         1         0       37m
kube-system   replicaset.apps/heapster-v1.5.2-5c58f64f8b                  1         1         1       37m
kube-system   replicaset.apps/kubernetes-dashboard-5c848cc544             1         1         0       37m
kube-system   replicaset.apps/monitoring-influxdb-grafana-v4-6d599df6bf   1         1         1       37m

修改 .zshrc

.bashrc 教程一大把

如果本地没有安装 kubectl 可以使用 alias; 否则请不要全部复制粘贴;

如果已经安装了 kubectl，可以用下面的命令覆盖配置文件：

microk8s.kubectl config view --raw > $HOME/.kube/config

-- 来自开篇的安装教程 http://www.imooc.com/article/291860

以下仅限于 zsh 用户; bash 用户百度下即可

vim ~/.zshrc

export PATH=$PATH:/usr/local/go/bin:/snap/bin
alias kubectl='microk8s.kubectl'
# 命令补全
if [ $commands[microk8s.kubectl] ]; then 
	source <(microk8s.kubectl completion zsh | 
	sed "s/complete -o default -F __start_kubectl kubectl/complete -o default -F __start_kubectl microk8s.kubectl/g" | 
	sed "s/complete -o default -o nospace -F __start_kubectl kubectl/complete -o default -o nospace -F __start_kubectl microk8s.kubectl/g"); 
fi

添加ctr proxy

microk8s.docker 命令在 1.17 版本被移除; 由 containerd 代替;

之前版本的, 需要修改 dockerd-env 加代理

sudo vim /var/snap/microk8s/current/args/containerd-env

HTTPS_PROXY=http://127.0.0.1:1082

重启 containerd 服务

sudo systemctl restart snap.microk8s.daemon-containerd.service

没有proxy 的也可以参照开篇链接教程, 条条大路通罗马, 不是非要proxy才能完成这个事情

修改内存/硬盘空间限制

sudo vim /var/snap/microk8s/current/args/kubelet

# 酌情复制
--eviction-hard="memory.available<1024Mi,nodefs.available<1Gi,imagefs.available<1Gi"
## 意思是: 当本node宿主机的 内存小于 1024Mi / 硬盘存储 小于 1Gi 时, 会将 pod 强制驱逐

这里之前有一些理解上的错误, 原本以为是 允许多大 内存/硬盘 使用, 后来发现并不是这样子的; 鉴于这篇文章现在有 7 个阅读, 我对这 7 位读者表示抱歉...
后面针对这样不明确的地方一定查证后再上传

防火墙 ufw

关于 CrashLoopBackOff 问题

sudo iptables -P FORWARD ACCEPT
# 1.17版本是 cni0; 之前版本是 cnr0, 参照官网 TroubleShooting
sudo ufw allow in on cni0 && sudo ufw allow out on cni0
sudo ufw default allow routed

重启 microk8s

microk8s.stop && microk8s.start

开启 add-on

sudo microk8s.enable dns dashboard
# 安装输出备忘部分
# enable dashbord后的输出部分, RBAC 未开启状态下需要依赖 token开头的两个命令 获取 token
If RBAC is not enabled access the dashboard using the default token retrieved with:

token=$(microk8s.kubectl -n kube-system get secret | grep default-token | cut -d " " -f1);microk8s.kubectl -n kube-system describe secret $token

In an RBAC enabled setup (microk8s.enable RBAC) you need to create a user with restricted
permissions as shown in:
https://github.com/kubernetes/dashboard/blob/master/docs/user/access-control/creating-sample-user.md

开启dashboard

开启proxy

kubectl proxy --address='0.0.0.0' --accept-hosts='^*$'

新开命令行, 使用 kubectl get service -n kube-system 查看dashboard的 ip

kubectl get service -n kube-system 
NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE
dashboard-metrics-scraper   ClusterIP   10.152.183.169   <none>        8000/TCP                 17h
heapster                    ClusterIP   10.152.183.76    <none>        80/TCP                   17h
kube-dns                    ClusterIP   10.152.183.10    <none>        53/UDP,53/TCP,9153/TCP   22h
kubernetes-dashboard        ClusterIP   10.152.183.237   <none>        443/TCP                  17h
monitoring-grafana          ClusterIP   10.152.183.197   <none>        80/TCP                   17h
monitoring-influxdb         ClusterIP   10.152.183.82    <none>        8083/TCP,8086/TCP        17h

浏览器访问 https://10.152.183.237 可以到达 k8s-dashboard 界面 *注意 https 一定要加上, 不能去访问 443 端口(10.152.183.237:443 是行不通的)

或者: http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#/login

dashboard 用户

csdn 大佬路子 https://blog.csdn.net/wucong60/article/details/81911859

#### 
### dashboard addon 启动之后, microk8s 会自动启动一个 token 认证服务 default-token-b96pr 在 default namespace 里面
### 可以通过命令:
### $ token=$(microk8s.kubectl -n kube-system get secret | grep default-token | cut -d " " -f1);microk8s.kubectl -n kube-system describe secret $token 
### 直接获取到 token
####

---

# 下面是手动创建secret
# 创建服务
kubectl create serviceaccount cluster-admin-dashboard-sa
# 启动
kubectl create clusterrolebinding cluster-admin-dashboard-sa --clusterrole=cluster-admin --serviceaccount=default:cluster-admin-dashboard-sa

# 获取 pod 名称
kubectl get secret | grep cluster-admin-dashboard-sa

# 获取 token
kubectl describe secrets/cluster-admin-dashboard-sa-token-82dwx

# 查看 token 服务
kubectl get serviceaccount

# 删除手动创建的服务
kubectl delete serviceaccount cluster-admin-dashboard-sa

https://github.com/kubernetes/dashboard/blob/master/docs/user/access-control/creating-sample-user.md

操作流程:

开启 RBAC 认证后的 官方推荐 操作流程

Create Service Account

mkdir ~/microk8s && cd ~/microk8s

vim dashboard-adminuser.yaml
# 写入文件
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kube-system
# EOF
# 应用更改
kubectl apply -f ./dashboard-adminuser.yaml
# 输出 serviceaccount/admin-user created

Create ClusterRoleBinding

touch role-bind.yaml
echo 'apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kubernetes-dashboard' > role-bind.yaml


kubectl apply -f ./role-bind.yaml          
# 输出 clusterrolebinding.rbac.authorization.k8s.io/admin-user created

获取token

kubectl -n kube-system describe secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}')

coredns 不启动 - deepin

snap 的安装 触发了 apparmor 的启动;

pod错误: CrashLoopBackOff

coredns日志:

kubectl logs -f coredns-xxxxxxx-xxxxx -n kube-system

:: socket permission denied; listen tcp port failed

暂行解决办法:

option#1. 关闭 apparmor https://wiki.debian.org/AppArmor/HowToUse#Disable_AppArmor

$ sudo mkdir -p /etc/default/grub.d
$ echo 'GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT apparmor=0"' 
  | sudo tee /etc/default/grub.d/apparmor.cfg
$ sudo update-grub
$ sudo reboot

option#2. 参考链接 https://blog.csdn.net/u014062332/article/details/100099196

删除 Evicted pod

microk8s 运行一天后, 发现有很多处于 Evicted(被k8s放弃) 的 pod, 原因不明,删了去求,原因是 kubectl 配置文件有问题->k8s检测到系统资源达到了阀值, 放弃了pod以释放资源

kubectl get pods -n kube-system | grep Evicted | awk '{print $1}' | xargs microk8s.kubectl delete pod -n kube-system

参考链接 https://serverfault.com/questions/972120/microk8s-keeps-evicting-pods

参考链接 https://blog.csdn.net/zzq900503/article/details/83788152