认识kubernetes
https://github.com/gjmzj/kubeasz
service是核心，service是由pod组成的，pod是由容器组成的，提供service的是容器，service和pod通过标签关联，pod运行在Node上，每个pod都有一个特殊的容器叫pause（共享网络、共享数据），其他容器叫做业务容器，
https://coding.net/u/aminglinux/p/yuanke_centos7/git/tree/master/k8s
1.是一个开源的，用于管理云平台中多个主机上的容器化的应用，Kubernetes的目标是让部署容器化的应用简单并且高效（powerful）,Kubernetes提供了应用部署，规划，更新，维护的一种机制。
http://docs.kubernetes.org.cn/227.html
https://www.cnblogs.com/xhyan/p/6656062.html
https://www.cnblogs.com/fengjian2016/p/6392900.html
https://kubernetes.io/zh/docs/tutorials/kubernetes-basics/
2.安装kubernetes，关闭防火墙
[root@centos-01 ~]# systemctl stop firewalld [root@centos-01 ~]# systemctl disable firewalld Removed symlink /etc/systemd/system/basic.target.wants/firewalld.service. Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service. [root@centos-01 ~]# setenforce 0 setenforce: SELinux is disabled [root@centos-01 ~]#
3.安装etcd（作用存储kubernetes里面的配置文件）和kubernetes
[root@centos-01 ~]# yum install -y etcd kubernetes
4.修改配置文件，将--selinux-enabled 改为 --selinux-enabled=false --insecure-registry gcr.io
[root@centos-01 ~]# vim /etc/sysconfig/docker
OPTIONS='--selinux-enabled=false --insecure-registry gcr.io --log-driver=journald --signature-verification=false'
if [ -z "${DOCKER_CERT_PATH}" ]; then
DOCKER_CERT_PATH=/etc/docker
fi
5.编辑apiserver配置文件，把--admission_control参数中的ServiceAccount删除
[root@centos-01 ~]# vim /etc/kubernetes/apiserver
6.准备工作，安装python-rhsm-certificates包，如果提示python-rhsm-certificates-1.19.10-1.el7_4.x86_64 被已安装的 subscription-manager-rhsm-certificates1.20.11-1.el7.centos.x86_64 取代
yum install python-rhsm-certificates
[root@centos-01 ~]# wget http://mirror.centos.org/centos/7/os/x86_64/Packages/python-rhsm-certificates-1.19.10-1.el7_4.x86_64.rpm --2018-12-11 04:01:39-- http://mirror.centos.org/centos/7/os/x86_64/Packages/python-rhsm-certificates-1.19.10-1.el7_4.x86_64.rpm 正在解析主机 mirror.centos.org (mirror.centos.org)... 213.184.126.230, 2605:9000:401:102::2 正在连接 mirror.centos.org (mirror.centos.org)|213.184.126.230|:80... 已连接。 已发出 HTTP 请求，正在等待回应... 200 OK 长度：42188 (41K) [application/x-rpm] 正在保存至: “python-rhsm-certificates-1.19.10-1.el7_4.x86_64.rpm” 100%[======================================================================================>] 42,188 66.1KB/s 用时 0.6s 2018-12-11 04:01:40 (66.1 KB/s) - 已保存 “python-rhsm-certificates-1.19.10-1.el7_4.x86_64.rpm” [42188/42188]) [root@centos-01 ~]# rpm2cpio python-rhsm-certificates-1.19.10-1.el7_4.x86_64.rpm |cpio -iv --to-stdout ./etc/rhsm/ca/redhatuep.pem > /etc/rhsm/ca/redhat-uep.pem 17 块 [root@centos-01 ~]#
7.配置docker加速器
vi /etc/docker/daemon.json//加入如下内容
{
"registry-mirrors": ["https://dhq9bx4f.mirror.aliyuncs.com"]
}
8.按顺序启动所有服务（红的是master节点上的，绿的是）
for s in etcd docker kube-apiserver kube-controller-manager kube-scheduler kubelet kube-proxy do systemctl start $s done
9.创建一个rc文件
vim mysql-rc.yaml
apiVersion: v1
kind: ReplicationController #副本控制器RC
metadata:
name: mysql #RC的名称，全局唯一
spec:
replicas: 1 #Pod副本的期待数量
selector:
app: mysql #符合目标的Pod拥有此标签
template: #根据此模板创建Pod的副本（实例）
metadata:
labels:
app: mysql #Pod副本拥有的标签，对应RC的Selector
spec:
containers: #Pod内容器的定义部分
- name: mysql #容器的名称
image: mysql:5.6 #容器对应的Docker image
ports:
- containerPort: 3306 #容器应用监听的端口号
env: #注入容器内的环境变量
- name: MYSQL_ROOT_PASSWORD
value: "123456"
10.创建rc
[root@centos-01 ~]# kubectl create -f mysql-rc.yaml replicationcontroller "mysql" created
查看是否pull成功了镜像，如果没有pull成功需要手动pull
docker images
docker pull registry.access.redhat.com/rhel7/pod-infrastructure:latest
docker pull mysql:5.6
11.查看命令干了什么
[root@centos-01 ~]# tail /var/log/messages（其实是docker在下载mysql镜像）
12.查看都有哪些rc
[root@centos-01 ~]# kubectl get rc NAME DESIRED CURRENT READY AGE mysql 1 1 0 5m
13.查看pod状态（状态变成running说明没问题）
[root@centos-01 ~]# kubectl get pod NAME READY STATUS RESTARTS AGE mysql-b57jv 0/1 Pending 0 7m [root@centos-01 ~]#
[root@centos-02 rhsm]# kubectl get pod NAME READY STATUS RESTARTS AGE mysql-n1jtc 1/1 Running 0 21m [root@centos-02 rhsm]#
14.查看service
[root@centos-01 ~]# kubectl get service
15.创建service（svc）文件
[root@centos-02 ~]# vim mysql-svc.yaml
apiVersion: v1
kind: Service
metadata:
name: mysql
spec:
ports:
- port: 3306
selector:
app: mysql
[root@centos-02 ~]# kubectl create -f mysql-svc.yaml
service "mysql" created
[root@centos-02 ~]#
[root@centos-02 ~]# kubectl get svc
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes 10.254.0.1 <none> 443/TCP 35m
mysql 10.254.73.183（就是我们的serviceIP） <none> 3306/TCP 35s
[root@centos-02 ~]#
16.安装mysql
[root@centos-02 ~]# yum install -y mysql
17.这样我们就可以通过10.254.73.183：3306访问mysql了
[root@centos-02 ~]# mysql -uroot -p123456 -h10.254.73.183 Welcome to the MariaDB monitor. Commands end with ; or g. Your MySQL connection id is 1 Server version: 5.6.42 MySQL Community Server (GPL) Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or 'h' for help. Type 'c' to clear the current input statement. MySQL [(none)]>
18.创建myweb
[root@centos-02 ~]# vim myweb-rc.yaml
kind: ReplicationController
metadata:
name: myweb
spec:
replicas: 1
selector:
app: myweb
template:
metadata:
labels:
app: myweb
spec:
containers:
- name: myweb
image: kubeguide/tomcat-app:v1
ports:
- containerPort: 8080
env:
- name: MYSQL_SERVICE_HOST
value: '10.254.73.183' #这里的IP需要通过kubect get svc 查看mysql的cluster ip（10.254.73.183）
- name: MYSQL_SERVICE_PORT
value: '3306'
[root@centos-02 ~]# kubectl create -f myweb-rc.yaml replicationcontroller "myweb" created [root@centos-02 ~]#
19.查看pod
[root@centos-02 ~]# kubectl get pod NAME READY STATUS RESTARTS AGE mysql-n1jtc 1/1 Running 0 2h myweb-1x5h9 0/1 ContainerCreating 0 1m [root@centos-02 ~]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE docker.io/mysql 5.6 a876cc5d29e4 3 weeks ago 256 MB registry.access.redhat.com/rhel7/pod-infrastructure latest 99965fb98423 14 months ago 209 MB [root@centos-02 ~]#
20.创建service
[root@centos-02 ~]# vim myweb-svc.yaml
kind: Service
metadata:
name: myweb
spec:
type: NodePort
ports:
- port: 8080
nodePort: 30001
selector:
app: myweb
[root@centos-02 ~]# kubectl create -f myweb-svc.yaml service "myweb" created [root@centos-02 ~]#
21.查看pod和service
[root@centos-02 ~]# kubectl get pod
NAME READY STATUS RESTARTS AGE
mysql-n1jtc 1/1 Running 0 2h
myweb-1x5h9 1/1 Running 0 8m
[root@centos-02 ~]# kubectl get svc
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes 10.254.0.1 <none> 443/TCP 2h
mysql 10.254.73.183 <none> 3306/TCP 2h
myweb 10.254.51.166 <nodes> 8080:30001/TCP 1m
[root@centos-02 ~]#
22.访问tomcat
[root@centos-02 ~]# curl -I 10.254.51.166:8080 HTTP/1.1 200 OK Server: Apache-Coyote/1.1 Content-Type: text/html;charset=UTF-8 Transfer-Encoding: chunked Date: Thu, 13 Dec 2018 20:01:56 GMT [root@centos-02 ~]#
[root@centos-02 ~]# curl 10.254.51.166:8080/demo/
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>HPE University Docker&Kubernetes Learning</title>
</head>
<body align="center">
<h2>Congratulations!!</h2>
<br></br>
<input type="button" value="Add..." onclick="location.href='input.html'" >
<br></br>
<TABLE align="center" border="1" width="600px">
<TR>
<TD>Name</TD>
<TD>Level(Score)</TD>
</TR>
<TR>
<TD>google</TD>
<TD>100</TD>
</TR>
<TR>
<TD>docker</TD>
<TD>100</TD>
</TR>
<TR>
<TD>teacher</TD>
<TD>100</TD>
</TR>
<TR>
<TD>HPE</TD>
<TD>100</TD>
</TR>
<TR>
<TD>our team</TD>
<TD>100</TD>
</TR>
<TR>
<TD>me</TD>
<TD>100</TD>
</TR>
</TABLE>
</body>
</html>
[root@centos-02 ~]#
[root@centos-02 ~]# curl 192.168.242.132:30001/demo/
23.通过浏览器访问，我们发现默认FORWARD是DROP,我们需要不FORWARD打开
[root@centos-02 ~]# iptables -nvL
Chain INPUT (policy ACCEPT 4 packets, 248 bytes)
pkts bytes target prot opt in out source destination
537K 532M KUBE-FIREWALL all -- * * 0.0.0.0/0 0.0.0.0/0
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
96 11569 DOCKER-ISOLATION all -- * * 0.0.0.0/0 0.0.0.0/0
96 11569 DOCKER all -- * docker0 0.0.0.0/0 0.0.0.0/0
78 10629 ACCEPT all -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 ACCEPT all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
3 180 ACCEPT all -- docker0 docker0 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 3 packets, 156 bytes)
pkts bytes target prot opt in out source destination
507K 188M KUBE-FIREWALL all -- * * 0.0.0.0/0 0.0.0.0/0
508K 188M KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */
Chain DOCKER (1 references)
pkts bytes target prot opt in out source destination
Chain DOCKER-ISOLATION (1 references)
pkts bytes target prot opt in out source destination
96 11569 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain KUBE-FIREWALL (2 references)
pkts bytes target prot opt in out source destination
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000
Chain KUBE-SERVICES (1 references)
pkts bytes target prot opt in out source destination
[root@centos-02 ~]#
[root@centos-02 ~]# iptables -P FORWARD ACCEPT [root@centos-02 ~]#
24.成功访问
25.我们发现多了一个HPE_APP表
[root@centos-02 ~]# mysql -uroot -p123456 -h10.254.73.183 Welcome to the MariaDB monitor. Commands end with ; or g. Your MySQL connection id is 11 Server version: 5.6.42 MySQL Community Server (GPL) Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or 'h' for help. Type 'c' to clear the current input statement. MySQL [(none)]> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | HPE_APP | | mysql | | performance_schema | +--------------------+ 4 rows in set (0.00 sec) MySQL [(none)]>
MySQL [(none)]> use HPE_APP; Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed MySQL [HPE_APP]> SHOW TABLES; +-------------------+ | Tables_in_HPE_APP | +-------------------+ | T_USERS | +-------------------+ 1 row in set (0.00 sec) MySQL [HPE_APP]> SELECT * FROM T_USERS; +----+-------------+-------+ | ID | USER_NAME | LEVEL | +----+-------------+-------+ | 1 | me | 100 | | 2 | our team | 100 | | 3 | HPE | 100 | | 4 | teacher | 100 | | 5 | docker | 100 | | 6 | google | 100 | | 7 | 15001316083 | 100 | +----+-------------+-------+ 7 rows in set (0.00 sec) MySQL [HPE_APP]>
26.命令总结
[root@centos-02 ~]# kubectl create -f ^C [root@centos-02 ~]# kubectl get pod NAME READY STATUS RESTARTS AGE mysql-n1jtc 1/1 Running 0 3h myweb-1x5h9 1/1 Running 0 25m [root@centos-02 ~]# kubectl get rc NAME DESIRED CURRENT READY AGE mysql 1 1 1 3h myweb 1 1 1 25m [root@centos-02 ~]# kubectl get svc NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes 10.254.0.1 <none> 443/TCP 3h mysql 10.254.73.183 <none> 3306/TCP 2h myweb 10.254.51.166 <nodes> 8080:30001/TCP 18m [root@centos-02 ~]#
kubernetes相关概念
1.kubernetes从物理上划分为master节点和node节点
2.RC中动态修改pod副本数量，下面两个rc分别有一个动态的pod，我们动态调整成2个mysql pod
[root@centos-02 ~]# kubectl get rc NAME DESIRED CURRENT READY AGE mysql 1 1 1 21h myweb 1 1 1 19h [root@centos-02 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE mysql-n1jtc 1/1 Running 0 21h myweb-1x5h9 1/1 Running 0 19h [root@centos-02 ~]#
[root@centos-02 ~]# kubectl scale rc mysql --replicas=2 replicationcontroller "mysql" scaled [root@centos-02 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE mysql-cc1tx 0/1 ContainerCreating 0 6s mysql-n1jtc 1/1 Running 0 21h myweb-1x5h9 1/1 Running 0 19h [root@centos-02 ~]# kubectl get rc NAME DESIRED CURRENT READY AGE mysql 2 2 2 21h myweb 1 1 1 19h [root@centos-02 ~]#
3.删除RC,RC对应的pod也会被删除掉
[root@centos-02 ~]# kubectl get rc NAME DESIRED CURRENT READY AGE mysql 2 2 2 22h myweb 1 1 1 19h [root@centos-02 ~]# kubectl delete rc myweb replicationcontroller "myweb" deleted [root@centos-02 ~]#
[root@centos-02 ~]# kubectl get rc NAME DESIRED CURRENT READY AGE mysql 2 2 2 22h myweb 1 1 1 19h [root@centos-02 ~]# kubectl delete rc myweb replicationcontroller "myweb" deleted [root@centos-02 ~]# kubectl get rc NAME DESIRED CURRENT READY AGE mysql 2 2 2 22h [root@centos-02 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE mysql-cc1tx 1/1 Running 0 14m mysql-n1jtc 1/1 Running 0 22h [root@centos-02 ~]#
4.svc中还是有myweb，需要手动删掉
[root@centos-02 ~]# kubectl get svc NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes 10.254.0.1 <none> 443/TCP 22h mysql 10.254.73.183 <none> 3306/TCP 21h myweb 10.254.51.166 <nodes> 8080:30001/TCP 19h [root@centos-02 ~]#
[root@centos-02 ~]# kubectl delete svc myweb service "myweb" deleted [root@centos-02 ~]# kubectl get svc NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes 10.254.0.1 <none> 443/TCP 22h mysql 10.254.73.183 <none> 3306/TCP 21h [root@centos-02 ~]#
5.Deployment 在1.2版本引入的概念，目的是为了解决pod编排问题，在内部使用了Replica Set，它和RC比较，相似度为90%以上，可以认为 是RC的升级版。 跟RC比较，最大的一个特点是可以知道pod部署的进度。
Deployment示例：
[root@centos-02 ~]# vim fr-dp.yaml
kind: Deployment
metadata:
name: frontend
spec:
replicas: 1
selector:
matchLabels:
tier: frontend
matchExpressions:
- {key: tier, operator: In, values: [frontend]}
template:
metadata:
labels:
app: app-demo
tier: frontend
spec:
containers:
- name: tomcat-demo
image: tomcat
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
6.创建frontend
[root@centos-02 ~]# kubectl create -f fr-dp.yaml deployment "frontend" created [root@centos-02 ~]#
[root@centos-02 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE frontend-141477217-20031 0/1 ContainerCreating 0 45s mysql-cc1tx 1/1 Running 0 1h mysql-n1jtc 1/1 Running 0 23h [root@centos-02 ~]#
[root@centos-02 ~]# kubectl get deployment NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE frontend 1 1 1 0 1m [root@centos-02 ~]#
7.查看pod情况
[root@centos-02 ~]# kubectl describe pod frontend-141477217-20031
Name: frontend-141477217-20031
Namespace: default
Node: 127.0.0.1/127.0.0.1
Start Time: Sat, 15 Dec 2018 00:15:50 +0800
Labels: app=app-demo
pod-template-hash=141477217
tier=frontend
Status: Pending
IP:
Controllers: ReplicaSet/frontend-141477217
Containers:
tomcat-demo:
Container ID:
Image: tomcat
Image ID:
Port: 8080/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Volume Mounts: <none>
Environment Variables: <none>
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
No volumes.
QoS Class: BestEffort
Tolerations: <none>
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
4m 4m 1 {default-scheduler } Normal Scheduled Successfully assigned frontend-141477217-20031 to 127.0.0.1
4m 4m 1 {kubelet 127.0.0.1} Warning MissingClusterDNS kubelet does not have ClusterDNS IP configured and cannot create Pod
using "ClusterFirst" policy. Falling back to DNSDefault policy.
4m 4m 1 {kubelet 127.0.0.1} spec.containers{tomcat-demo} Normal Pulling pulling image "tomcat"
[root@centos-02 ~]#
8.查看下有没有pull下来tomcat的镜像
[root@centos-02 ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.io/tomcat latest 48dd385504b1 6 days ago 475 MB
docker.io/mysql 5.6 a876cc5d29e4 4 weeks ago 256 MB
registry.access.redhat.com/rhel7/pod-infrastructure latest 99965fb98423 14 months ago 209 MB
docker.io/kubeguide/tomcat-app v1 a29e200a18e9 2 years ago 358 MB
[root@centos-02 ~]#
9.HPA:在1.1版本，kubernetes官方发布了HPA，实现pod的动态扩容、缩容，它属于一种kubernetes的资源对象。它通过追踪分析 RC控制的所有目标pod的负载变化情况，来决定是否需要针对性地调整目标Pod的副本数，这是HPA的实现原理。
pod负载度量指标： 1）CpuUtilizationPercentage 目标pod所有副本自身的cpu利用率平用均值。一个pod自身的cpu利用率＝该pod当前cpu的使用量／pod Request值。如果某 一个时刻，CPUUtilizationPercentage的值超过了80%，则判定当前的pod已经不够支撑业务，需要增加pod。 2）应用程序自定义的度量指标，比如服务每秒内的请求数（TPS或QPS） HPA示例： apiVerion: autosacling/v1 kind: HorizontalPodAutoscaler metadata: name: php-apache namespace: default spec: maxReplicas: 10 minReplicas: 1 scaleTargetRef: kind: Deployment name: php-apache targetCPUUtilizationPercentage: 90 说明：HPA控制的目标对象是一个名叫php-apache的Deployment里的pod副本，当cpu平均值超过90%时就会扩容，pod副本 数控制范围是1－10. 除了以上的xml文件定义HPA外，也可以用命令行的方式来定义： kubectl autoscale deployment php-apache --cpu-percent=90 --min=1 --max=10
10.Service是kubernetes中最核心的资源对象之一，Service可以理解成是微服务架构中的一个“微服务”，pod、RC、 Deployment都是为Service提供嫁衣的。
简单讲一个service本质上是一组pod组成的一个集群，前面我们说过service和pod之间是通过Label来串起来的，相同Service的 pod的Label一样。同一个service下的所有pod是通过kube-proxy实现负载均衡，而每个service都会分配一个全局唯一的虚拟 ip，也叫做cluster ip。在该service整个生命周期内，cluster ip是不会改变的，而在kubernetes中还有一个dns服务，它把 service的name和cluster ip映射起来。
11.查看pod的IP地址以及端口
[root@centos-02 ~]# kubectl get endpoints NAME ENDPOINTS AGE kubernetes 192.168.242.132:6443 23h mysql 172.17.0.2:3306,172.17.0.4:3306 23h [root@centos-02 ~]#
12.查看service分配的cluster ip
[root@centos-02 ~]# kubectl get svc mysql -o yaml
apiVersion: v1
kind: Service
metadata:
creationTimestamp: 2018-12-13T17:43:37Z
name: mysql
namespace: default
resourceVersion: "2329"
selfLink: /api/v1/namespaces/default/services/mysql
uid: 9ebfd5d8-fefe-11e8-b6e3-000c2959c2d2
spec:
clusterIP: 10.254.73.183
ports:
- port: 3306
protocol: TCP
targetPort: 3306
selector:
app: mysql
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
[root@centos-02 ~]#
13.Namespace当kubernetes集群中存在多租户的情况下，就需要有一种机制实现每个租户的资源隔离。而namespace的目的就是为了实现资 源隔离。
查看集群所有的namespace
[root@centos-02 ~]# kubectl get namespace NAME STATUS AGE default Active 1d kube-system Active 1d [root@centos-02 ~]#
[root@centos-02 ~]# vim dev-ns.yaml apiVersion: v1 kind: Namespace metadata: name: dev
14.创建dev namespace
[root@centos-02 ~]# kubectl create -f dev-ns.yaml namespace "dev" created [root@centos-02 ~]#
15.获取namespace
[root@centos-02 ~]# kubectl get ns
NAME STATUS AGE
default Active 1d
dev Active 37s
kube-system Active 1d
[root@centos-02 ~]#
16.定义pod
[root@centos-02 ~]# vim busybox-pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: busybox
namespace: dev
spec:
containers:
- image: busybox
command:
- sleep
- "500"
name: busybox
[root@centos-02 ~]# kubectl create -f busybox-pod.yaml pod "busybox" created [root@centos-02 ~]#
17.我们直接get pods不能查看到busybox，需要指定namespace为dev查看
[root@centos-02 ~]# kubectl get pods NAME READY STATUS RESTARTS AGE frontend-141477217-20031 1/1 Running 0 1h mysql-cc1tx 1/1 Running 0 3h mysql-n1jtc 1/1 Running 0 1d [root@centos-02 ~]#
[root@centos-02 ~]# kubectl get pods -n dev NAME READY STATUS RESTARTS AGE busybox 1/1 Running 0 4m [root@centos-02 ~]#
[root@centos-02 ~]# kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE default frontend-141477217-20031 1/1 Running 0 1h default mysql-cc1tx 1/1 Running 0 3h default mysql-n1jtc 1/1 Running 0 1d dev busybox 1/1 Running 0 5m [root@centos-02 ~]#
kubectl get pods -n dev
kubectl命令用法
语法： kubectl [command] [TYPE] [NAME] [flags] 1 command：子命令，用于操作Kubernetes集群资源对象的命令，如create, delete, describe, get, apply等 2 TYPE：资源对象的类型，如pod, service, rc, deployment, node等，可以单数、复数以及简写（pod, pods, po/service, services, svc） 3 NAME：资源对象的名称，不指定则返回所有，如get pod 会返回所有pod， get pod nginx， 只返回nginx这个pod 4 flags：kubectl子命令的可选参数，例如-n 指定namespace，-s 指定apiserver的URL
资源对象类型列表 可以用这个命令获取到： kubectl explain 或 kubectl api-resources
名称 简写 componentsstatuses cs daemonsets ds deployment deploy events ev endpoints ep horizontalpodautoscalers hpa ingresses ing jobs limitranges limits nodes no namspaces ns pods po persistentvolumes pv persistentvolumeclaims pvc resourcequotas quota replicationcontrollers rc secrets serviceaccounts sa services svc
特殊用法： kubectl get pods pod1 pod2 kubectl get pod/pod1 rc/rc1 kubectl create -f pod1.yaml -f rc1.yaml -f service1.yaml
kubectl子命令 主要包括对资源的创建、删除、查看、修改、配置、运行等 kubectl --help 可以查看所有子命令 kubectl参数 kubectl options 可以查看支持的参数，例如--namespace指定所在namespace kubectl输出格式 kubectl命令可以用多种格式对结果进行显示，输出格式通过-o参数指定： -o支持的格式有 输出格式 说明 custom-columns=<spec> 根据自定义列名进行输出，逗号分隔 custom-columns-file=<filename> 从文件中获取自定义列名进行输出 json 以JSON格式显示结果 jsonpath=<template> 输出jasonpath表达式定义的字段信息 jasonpath-file=<filename> 输出jsonpath表达式定义的字段信息，来源于文件 name 仅输出资源对象的名称 wide 输出更多信息，比如会输出node名 yaml 以yaml格式输出 举例： kubectl get pod -o wide kubectl get pod -o yaml kubectl get pod -o custom-columns=NAME:.metadata.name,RESC:.metadata.resourceVersion kubectl get pod --sort-by=.metadata.name //按name排序 kubectl命令示例： 1）创建资源对象 根据yaml文件创建service和deployment kubectl create -f my-service.yaml -f my-deploy.yaml 也可以指定一个目录，这样可以一次性根据该目录下所有yaml或json文件定义资源 kubectl create -f <directory> 2）查看资源对象 查看所有pod kubectl get pods 查看deployment和service kubectl get deploy,svc 3）描述资源对象 显示node的详细信息 kubectl describe nodes <node-name> 显示pod的详细信息 kubectl describe pods/<pod-name> 显示deployment管理的pod信息 kubectl describe pods <deployment-name> 4）删除资源对象 基于yaml文件删除 kubectl delete -f pod.yaml 删除所有包含某个label的pod和service kubectl delete po,svc -l name=<lable-name> 删除所有pod kubectl delete po --all 5）执行容器的命令 在pod中执行某个命令，如date kubectl exec <pod-name> date //pod-name如果不加，默认会选择第一个pod 指定pod的某个容器执行命令 kubectl exec <pod-name> date 进入到pod的容器里 kubectl exec -it <pod-name> bash 6）查看容器日志 kubectl logs <pod-name> 可以动态查看，类似于tail -f kubectl logs -f <pod-name> -c <container-name>
搭建kubernetes集群（ansible-playbook）-1
1.软硬件限制（详情见https://coding.net/u/aminglinux/p/yuanke_centos7/git/tree/master/k8s）
cpu和内存 master：至少1核两g，推荐两核4g，node至少1核2g
linux系统内核版本至少3.10，推荐centos7/RHEL7
docker 至少1.9版本，推荐1.12+
etcd至少2.0版本，推荐3.0+
2.四台机器全部执行
yum update yum install epel-release yum install python
3.deploy节点安装和准备ansible
(1)130服务器安装pip
yum install -y python-pip git
(2)升级pip源
pip install pip --upgrade -i http://mirrors.aliyun.com/pypi/simple/ --trusted-host mirrors.aliyun.com
(3)安装ansible （pip和yum挺像的主要用于安装python下的插件），如果这种方式安装失败用yum安装（yum list|grep ansible、 yum install -y ansible）
[root@centos-04 ~]# pip install --no-cache-dir ansible -i http://mirrors.aliyun.com/pypi/simple/ --trusted-host mirrors.aliyun.com
Installing collected packages: MarkupSafe, jinja2, PyYAML, idna, enum34, six, pycparser, cffi, asn1crypto, cryptography, pynacl, pyasn1, bcrypt, paramiko, ansible Running setup.py install for PyYAML ... done Running setup.py install for pycparser ... done Running setup.py install for ansible ... done Successfully installed MarkupSafe-1.1.0 PyYAML-3.13 ansible-2.7.5 asn1crypto-0.24.0 bcrypt-3.1.5 cffi-1.11.5 cryptography-2.4.2 enum34-1.1.6 idna-2.8 jinja2-2.10 paramiko-2.4.2 pyasn1-0.4.4
pycparser-2.19 pynacl-1.3.0 six-1.12.0 [root@centos-04 ~]#
deploy节点配置免密码登录
1.生成密钥对
[root@centos-04 ~]# ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: SHA256:qrghr27RSPWCV5mBazMJiT6V3KDX0+s9twBLSnjemac root@centos-04 The key's randomart image is: +---[RSA 2048]----+ |. o.=.+ | |.o.*.=. | |..=.=o . | | =.O... . | |. *.+o +S | | o .+ =.* | |. o o.* = . | | + o . o + . | |+o+.. E . | +----[SHA256]-----+ [root@centos-04 ~]#
[root@centos-04 ~]# for ip in 130 131 132 133; do ssh-copy-id 192.168.242.$ip; done
2.登录各个机器测试（ctrl+d退出）
[root@centos-04 ~]# for ip in 130 131 132 133; do ssh 192.168.242.$ip; done Last login: Tue Dec 18 19:04:47 2018 from 192.168.242.1 [root@centos-04 ~]# 登出 Connection to 192.168.242.130 closed. Last login: Tue Dec 18 19:08:23 2018 from 192.168.242.1 ABRT 已检测到 '4' 个问题。预了解详细信息请执行：abrt-cli list --since 1545131303 [root@centos-01 ~]# 登出 Connection to 192.168.242.131 closed. Last login: Tue Dec 18 19:08:14 2018 from 192.168.242.1 [root@centos-02 ~]# 登出 Connection to 192.168.242.132 closed. Last login: Tue Dec 18 19:06:44 2018 from 192.168.242.1 [root@centos-03 ~]# 登出 Connection to 192.168.242.133 closed. [root@centos-04 ~]#
deploy上编排k8s
[root@centos-04 ~]# git clone https://github.com/gjmzj/kubeasz.git [root@centos-04 ~]# mkdir -p /etc/ansible [root@centos-04 ~]# mv kubeasz/* /etc/ansible/
[root@centos-04 ~]# cd /etc/ansible/ [root@centos-04 ansible]# du -sh 2.6M . [root@centos-04 ansible]# ls 01.prepare.yml 05.kube-node.yml 20.addnode.yml 24.restore.yml bin manifests tools 02.etcd.yml 06.network.yml 21.addmaster.yml 90.setup.yml docs pics 03.docker.yml 07.cluster-addon.yml 22.upgrade.yml 99.clean.yml down README.md 04.kube-master.yml 11.harbor.yml 23.backup.yml ansible.cfg example roles [root@centos-04 ansible]#
配置集群参数
[root@centos-04 ansible]# cp example/hosts.m-masters.example hosts [root@centos-04 ansible]#
[root@centos-04 ansible]# vim hosts （根据实际情况修改IP地址）
[deploy]
192.168.242.130 NTP_ENABLED=no
[etcd] 192.168.242.130 NODE_NAME=etcd1 192.168.242.131 NODE_NAME=etcd2 192.168.242.132 NODE_NAME=etcd3
[kube-master]
192.168.242.130
192.168.242.133
[lb] 192.168.242.130 LB_IF="ens33" LB_ROLE=backup
192.168.242.133 LB_IF="eno16777736" LB_ROLE=master
[kube-node] 192.168.242.131 192.168.242.132
K8S_VER="v1.11"
MASTER_IP="192.168.242.150"
从百度云网盘下载二进制文件 https://pan.baidu.com/s/1c4RFaA#list/path=%2F 可以根据自己所需版本，下载对应的tar包，这里我下载1.11 经过一番折腾，最终把k8s.1-11-2.tar.gz的tar包放到了depoly上，上传包-解压-移动到bin目录
[root@centos-04 ~]# rz rz waiting to receive. Starting zmodem transfer. Press Ctrl+C to cancel. 100% 214046 KB 9306 KB/s 00:00:23 0 Errorss [root@centos-04 ~]# ls anaconda-ks.cfg k8s.1-11-3.tar.gz kubeasz [root@centos-04 ~]#
tar zxvf k8s.1-11-2.tar.gz mv bin/* /etc/ansible/bin/
[root@centos-04 ~]# cd /etc/ansible/bin/ [root@centos-04 bin]# ls bridge docker dockerd etcdctl kube-controller-manager loopback calicoctl docker-compose docker-init flannel kubectl portmap cfssl docker-containerd docker-proxy helm kubelet readme.md cfssl-certinfo docker-containerd-ctr docker-runc host-local kube-proxy cfssljson docker-containerd-shim etcd kube-apiserver kube-scheduler [root@centos-04 bin]#
创建证书和安装准备
[root@centos-04 ansible]# ansible-playbook 01.prepare.yml
安装etcd集群
[root@centos-04 ansible]# ansible-playbook 02.etcd.yml
检查etcd节点健康状况：（如果提示etcdctl命令不存在，先执行bash）
for ip in 130 131 132 ; do ETCDCTL_API=3 etcdctl --endpoints=https://192.168.242.$ip:2379 -- cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem endpoint healt; done
安装docker
ansible-playbook 03.docker.yml
安装master节点
ansible-playbook 04.kube-master.yml
查看集群状态
kubectl get componentstatus
安装node节点
[root@centos-04 ansible]# ansible-playbook 05.kube-node.yml
查看node节点
kubectl get nodes
部署集群网络
ansible-playbook 06.network.yml
kubectl get pod -n kube-system
安装集群插件(dns, dashboard)
ansible-playbook 07.cluster-addon.yml
查看kube-system namespace下的服务
kubectl get svc -n kube-system
一步到位安装（上面七步可以直接用下面命令）
ansible-playbook 90.setup.yml
查看集群信息：
kubectl cluster-info
查看node/pod使用资源情况：
kubectl top node kubectl top pod --all-namespaces
测试DNS
创建nginx service
kubectl run nginx --image=nginx --expose --port=80
创建busybox 测试pod
kubectl run busybox --rm -it --image=busybox /bin/sh //进入到busybox内部 nslookup nginx.default.svc.cluster.local //结果如下 Server: 10.68.0.2 Address: 10.68.0.2:53 Name: nginx.default.svc.cluster.local Address: 10.68.9.156
备份和恢复
[root@centos-04 ~]# cd [root@centos-04 ~]# kubectl run mysql --image=mysql:5.6 --expose --port=3306 （自动创建mysql的service和mysql的deployment）
创建备份目录
[root@centos-04 ~]# mkdir -p /backup/k8s [root@centos-04 ~]#
备份etcd数据
[root@centos-04 ~]# ETCDCTL_API=3 etcdctl snapshot save /backup/k8s/snapshot.db
备份ca证书
[root@centos-04 ~]# cp /etc/kubernetes/ssl/ca* /backup/k8s/ [root@centos-04 ~]#
模拟集群崩溃
deploy节点执行 ansible-playbook /etc/ansible/99.clean.yml
恢复步骤如下（在deploy节点）：
恢复ca证书（我靠有问题，完了完了，我们用户一键安装重新安装一遍吧）
mkdir -p /etc/kubernetes/ssl cp /backup/k8s/ca* /etc/kubernetes/ssl/
[root@centos-04 ~]# cp /backup/k8s/ca* /etc/kubernetes/ssl/ cp: 无法获取"/backup/k8s/ca*" 的文件状态(stat): 没有那个文件或目录 [root@centos-04 ~]#
ansible-playbook 90.setup.yml
检查etcd是否成功
for ip in 130 131 132 ; do ETCDCTL_API=3 etcdctl --endpoints=https://192.168.242.$ip:2379 -- cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem endpoint healt; done
检查master节点的集群状态
kubectl get componentstatus
查看node节点
kubectl get nodes
查看kube-system namespace下的服务
kubectl get svc -n kube-system
查看集群信息
kubectl cluster-info
创建nginx
[root@centos-04 ansible]# kubectl run nginx --image=nginx --expose --port=80
创建mysql
[root@centos-04 ansible]# history |grep run |grep mysql 935 kubectl run mysql --image=mysql:5.6 --expose --port=3306 958 history |grep run |grep mysql [root@centos-04 ansible]# kubectl run mysql --image=mysql:5.6 --expose --port=3306
查看pod所在的节点(可以看到ip)
kubectl get pod -o wide
删除某个节点
创建mysql失败查看logs发现需要创建初始化密码
我们重新备份
执行clear
ansible-playbook /etc/ansible/99.clean.yml
恢复ca证书
mkdir -p /etc/kubernetes/ssl cp /backup/k8s/ca* /etc/kubernetes/ssl/
重建集群
cd /etc/ansible ansible-playbook 01.prepare.yml ansible-playbook 02.etcd.yml ansible-playbook 03.docker.yml ansible-playbook 04.kube-master.yml ansible-playbook 05.kube-node.yml
恢复etcd数据
停止服务
ansible etcd -m service -a 'name=etcd state=stopped'
清空文件
ansible etcd -m file -a 'name=/var/lib/etcd/member/ state=absent'
登录所有的etcd节点，参照本etcd节点/etc/systemd/system/etcd.service的服务文件，替换如下{{}}中变量后执行（在每台机器执行下面的命令都需要修改对应的红色部分，改为对应的etcd* 和对应的ip）
cd /backup/k8s/ ETCDCTL_API=3 etcdctl snapshot restore snapshot.db --name etcd1 --initialcluster etcd1=https://192.168.242.130:2380,etcd2=https://192.168.242.131:2380,etcd3=https://192.168.242.132:2380 --initial-cluster-token etcd-cluster-0 --initial-advertise-peer-urls https://192.168.111.128:2380
将128服务器的backup目录拷贝到129 130服务器
执行上面的步骤后，会生成{{ NODE_NAME }}.etcd目录（三台机器都执行下面的对应命令）
cp -r etcd1.etcd/member /var/lib/etcd/ systemctl restart etcd
检查是否都好了
在deploy节点重建网络
ansible-playbook /etc/ansible/tools/change_k8s_network.yml
不想手动恢复，可以用ansible自动恢复 需要一键备份
ansible-playbook /etc/ansible/23.backup.yml
检查/etc/ansible/roles/cluster-backup/files目录下是否有文件
tree /etc/ansible/roles/cluster-backup/files/ //如下 ├── ca # 集群CA 相关备份 │ ├── ca-config.json │ ├── ca.csr │ ├── ca-csr.json │ ├── ca-key.pem │ └── ca.pem ├── hosts # ansible hosts备份 │ ├── hosts # 最近的备份 │ └── hosts-201807231642 ├── readme.md └── snapshot # etcd 数据备份 ├── snapshot-201807231642.db └── snapshot.db # 最近的备份
模拟故障：
ansible-playbook /etc/ansible/99.clean.yml
修改文件/etc/ansible/roles/cluster-restore/defaults/main.yml，指定要恢复的etcd快照备份，如果不修改就是最新的一次
恢复操作：
ansible-playbook /etc/ansible/24.restore.yml ansible-playbook /etc/ansible/tools/change_k8s_network.yml