理论知识储备：
想了解vxlan网络的知识：https://www.cnblogs.com/shuiguizi/p/10923841.html
想了解docker网络的原理知识：https://www.cnblogs.com/shuiguizi/p/10922049.html
提前准备：
镜像使用centos和nginx，为了方便，将从官网上下载的镜像安装一些工具再重新commit得到新的镜像
yum install net-tools
yum install iputils
yum install iproute *
配置步骤：
0，安装启动etcd，步骤见网络.
在这里两个主机都部署etcd，它会自动选举出leader，访问etcd数据库的话，两个地址都可以。
1，重新配置docker daemon
vi /usr/lib/systemd/system/docker.service
--cluster-store=etcd://106.y.y.3:2379
--cluster-advertise=106.y.y.31:2375
--cluster-store=etcd://188.x.x.113:2379
--cluster-advertise=188.x.x.113:2375
systemctl daemon-reload
systemctl restart docker.service
2，创建docker network
#docker network create ov_net2 -d overlay
[root@master ~]# docker network ls
NETWORK ID NAME DRIVER SCOPE
731d1b63b387 ov_net2 overlay global
3，分别在两个节点上创建两个docker 容器
master：
docker run -ti -d --network=ov_net2 --name=centos21 centos:wxy /bin/sh
minion：
docker run -d --name nginx --network=ov_net2 nginx22
开始解析
1.这个overlay网络的信息情况
[root@master ~]# docker network inspect ov_net2 [ { "Name": "ov_net2", "Id": "731d1b63b38768022160534b619d09d2e0fb139a7504070bf370a7706ed8ee9e", "Created": "2019-05-14T20:08:29.045284861+08:00", "Scope": "global", "Driver": "overlay", "EnableIPv6": false, "IPAM": { "Driver": "default", "Options": {}, "Config": [ { "Subnet": "10.0.1.0/24", "Gateway": "10.0.1.1" } ] }, "Internal": false, "Attachable": false, "Containers": { "d7dc5bf71ccb2e5fb1f6b98dd47b4f79f44a8b73e9536f3f15b036ba0c94f55d": { "Name": "centos21", "EndpointID": "c7771dca216130e46b60cb921d4488eb82f9a5e1e168ec4d7a9d91f183e82ea6", "MacAddress": "02:42:0a:00:01:02", "IPv4Address": "10.0.1.2/24", "IPv6Address": "" }, "ep-611f45864389f90630fa70340dddd4c76b16ac070c49f60aa1679c753b41db7f": { "Name": "nginx22", "EndpointID": "611f45864389f90630fa70340dddd4c76b16ac070c49f60aa1679c753b41db7f", "MacAddress": "02:42:0a:00:01:03", "IPv4Address": "10.0.1.3/24", "IPv6Address": "" } }, "Options": {}, "Labels": {} } ]
2，不同host上的容器是怎么知道对方的地址情况呢？答：从etcd中读取到的。
[root@minion ~]# etcdctl get /docker/network/v1.0/endpoint/731d1b63b38768022160534b619d09d2e0fb139a7504070bf370a7706ed8ee9e/611f45864389f90630fa70340dddd4c76b16ac070c49f60aa1679c753b41db7f {"anonymous":false,"disableResolution":false,"ep_iface":{"addr":"10.0.1.3/24","dstPrefix":"eth","mac":"02:42:0a:00:01:03","routes":null,"srcName":"vethb5c341b","v4PoolID":"GlobalDefault/10.0.1.0/24","v6PoolID":""},"exposed_ports":[{"Proto":6,"Port":80}],"generic":{"com.docker.network.endpoint.exposedports":[{"Proto":6,"Port":80}],"com.docker.network.portmap":[]},"id":"611f45864389f90630fa70340dddd4c76b16ac070c49f60aa1679c753b41db7f","ingressPorts":null,"joinInfo":{"StaticRoutes":null,"disableGatewayService":false},"locator":"106.13.146.31","myAliases":["42c3eff8768d"],"name":"nginx22","sandbox":"f7a0ce169bd7690b45887a462efc169953150311dbb03e4bb2ccaf17ab75add8","svcAliases":null,"svcID":"","svcName":"","virtualIP":"u003cnilu003e"}
3，验证连接情况
[root@master ~]# ip netns exec a740da7c2043 ping 10.0.1.3 -c 2 PING 10.0.1.3 (10.0.1.3) 56(84) bytes of data. From 10.0.1.2 icmp_seq=1 Destination Host Unreachable From 10.0.1.2 icmp_seq=2 Destination Host Unreachable
4，看过Docker原理的都知道，一个容器其实就是创建了一个namespce；另外对于overlay网络为每个host还会再创建一个用于vxlan连接的namespace
[root@master ~]# ip netns exec 1-731d1b63b3 tcpdump -i vxlan1 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on vxlan1, link-type EN10MB (Ethernet), capture size 262144 bytes 20:36:47.283514 ARP, Request who-has 10.0.1.3 tell 10.0.1.2, length 28 20:36:48.304014 ARP, Request who-has 10.0.1.3 tell 10.0.1.2, length 28 20:36:49.328182 ARP, Request who-has 10.0.1.3 tell 10.0.1.2, length 28 [root@master ~]# ip netns exec 1-731d1b63b3 netstat -i Kernel Interface table Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg br0 1450 35 0 0 0 14 0 0 0 BMRU lo 65536 0 0 0 0 0 0 0 0 LRU veth2 1450 35 0 0 0 25 0 0 0 BMRU vxlan1 1450 0 0 0 0 0 0 39 0 BMRU
解析：
tcpdump的时候vxlan接口有发出数据，
netstat统计的时候，veth2中接收的数据有增加，而vxlan1中没有，
这说明什么？
简述bridge的发包原理：
将各种接口add到网桥上，当网桥的一个接口接收到数据，它会根据数据的目的地址决定是上送上层协议栈还是转发，在这里无论arp报文还是icmp报文都属于要转发。
首先，于是报文经过bridge内部的处理从网桥的其他接口出去，包括vxlan1接口，此处的代码属于内核的L2层，如果有tcpdump监听接口则会将报文报给tcpdump一份
然后，报文进入驱动，vxlan的驱动，可以认为驱动就是介于二层链路层和物理设备之间，netstat -i 统计的就是驱动上数据收发情况。
好了，回来，以上可以知道数据会在vxlan驱动那里被丢掉了，而在驱动都做了什么呢？vxlan的驱动简单说就是查fdb表指导转发，如下可以发现
[root@master ~]# ip netns exec 1-731d1b63b3 bridge fdb 12:54:57:62:92:74 dev vxlan1 vlan 1 master br0 permanent 12:54:57:62:92:74 dev vxlan1 master br0 permanent
解析一下，12:54:57:62:92:74为vxlan1也是网桥br0的接口的mac地址
第一条：表示目的mac关联的是vxlan1接口，也可以就认为这个mac地址就是这个接口的，后面的都是修饰vxlan1的：然后他属于vlan1的，从属于br0桥
对比我自己搭建vxlan网络时，namespace中的fdb表的情况
[root@minion ~]# ip netns exec ns200 bridge fdb f2:4d:be:62:09:50 dev vxlan20 vlan 1 master br-vx2 permanent f2:4d:be:62:09:50 dev vxlan20 master br-vx2 permanent 00:00:00:00:00:00 dev vxlan20 dst 188.131.210.113 via ifindex 2 link-netnsid 0 self permanent
rount1：手动再fdb表中添加缺省路由
和之前用namespace模拟的vxlan隧道比较，可以发现他少了缺省路由，即数据不知道从哪里出去，数据本来应该从真正的物理接口eth0出去，所以这里就尝试了手动配置，包括两个节点
ip netns exec 1-731d1b63b3 bridge fdb add 00:00:00:00:00:00 dev vxlan1 dst 106.13.146.3
不行，
结果：
还是不行，数据还是不知道出去哪里，原因是没有指定出口接口，fdb表增加条目的时候无法指定非本namespace下的接口我也是醉了....
round2:新创建一个namespace，替换Docker创建的那个用于vxlan连接的namespace，再把容器代表的接口添加到该ns的桥上
---------------------------未完成，待续--------------------------------------------
============================================
只有使能swarm才能启动gossip协议，进行节点的发现
vi /usr/lib/systemd/system/docker.service
systemctl daemon-reload
systemctl restart docker
]# docker swarm init --advertise-addr=188.131.210.113:2377
Swarm initialized: current node (m5jpgmwxow5ec256vw8bpgxi9) is now a manager.
To add a worker to this swarm, run the following command:
docker swarm join
--token SWMTKN-1-5vpumm8bssl8tsa7aq9bc9ytpgxdob5rp5dm0y4b8zed3ef5e9-eckhrx62u9vb0o95xxam98qjc
188.131.210.113:2377
坑1：端口号之前设置的是2375，则node在加入master的时候会发生
[root@minion ~]# docker swarm join --token SWMTKN-1-4dt7opomolsz9q2kdykheknj2cbmj8sgydxcljl99h07ob9dtj-0acex2ahyx98wm8fuc6rt0k4i 188.131.210.113:2375
Error response from daemon: rpc error: code = 14 desc = grpc: the connection is unavailable
原因是：
For this test, you need two different Docker hosts that can communicate with each other. Each host must have Docker 17.06 or higher with the following ports open between the two Docker hosts:
TCP port 2377 for cluster management communications
TCP and UDP port 7946 for communication among nodes
UDP port 4789 for overlay network traffic
这个原理是这样的：
2377接口
用来集群管理方面的通信，比如某节点想要加入集群啊，那么他就向master节点的该接口发送加入消息
7946接口
用来在集群节点之间进行通信，它首先利用tcp：7946向其peer发消息，如果收到正确回应，说明线路畅通可以保持联系
然后使用udp:7946将自己节点中需要共享的网络信息同步出去（bulk sync）
于是，在master上创建的某个network也就能够被其他节点所发现了
--wxy：这个就是所谓的网络管理平面吧，是不是通过这个替代了etcd等k-v存储结构
4789接口
是不是很熟悉？没错，正是vxlan使用的缺省接口，即用来为overlay网络传输数据，就是vxlan网络中tunnel两端的vtep使用的端口号，不同主机上的container就是用他来通信的
解决办法：将端口号改成2377好了
[root@master ~]# docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
m5jpgmwxow5ec256vw8bpgxi9 * master.wxy Ready Active Leader
z50ektieet4esj1gwlfaisdc4 minion.wxy Ready Active
[root@master ~]# docker network ls
NETWORK ID NAME DRIVER SCOPE
8808070a27e1 bridge bridge local
e0407f6da9d8 docker_gwbridge bridge local
e866d30f43bf host host local
7j887qknji9s ingress overlay swarm
424fac469906 none null local
[root@master ~]#
[root@master ~]# docker network create -d overlay nginx-net
s8o2cknp3hc8l9cs5uqm4gskd
[root@master ~]# docker network ls
NETWORK ID NAME DRIVER SCOPE
8808070a27e1 bridge bridge local
e0407f6da9d8 docker_gwbridge bridge local
e866d30f43bf host host local
7j887qknji9s ingress overlay swarm
s8o2cknp3hc8 nginx-net overlay swarm
424fac469906 none null local
docker network create --driver=overlay --attachable ov-test
[root@master 5.0.10-1.el7.elrepo.x86_64]# ip netns exec 1-po3p8i2id3 bridge fdb show dev vxlan1
02:42:0a:00:00:04 master br0
9e:7e:18:eb:71:77 vlan 1 master br0 permanent
9e:7e:18:eb:71:77 master br0 permanent
02:42:0a:00:00:04 dst 172.16.0.4 link-netnsid 0 self permanent ---重要，
说明：
已经通过网络控制平面就知道对方的mac地址了，但是目的地址是不对的，不能是对方的私网地址（172.16.0.4正是minion的私网地址），得想办法改变这个地址，难道是我加入swarm使用的地址不对？改一下
[root@master ~]# docker network inspect ov-test
[
{
"Name": "ov-test",
"Id": "po3p8i2id3f9thvd7b8qiuu1o",
"Created": "2019-05-19T19:12:30.411636993+08:00",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.0.0.0/24",
"Gateway": "10.0.0.1"
}
]
},
"Internal": false,
"Attachable": true,
"Containers": {
"18d6f2f74b8d615f67a2c270f846102bc043d9ee63514a1edaa1087964e0486f": {
"Name": "centos",
"EndpointID": "65e8f927972704b27e19647040d5abb999d7142f327a26234ff01775fdf95991",
"MacAddress": "02:42:0a:00:00:02",
"IPv4Address": "10.0.0.2/24",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "4097"
},
"Labels": {},
"Peers": [
{
"Name": "master.wxy-61c0f37910dd",
"IP": "188.131.210.113"
},
{
"Name": "minion.wxy-362fc1004972",
"IP": "172.16.0.4"
}
]
}
]
[root@minion ~]# docker network inspect ov-test
[
{
"Name": "ov-test",
"Id": "po3p8i2id3f9thvd7b8qiuu1o",
"Created": "2019-05-19T19:13:41.541163575+08:00",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.0.0.0/24",
"Gateway": "10.0.0.1"
}
]
},
"Internal": false,
"Attachable": true,
"Containers": {
"12459f10b7f68e5eba856a446c46a24bce7d2a7f75b7db0cbd896bc72772455b": {
"Name": "nginx",
"EndpointID": "8ace966fe08b40b75f1e67d7c7a77e0e99abb3b5f9dc56f6df5c45b695698a85",
"MacAddress": "02:42:0a:00:00:04",
"IPv4Address": "10.0.0.4/24",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "4097"
},
"Labels": {},
"Peers": [
{
"Name": "master.wxy-61c0f37910dd",
"IP": "188.131.210.113"
},
{
"Name": "minion.wxy-362fc1004972",
"IP": "172.16.0.4"
}
]
}
]
修正后：
docker swarm init --advertise-addr=188.131.210.113:2377
docker swarm join
--token SWMTKN-1-4m6s7rs3hkhh4bhdv78gymjc7bbq663cfw6qy4yh9ysja2nuvb-bknjkrcu52tvo50dl9ermvxnr
--advertise-addr 106.13.146.3:2377
188.131.210.113:2377
docker swarm join
--token SWMTKN-1-1qvdgad5v0q5jafdw2ym20lcffi2cd0d4fsuxlr9p5dg8ogqrp-48ti1h5gccibyqx0phzboanq5
--advertise-addr 106.13.146.3:2377
188.131.210.113:2377
docker swarm join
--token SWMTKN-1-4m6s7rs3hkhh4bhdv78gymjc7bbq663cfw6qy4yh9ysja2nuvb-bknjkrcu52tvo50dl9ermvxnr
--advertise-addr 106.13.146.3:2377
188.131.210.113:2377
docker network create --driver=overlay --attachable ov-test
[root@master ~]# docker run -d -ti --network=ov-test2 --name=centos2 centos:wxy /bin/sh
[root@minion ~]# docker run -d --network=ov-test2 --name=nginx2 nginx
journalctl -u docker.service
1，第一个错误，node节点的ip配置错
evel=error msg="periodic bulk sync failure for network 5s2fbdra2g5m9qt14wmiveir8: bulk sync failed on node minion.wxy-611e1a90a99f: failed to send a TCP message during bulk sync: dial tcp 106.13.146.31:7946: connect: connection refused"
2，第二个错误，node节点在加入集群的时候没有指定advertise ip，这样master在sync时访问的是对方的小网ip
vel=error msg="Error in responding to bulk sync from node 172.16.0.4: failed to send a TCP message during bulk sync: dial tcp 172.16.0.4:7946: i/o timeout"
3，问题2，网络不共享
定位过程：发现master有发tcp:7946且有回应，然后又发了udp:7946,node也收到了，但是发现携带的数据内容不大，所以猜测是没有将网络信息打包
Server Version: 1.13.1
iptables -A INPUT -p udp --dport 4789 -j ACCEPT
================
在minonin节点上使能集群
docker swarm init --advertise-addr=106.13.146.3:2377
iptables rule and container port has conflicts.
try :
sudo iptables -t nat -L -n --line-numbers | grep 7946
sudo iptables -t nat -D DOCKER 6926
===解析下命名空间=======================
[root@minion ~]# ip addr
3: docker_gwbridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:93:81:c9:8d brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 scope global docker_gwbridge
valid_lft forever preferred_lft forever
inet6 fe80::42:93ff:fe81:c98d/64 scope link
valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:d5:cc:f2:db brd ff:ff:ff:ff:ff:ff
inet 10.0.78.1/24 scope global docker0
valid_lft forever preferred_lft forever
10: vethd59e050@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker_gwbridge state UP group default
link/ether 22:8d:34:2b:9a:c6 brd ff:ff:ff:ff:ff:ff link-netnsid 1
inet6 fe80::208d:34ff:fe2b:9ac6/64 scope link
valid_lft forever preferred_lft forever
[root@minion ~]# ip netns exec ingress_sbox ip addr
7: eth0@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
inet 10.255.0.4/16 scope global eth0 ---沙箱容器(netwok namespace)
9: eth1@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
inet 172.17.0.2/16 scope global eth1
[root@minion ~]# ip netns exec 1-ystd5xxiui ip addr
2: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
link/ether aa:cb:0d:a4:a6:b9 brd ff:ff:ff:ff:ff:ff
inet 10.255.0.1/16 scope global br0
6: vxlan1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master br0 state UNKNOWN group default
link/ether aa:cb:0d:a4:a6:b9 brd ff:ff:ff:ff:ff:ff link-netnsid 0
8: veth2@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master br0 state UP group default
link/ether d2:04:e3:eb:fc:52 brd ff:ff:ff:ff:ff:ff link-netnsid 1
[root@minion ~]# ip netns
1-ystd5xxiui (id: 0)
ingress_sbox (id: 1)
其中master同
[root@minion ~]# docker network inspect ingress
[
{
"Name": "ingress",
"Id": "ystd5xxiuiuqjbpn76bbzcaws",
"Created": "2019-05-20T18:35:25.014017659+08:00",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.255.0.0/16",
"Gateway": "10.255.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Containers": {
"ingress-sbox": {
"Name": "ingress-endpoint",
"EndpointID": "081aaac8d188d9cb4c190bbb5863be933dcc2b98cde071dce9b035c4ea6df957",
"MacAddress": "02:42:0a:ff:00:04",
"IPv4Address": "10.255.0.4/16", -----注意这个ip
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "4096"
},
"Labels": {},
"Peers": [
{
"Name": "master.wxy-b33a341ba33b",
"IP": "188.131.210.113"
},
{
"Name": "minion.wxy-47481bf33513",
"IP": "106.13.146.3"
}
]
}
]
[root@master ~]# docker network inspect ingress
[
{
"Name": "ingress",
"Id": "ystd5xxiuiuqjbpn76bbzcaws",
"Created": "2019-05-20T18:34:27.512773865+08:00",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.255.0.0/16",
"Gateway": "10.255.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Containers": {
"ingress-sbox": {
"Name": "ingress-endpoint",
"EndpointID": "77cc852efe15111e141ae78ee9c29ccf88637599068044677d64f0107f5db78a",
"MacAddress": "02:42:0a:ff:00:03",
"IPv4Address": "10.255.0.3/16",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "4096"
},
"Labels": {},
"Peers": [
{
"Name": "master.wxy-b33a341ba33b",
"IP": "188.131.210.113"
},
{
"Name": "minion.wxy-47481bf33513",
"IP": "106.13.146.3"
}
]
}
]
说明：ingress-sbox其实是一个容器，就叫沙箱容器把，深层次来说就是一个namespace，这个ns的名字就叫ingress_sbox，这样之后加入该network的容器实际上就是共享了这个namespace（network类型），当然每一个沙箱容器都还会再配备一个namespace，用来承载vxlan相关的内容，即overlay网络通信功能的namespace。
验证下这个network下是否可以互通呢？
master的沙箱：10.255.0.3
minomon的沙箱：10.255.0.4
[root@minion ~]# ip netns exec ingress_sbox ping 10.255.0.3
PING 10.255.0.3 (10.255.0.3) 56(84) bytes of data.
64 bytes from 10.255.0.3: icmp_seq=1 ttl=64 time=9.64 ms
64 bytes from 10.255.0.3: icmp_seq=2 ttl=64 time=9.55 ms
64 bytes from 10.255.0.3: icmp_seq=3 ttl=64 time=9.75 ms
[root@minion ~]# ip netns exec 1-ystd5xxiui bridge fdb show dev vxlan1
02:42:0a:ff:00:03 master br0
aa:cb:0d:a4:a6:b9 vlan 1 master br0 permanent
aa:cb:0d:a4:a6:b9 master br0 permanent
02:42:0a:ff:00:03 dst 188.131.210.113 link-netnsid 0 self permanent --这个正是master沙箱的
第二步：新起容器加入到ingress网络中
/usr/bin/docker-current: Error response from daemon: Could not attach to network ingress: rpc error: code = 7 desc = network ingress not manually attachable.
第三步：