nvgre

GRE RFC2784 工作原理

Structure of a GRE Encapsulated Packet

   A GRE encapsulated packet has the form:

    ---------------------------------
    |                               |
    |       Delivery Header         |
    |                               |
    ---------------------------------
    |                               |
    |       GRE Header              |
    |                               |
    ---------------------------------
    |                               |
    |       Payload packet          |
    |                               |
    ---------------------------------

   This specification is generally concerned with the structure of the
   GRE header, although special consideration is given to some of the
   issues surrounding IPv4 payloads.

GRE Header

   The GRE packet header has the form:

    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |C|       Reserved0       | Ver |         Protocol Type         |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |      Checksum (optional)      |       Reserved1 (Optional)    |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Key and Sequence Number Extensions to GRE RFC2890

 

Extensions to GRE Header

   The GRE packet header[1] has the following format:

     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |C|       Reserved0       | Ver |         Protocol Type         |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |      Checksum (optional)      |       Reserved1 (Optional)    |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   The proposed GRE header will have the following format:

    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |C| |K|S| Reserved0       | Ver |         Protocol Type         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |      Checksum (optional)      |       Reserved1 (Optional)    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                         Key (optional)                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                 Sequence Number (Optional)                    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+


     Key Present (bit 2)

     If the Key Present bit is set to 1, then it indicates that the
     Key field is present in the GRE header.  Otherwise, the Key
     field is not present in the GRE header.

     Sequence Number Present (bit 3)

     If the Sequence Number Present bit is set to 1, then it
     indicates that the Sequence Number field is present.
     Otherwise, the Sequence Number field is not present in the GRE
     header.

     The Key and the Sequence Present bits are chosen to be
     compatible with RFC 1701 [2].

NVGRE RFC 7637

NVGRE是一个由RFC 2784定义和RFC 2890扩展的通道协议。 微软的blog

Outer Ethernet Header:
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                (Outer) Destination MAC Address                |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |(Outer)Destination MAC Address |  (Outer)Source MAC Address    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                  (Outer) Source MAC Address                   |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |Optional Ethertype=C-Tag 802.1Q| Outer VLAN Tag Information    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |       Ethertype 0x0800        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   Outer IPv4 Header:
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |Version|  HL   |Type of Service|          Total Length         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |         Identification        |Flags|      Fragment Offset    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  Time to Live | Protocol 0x2F |         Header Checksum       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                      (Outer) Source Address                   |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                  (Outer) Destination Address                  |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

GRE Header: 

   key was set to 1

   Protocol Type field in the GRE header is set to 0x6558

   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |0| |1|0|   Reserved0     | Ver |   Protocol Type 0x6558        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |               Virtual Subnet ID (VSID)        |    FlowID     |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Inner Ethernet Header 
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                (Inner) Destination MAC Address                |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |(Inner)Destination MAC Address |  (Inner)Source MAC Address    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                  (Inner) Source MAC Address                   |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |       Ethertype 0x0800        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Inner IPv4 Header:
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |Version|  HL   |Type of Service|          Total Length         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |         Identification        |Flags|      Fragment Offset    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  Time to Live |    Protocol   |         Header Checksum       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                       Source Address                          |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                    Destination Address                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                    Options                    |    Padding    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                      Original IP Payload                      |
   |                                                               |
   |                                                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

               Figure 1: GRE Encapsulation Frame Format

The best starting place is to first layout the addressing scheme for IP addresses and subnets that you'd like to virtualize.  When configuring Hyper-V Network Virtualization, there are two types of IP Addresses that you'll be interacting with:

• Provider Addresses (PA) - these are unique IP addresses assigned to each Hyper-V host that are routable across the physical network infrastructure.  I like to think of "PA" addresses as "Physical Addresses", because they are assigned to physical Hyper-V hosts.  Each Hyper-V host requires at least one PA to be assigned.
   
• Customer Addresses (CA) - these are unique IP addresses assigned to each Virtual Machine that will be participating on a virtualized network.  I like to think of "CA" addresses as "Container Addresses", because they are the IP Addresses assigned to each VM "container" for use by the guest operating system running inside that VM.  Using NVGRE, multiple CA's for VMs running on a Hyper-V host can be tunneled using a single PA on that Hyper-V host.  CA's must be unique across all VMs on the same virtualized network, but CA's do not need to be unique across virtualized networks (such as in multi-tenant scenarios where each customer's VMs are isolated on separate virtualized networks).

Let's look at a simple example of NVGRE with two Hyper-V hosts using PA's and CA's:

In this example, you'll note that each Hyper-V host is assigned one PA address ( e.g., 192.168.x.x ) used for tunneling NVGRE traffic across two physical subnets ( e.g., 192.168.1.x/24 and 192.168.2.x/24 ) on the physical network.  In addition, each VM is assigned a CA address ( e.g., 10.x.x.x ) that is unique within each virtualized network and is tunneled inside the NVGRE tunnel between hosts. 

To separate the traffic between the two virtualized networks, the GRE headers on the tunneled packets include a GRE Key that provides a unique Virtual Subnet ID ( e.g., 5001 and 6001 ) for each virtualized network. 

Based on this configuration, we have two virtualized networks ( e.g., the "Red" network and the "Blue" network ) that are isolated from one another as separate IP networks and extended across two physical Hyper-V hosts located on two different physical subnets.

Once you have the following defined for your environment in a worksheet, you're ready to move on to the next steps in configuring Hyper-V Network Virtualization:

• PA's for each Hyper-V Host
• CA's for each Virtual Machine
• Virtual Subnet ID's for each subnet to be virtualized

Neutron 理解 (3): Open vSwitch + GRE/VxLAN 组网 [Netruon Open vSwitch + GRE/VxLAN Virutal Network]

Tunneling And Network Virtualization: NVGRE, VXLAN

 

Demo:

如何在Linux环境创建GRE Tunnel

Using GRE Tunnels with Open vSwitch

普通的GRE  应该是需要arp代理吗？

 script： gre.sh 

#!/bin/bash
# sudo apt install bridge-utils
REMOTE_IP=$1
SUBNET=$2  # HOST1: 192.168.0.1, HOST2: 192.169.0.1
GREIP=$3   # HOST1: 10.10.10.1, HOST2: 10.10.10.2
R_GREIP=$4 # HOST1: 10.10.10.2, HOST2: 10.10.10.1
DEV=$5
LOCAL_IP=`ip addr show  $DEV| awk '/inet /{split($2,a,"/"); print a[1]}'`
sudo ip tunnel add gre1 mode gre remote $REMOTE_IP local $LOCAL_IP ttl 255
sudo ip link set gre1 up

sudo ip addr add $GREIP/24 dev gre1
# sudo ip route add ${SUBNET%.*}.0/24 via $R_GREIP dev gre1  # 不能工作
sudo ip route add ${SUBNET%.*}.0/24 dev gre1 #为 gre 添加ip
sudo echo 1 > /proc/sys/net/ipv4/ip_forward #让服务器支持转发
# HOST1
sudo iptables -t nat -A POSTROUTING -d  ${SUBNET%.*}.0/24 -j SNAT --to $GREIP#否则访问 ${SUBNET%.*}.0/24网段不通

# HOST2
iptables -t nat -A POSTROUTING -s $GREIP -d ${SUBNET%.*}.0/24  -j SNAT --to $SUBNET #否则192.168.1.X等机器访问10.1.1.x网段不通
iptables -A FORWARD -s $GREIP -m state --state NEW -m tcp -p tcp --dport 3306 -j DROP #禁止直接访问线上的3306，防止内网被破

sudo brctl addbr br1  # sudo ifconfig br1 192.169.0.7/24
sudo ip link set br1 up
# sudo brctl addif br1 gre1 # 不能工作

sudo ip link add type veth
sudo ifconfig veth0 ${SUBNET%.*}.7/24 up
sudo ifconfig veth0 mtu 1450
sudo ifconfig veth1 up
sudo ifconfig veth1 mtu 1450
sudo brctl addif br1 veth1

ip route show

on host 1:   $ ./gre.sh 10.0.0.52  192.168.0.1 10.10.10.1 10.10.10.2 ens3

on host 2:   $ ./gre.sh 10.0.0.32  192.169.0.1 10.10.10.2 10.10.10.1 ens3

on host 1

sudo ovs-vsctl add-br br0
sudo ovs-vsctl add-port br0 tep0 -- set interface tep0 type=internal
sudo ifconfig tep0 192.168.200.20 netmask 255.255.255.0
sudo ovs-vsctl add-br br2
sudo ovs-vsctl add-port br2 gre0 -- set interface gre0 type=gre options:remote_ip=192.168.200.21
route

# ip link add br0 type bridge

sudo ip tuntap add mode tap

sudo ifconfig tap0 192.168.200.20 netmask 255.255.255.0

sudo ip link set tap0 up

sudo ip link set br1 up

sudo brctl addif br1 tap0

sudo brctl addif br1 ens3  # 该命令会导致网络访问不了

sudo ip link add type veth
sudo ifconfig veth0 192.167.0.6/24 up
sudo ifconfig veth0 mtu 1450
sudo ifconfig veth1 up
sudo ifconfig veth1 mtu 1450
sudo ovs-vsctl add-port br2 veth1

$ sudo ovs-vsctl add-port br0 ens3  # 该命令会导致网络访问不了

on host 2

sudo ovs-vsctl add-br br0
sudo ovs-vsctl add-port br0 tep0 -- set interface tep0 type=internal
sudo ifconfig tep0 192.168.200.21 netmask 255.255.255.0
sudo ovs-vsctl add-br br2
sudo ovs-vsctl add-port br2 gre0 -- set interface gre0 type=gre options:remote_ip=192.168.200.20
route

$ sudo ovs-vsctl show
ffb98c3f-a7a4-4287-b84a-c7c2b2616c72
    Bridge "br0"
        Port "tep0"
            Interface "tep0"
                type: internal
        Port "br0"
            Interface "br0"
                type: internal
    Bridge "br2"
        Port "br2"
            Interface "br2"
                type: internal
        Port "gre0"
            Interface "gre0"
                type: gre
                options: {remote_ip="192.168.200.21"}
    ovs_version: "2.5.2"
$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         localhost       0.0.0.0         UG    0      0        0 ens3
10.0.0.0        *               255.255.255.0   U     0      0        0 ens3
169.254.169.254 localhost       255.255.255.255 UGH   0      0        0 ens3
192.168.200.0   *               255.255.255.0   U     0      0        0 tep0

$ sudo ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether fa:16:3e:88:b0:29 brd ff:ff:ff:ff:ff:ff
3: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1
    link/ether b6:98:ba:ee:7d:b6 brd ff:ff:ff:ff:ff:ff
4: br2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1
    link/ether a2:58:66:5a:94:4a brd ff:ff:ff:ff:ff:ff
5: br0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1
    link/ether 3e:2f:8d:26:56:47 brd ff:ff:ff:ff:ff:ff
6: tep0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1
    link/ether 62:32:8c:1d:2b:99 brd ff:ff:ff:ff:ff:ff
7: gre0@NONE: <NOARP> mtu 1476 qdisc noop state DOWN mode DEFAULT group default qlen 1
    link/gre 0.0.0.0 brd 0.0.0.0
8: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
9: gre_sys@NONE: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65490 qdisc pfifo_fast master ovs-system state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether a6:ee:6f:a2:0e:22 brd ff:ff:ff:ff:ff:ff

$ sudo ip addr show                                                                                 [7/841]
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast state UP group default qlen 1000
    link/ether fa:16:3e:88:b0:29 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.54/24 brd 10.0.0.255 scope global ens3
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe88:b029/64 scope link
       valid_lft forever preferred_lft forever
3: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1
    link/ether b6:98:ba:ee:7d:b6 brd ff:ff:ff:ff:ff:ff
4: br2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1
    link/ether a2:58:66:5a:94:4a brd ff:ff:ff:ff:ff:ff
5: br0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1
    link/ether 3e:2f:8d:26:56:47 brd ff:ff:ff:ff:ff:ff
6: tep0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1
    link/ether 62:32:8c:1d:2b:99 brd ff:ff:ff:ff:ff:ff
    inet 192.168.200.20/24 brd 192.168.200.255 scope global tep0
       valid_lft forever preferred_lft forever
    inet6 fe80::6032:8cff:fe1d:2b99/64 scope link
       valid_lft forever preferred_lft forever
7: gre0@NONE: <NOARP> mtu 1476 qdisc noop state DOWN group default qlen 1
    link/gre 0.0.0.0 brd 0.0.0.0
8: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
9: gre_sys@NONE: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65490 qdisc pfifo_fast master ovs-system state UNKNOWN group default qlen 1000
    link/ether a6:ee:6f:a2:0e:22 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a4ee:6fff:fea2:e22/64 scope link
       valid_lft forever preferred_lft forever

$ sudo ovs-vsctl add-port br0 ens3   # 该命令会导致网络访问不了

sudo ip link add type veth
sudo ifconfig veth0 192.167.0.6/24 up
sudo ifconfig veth0 mtu 1450
sudo ifconfig veth1 up
sudo ifconfig veth1 mtu 1450
sudo ovs-vsctl add-port br2 veth1

  

$ ip link help
...
TYPE := { vlan | veth | vcan | dummy | ifb | macvlan | macvtap |
bridge | bond | ipoib | ip6tnl | ipip | sit | vxlan |
gre | gretap | ip6gre | ip6gretap | vti | nlmon |
bond_slave | ipvlan | geneve | bridge_slave | vrf }

深入理解 GRE tunnel GRE 与IPIP的区别。 ipip tunnel 是端对端的，通信也就只能是点对点的，而 GRE tunnel 却可以进行多播。

 该ppt中内置了GRE和IPIP的包， 可供大家分析。