聊一聊sockmap 以及ebpf

　　之前聊过tcpdump 抓包原理，tcpdump使用packet 抓包，使用packet_map 完成零拷贝。但是这个零拷贝也有点假，何为假呢？从网卡到内存走的dma，哪能不能直接从dma拷贝到用户空间呢？?  使用dpdk直接从网卡中轮询数据？

如果使用现有的tcpip协议栈，反正内核态需要处理网络数据，那就将数据线从网卡缓存dma拷贝到内核态buff吧，然后再来一个所谓的零拷贝到用户空间吧。。。

在https://blog.cloudflare.com/sockmap-tcp-splicing-of-the-future/ 这篇文章中有相关介绍

怎样使用sockmap

SOCKMAP or specifically "BPF_MAP_TYPE_SOCKMAP", is a type of an eBPF map

　　bpf map 一个古老的东西，以前也没人管他，结构现在又整出新东西来了。对于之前使用bpf也就只有在packet socket 抓包过滤协议报文时才会使用，tcpdump -dd   xxx 找出对应的bpf字节掩码或者tcpdump -d  xxx 生成指令集

比如

root@fp:~# tcpdump -d -i ens33 tcp and port 80
(000) ldh      [12]
(001) jeq      #0x86dd          jt 2    jf 8
(002) ldb      [20]
(003) jeq      #0x6             jt 4    jf 19
(004) ldh      [54]
(005) jeq      #0x50            jt 18    jf 6
(006) ldh      [56]
(007) jeq      #0x50            jt 18    jf 19
(008) jeq      #0x800           jt 9    jf 19
(009) ldb      [23]
(010) jeq      #0x6             jt 11    jf 19
(011) ldh      [20]
(012) jset     #0x1fff          jt 19    jf 13
(013) ldxb     4*([14]&0xf)
(014) ldh      [x + 14]
(015) jeq      #0x50            jt 18    jf 16
(016) ldh      [x + 16]
(017) jeq      #0x50            jt 18    jf 19
(018) ret      #262144
(019) ret      #0
root@fp:~# tcpdump -dd -i ens33 tcp and port 80
{ 0x28, 0, 0, 0x0000000c },
{ 0x15, 0, 6, 0x000086dd },
{ 0x30, 0, 0, 0x00000014 },
{ 0x15, 0, 15, 0x00000006 },
{ 0x28, 0, 0, 0x00000036 },
{ 0x15, 12, 0, 0x00000050 },
{ 0x28, 0, 0, 0x00000038 },
{ 0x15, 10, 11, 0x00000050 },
{ 0x15, 0, 10, 0x00000800 },
{ 0x30, 0, 0, 0x00000017 },
{ 0x15, 0, 8, 0x00000006 },
{ 0x28, 0, 0, 0x00000014 },
{ 0x45, 6, 0, 0x00001fff },
{ 0xb1, 0, 0, 0x0000000e },
{ 0x48, 0, 0, 0x0000000e },
{ 0x15, 2, 0, 0x00000050 },
{ 0x48, 0, 0, 0x00000010 },
{ 0x15, 0, 1, 0x00000050 },
{ 0x6, 0, 0, 0x00040000 },
{ 0x6, 0, 0, 0x00000000 },

好久不使用了。

　　ebpf是由bpf延伸过来；eBPF支持在用户态将C语言编写的一小段“内核代码”注入到内核中运行，注入时要先用llvm编译得到使用BPF指令集的elf文件，然后从elf文件中解析出可以注入内核的部分，最后用bpf_load_program方法完成注入。 用户态程序和注入到内核中的程序通过共用一个位于内核中map实现通信。为了防止注入的代码导致内核崩溃，eBPF会对注入的代码进行严格检查，拒绝不合格的代码的注入。

BCC是一个python库，实现了map创建、代码编译、解析、注入等操作，使开发人员只需聚焦于用C语言开发要注入的内核代码

• ebpf有自己的字节码语言
• ebpf运行在内核，但是不能随心所欲的访问内核的内存，需要通过内核提供的函数去受限制严格的访问

• 可以和用户空间程序通过bpf映射通信
• 增加了名为bpf的系统调用，为用户态程序提供与内核中的eBPF进行交互的途径
• 相对于 BPF，eBPF 带来的改变可谓是革命性的：
  • 　　一方面，不再仅仅是进行报文复制和过滤，网络方面可以切入到更深的层次，它已经为内核追踪(Kernel Tracing)、应用性能调优/监控、流控(Traffic Control)等领域带来了激动人心的变革；
  • 　　另一方面，在接口的设计以及易用性上，eBPF 也有了较大的改进。
   

对于ebpf的使用：参考如下文章

1. LWN.net: A thorough introduction to eBPF

2. LWN.net: An introduction to the BPF Compiler Collection

3. LWN.net: Some advanced BCC topics

4. LWN.net: Using user-space tracepoints with BPF

目ebpf的入口函数 

#include <linux/bpf.h>
int bpf(int cmd, union bpf_attr *attr, unsigned int size)

kernel/bpf/syscall.c bpf的系统调用
include/uapi/linux/bpf.h bpf系统调用的头文件

• 我们知道c/ebpf是通过在内核函数加入hook机制，那么对于通过ebpf实现的sockmap其hook是在 哪里实现的呢？

　　根据之前对linux tcp/ip协议栈的分析：可知kernel 通过sk_data_ready  唤醒user_process 读取内核协议栈的数据，此时会出现一次上下文切换。

而sockmap此时是通过一种Stream Parser (strparser)的机制完成，将数据包转发给ebpf处理，而ebpf实现了数据流的重定向。

即原本通过sk_data_reay 去wakeup user_process 用户进程的， 后续只是执行别的操作， 在内核里面替换了回调函数的具体实现：

sk->sk_data_ready = smap_data_ready;

static int smap_init_sock(struct smap_psock *psock,
              struct sock *sk)
{
    static const struct strp_callbacks cb = {
        .rcv_msg = smap_read_sock_strparser,
        .parse_msg = smap_parse_func_strparser,
        .read_sock_done = smap_read_sock_done,
    };

    return strp_init(&psock->strp, sk, &cb);
}

static void smap_start_sock(struct smap_psock *psock, struct sock *sk)
{
    if (sk->sk_data_ready == smap_data_ready)
        return;
//保存原有的回调唤醒wakeup函数
    psock->save_data_ready = sk->sk_data_ready;
    psock->save_write_space = sk->sk_write_space;
    psock->save_state_change = sk->sk_state_change;
//替换wakeup 回调
    sk->sk_data_ready = smap_data_ready;
    sk->sk_write_space = smap_write_space;
    sk->sk_state_change = smap_state_change;
    psock->strp_enabled = true;
}

详细看一下其sockmap wakeup回调

其回调函数需要hold on sock  同时关闭bh，

/* Called with lock held on socket */
static void smap_data_ready(struct sock *sk)
{
    struct smap_psock *psock;

    rcu_read_lock();
    psock = smap_psock_sk(sk);
    if (likely(psock)) {
        write_lock_bh(&sk->sk_callback_lock);
        strp_data_ready(&psock->strp);
        write_unlock_bh(&sk->sk_callback_lock);
    }
    rcu_read_unlock();
}

/* Lower sock lock held */
void strp_data_ready(struct strparser *strp)
{
    if (unlikely(strp->stopped))
        return;

    /* This check is needed to synchronize with do_strp_work.
     * do_strp_work acquires a process lock (lock_sock) whereas
     * the lock held here is bh_lock_sock. The two locks can be
     * held by different threads at the same time, but bh_lock_sock
     * allows a thread in BH context to safely check if the process
     * lock is held. In this case, if the lock is held, queue work.
     */
    if (sock_owned_by_user(strp->sk)) {
        queue_work(strp_wq, &strp->work);
        return;
    }

    if (strp->paused)
        return;

    if (strp->need_bytes) {
        if (strp_peek_len(strp) >= strp->need_bytes)
            strp->need_bytes = 0;
        else
            return;
    }

    if (strp_read_sock(strp) == -ENOMEM)
        queue_work(strp_wq, &strp->work);
}

其回调函数是为了启动work_queue，其队列初始化实在strpinit是完成的，其回调函数为：do_strp_work

static void do_strp_work(struct strparser *strp)
{
    read_descriptor_t rd_desc;

    /* We need the read lock to synchronize with strp_data_ready. We
     * need the socket lock for calling strp_read_sock.
     */
    strp->cb.lock(strp);

    if (unlikely(strp->stopped))
        goto out;

    if (strp->paused)
        goto out;

    rd_desc.arg.data = strp;

    if (strp_read_sock(strp) == -ENOMEM)//最主要的处理函数
        queue_work(strp_wq, &strp->work);

out:
    strp->cb.unlock(strp);
}

其实现功能：

1、检查是否有ops->read_sock回调函数，目前只有tcp支持ops->read_sock回调：tcp_read_sock其注释如下，主要实现了sendfile 零copy功能

/*
 This routine provides an alternative to tcp_recvmsg() for routines
 * that would like to handle copying from skbuffs directly in 'sendfile'
 * fashion.
 * Note:
 *    - It is assumed that the socket was locked by the caller.
 *    - The routine does not block.
 *    - At present, there is no support for reading OOB data
 *      or for 'peeking' the socket using this routine
 *      (although both would be easy to implement).
 */

2、执行strp->cb.read_sock_done-  根据其初始化实现不同而不同

* Called with lock held on lower socket */
static int strp_read_sock(struct strparser *strp)
{
    struct socket *sock = strp->sk->sk_socket;
    read_descriptor_t desc;

    if (unlikely(!sock || !sock->ops || !sock->ops->read_sock))
        return -EBUSY;

    desc.arg.data = strp;
    desc.error = 0;
    desc.count = 1; /* give more than one skb per call */

    /* sk should be locked here, so okay to do read_sock */
    sock->ops->read_sock(strp->sk, &desc, strp_recv);

    desc.error = strp->cb.read_sock_done(strp, desc.error);

    return desc.error;
}

 tcp协议栈调用strp->cb流程： sk_data_ready -->strp_read_sock -->sock->ops->read_sock(strp->sk, &desc, strp_recv)

关于Stream Parser (strparser)的介绍见：https://www.kernel.org/doc/Documentation/networking/strparser.txt  

　　The stream parser (strparser) is a utility that parses messages of an application layer protocol running over a data stream. The stream parser works in conjunction with an upper layer in the kernel to provide kernel support for application layer messages. For instance, Kernel
Connection Multiplexor (KCM) uses the Stream Parser to parse messages using a BPF program. The strparser works in one of two modes: receive callback or general mode.

 eBPF-可编程内核调试利器

它提供了一种在不修改内核代码的情况下，可以灵活修改内核处理策略的方法：要使用 eBPF 提供的能力，需要完成 “编写BPF代码-编译成字节码-注入内核-获取结果-展示” 这一整套流程，而且会非常复杂

目前有ebpf工具，比如bcc工具集，说道工具集就会提到 Brendan Gregg，他是perfermance届的牛人。他开发了很多perf相关的工具和脚本：perf_events、perf-tools、bcc、Flame Graphs

bcc 是通过 Python 编写的一个 eBPF 工具集，对于bcc 工具集安装使用就不多说了，

之前的bcc代码中我们知道其程序是分为两部分的，一部分是C语言，另一部分是基于Python的。本篇是关于Ｃ语言部分的。

事件和参数

1.1     kprobes

使用kprobe的语法是:

kprobe__kernel_function_name

其中kprobe__是前缀，用于给内核函数创建一个kprobe(内核函数调用的动态跟踪)。也可通过C语言函数定义一个C函数，然后使用python的BPF.attach_kprobe()来关联到内核函数。

例如：

int kprobe__tcp_v4_connect(struct pt_regs *ctx, struct sock *sk)

            其中参数struct pt_regs *ctx是寄存器和BPF文件

            sock *sk是tcp_v4_connect的第一个参数。

1.2     kretprobes

kretprobes动态跟踪内核函数的返回，语法如下：

kretprobe__kernel_function_name，前缀是kretprobe__。也可以使用python的BPF.attach_kretprobe()来关联Ｃ函数到内核函数。

            例如：

int kretprobe__tcp_v4_connect(struct pt_regs *ctx)

{     int ret = PT_REGS_RC(ctx);     [...]

}

1.3     Tracepoints

语法如下：TRACEPOINT_PROBE(category,event)

TRACEPOINT_PROBE是一个宏， tracepiont定义的方式是categroy:event，

            可以的参数是通过结构体args获取的，获取参数相关格式的方法是cat文件：

/sys/kernel/debug/tracing/events/category/event/format

            结构体args可以在函数中使用例如：

TRACEPOINT_PROBE(random, urandom_read) {    

// args is from /sys/kernel/debug/tracing/events/random/urandom_read/format     bpf_trace_printk("%d\n", args->got_bits);     return 0;

}

1.4     uprobes

通过python的BPF.attach_uprobe()可以将普通Ｃ函数关联到uprobe探针。

参数可以通过PT_REGS_PARM宏来检测。

            程序本身名字使用宏PT_REGS_PARM1,第一个参数使用宏PT_REGS_PARM2。

1.5     uretprobes

同uprobes，只不过该探针是在函数返回时候触发。

     返回的值通过PT_REGS_RC(ctx)获取，例如：

BPF_HISTOGRAM(dist); int count(struct pt_regs *ctx) {     dist.increment(PT_REGS_RC(ctx));     return 0;

}

在直方图dist中，根据增加返回键对应的值。

1.6     USDT probes

User Statically-Defined Tracing用户静态定义的探针，会在应用和库中定义用来提供用户级别追踪。BPF中提供对USDT的支持方法是enable_probe。

使用方法同其他probes，定义C函数，然后通过USDT.enable_probe()来关联USDT　probe。

            参数可以通过bpf_usdt_readarg(index,ctx,&addr)来读取。

            例如：

int do_trace(struct pt_regs *ctx) {   

 uint64_t addr;    

char path[128];    

bpf_usdt_readarg(6, ctx, &addr);    

bpf_probe_read(&path, sizeof(path), (void *)addr);     bpf_trace_printk("path:%s\n", path);    

return 0;

};

2.   数据

上节是关于事件和参数的，这边是看下如何获取所监控函数中数据。

2.1     bpf_probe_read

语法：int bpf_probe_read(void *dst, int size, const void *src)

如果成功返回０.

            该函数将内容复制到BPF栈中，用于后续使用。为了安全起见，所有内存读取都通过bpf_probe_read函数。

2.2     bpf_probe_read_str

语法：int bpf_probe_read_str(void *dst, int size, const void *src)

复制NULL结尾的字符串到BPF栈，用于后续使用。

2.3     bpf_ktime_get_ns

语法：u64 bpf_ktime_get_ns(void)

获取当前时间，以纳秒方式。

2.4     bpf_get_current_pid_tgid

语法：u64 bpf_get_current_pid_tgid(void)

            低３２位为当前进程ID,高32位是组ID。

2.5     bpf_get_current_uid_gid

语法：u64 bpf_get_current_uid_gid(void)

返回用户ID和组ID.

2.6     bpf_get_current_common

语法：bpf_get_current_comm(char *buf, int size_of_buf)

用当前进程名字填充第一个参数地址。

2.7     bpf_get_current_task

返回指向当前task_struct对象的指针。可用于计算CPU在线时间，内核线程，运行队列和其他相关信息。

2.8     bpf_log2l

返回提供值的log-2结果，经常用于创建直方图索引构建直方图。

2.9     bpf_get_prandom_u32

返回一个无符号32位伪随机值。

3.   调试

3.1     bpf_override_return

语法：int bpf_override_return(struct pt_regs *, unsigned long rc)

            用于关联到函数入口的kprobe，忽略要执行的函数，返回rc，用于目标的故障注入。

            这个函数在允许故障注入的kprobe白名单中才能工作。白名单在内核代码树中会标记BPF_ALOW_ERROR_INJECTION()。

4.   输出

4.1     bpf_trace_printk

语法：int bpf_trace_printk(const char *fmt, int fmt_size, ...)

            简单的内核printf函数，输出到/sys/kernel/debug/tracing/trace_pipe。这个在之前有描述过，其最多３个参数，智能输出%s，而且trace_pipe是全局共享的并发输出会有问题，生产中工具使用BPF_PERF_OUTPUT()函数。

4.2     BPF_PERF_OUTPUT

通过perf ring buffer创建BPF表，将定义的事件数据输出。这个是将数据推送到用户态的建议方法。

例如：

struct data_t {

    u32 pid;

    u64 ts;

    char comm[TASK_COMM_LEN];

};

BPF_PERF_OUTPUT(events);

 

int hello(struct pt_regs *ctx) {

    struct data_t data = {};

 

    data.pid = bpf_get_current_pid_tgid();

    data.ts = bpf_ktime_get_ns();

    bpf_get_current_comm(&data.comm, sizeof(data.comm));

 

    events.perf_submit(ctx, &data, sizeof(data));

 

    return 0;

}

代码中的输出表是events,数据通过events.perf_submit来推送。

4.3     perf_submit

语法：int perf_submit((void *)ctx, (void *)data, u32 data_size)

该函数是BPF_PERF_OUTPUT表的方法，将定义的事件数据推到用户

 

#!/usr/bin/python

from __future__ import print_function
from bcc import BPF
from bcc.utils import printb

bpf_text = """
#include <uapi/linux/ptrace.h>
#include <net/sock.h>
#include <bcc/proto.h>
#include <uapi/linux/icmp.h>
#include <linux/icmp.h>
#include <uapi/linux/ip.h>
#include <linux/ip.h>


int icmp_rcv_cb(struct pt_regs *ctx, struct sk_buff *skb)
{
            struct icmphdr *icmph ;
            struct iphdr *iph = ip_hdr(skb);
            //bpf_trace_printk("ipsrc:%x  ipdst:%x \n",iph->saddr, iph->daddr);
            icmph = (struct icmphdr *)skb->data;
            bpf_trace_printk("devname:%s ----- icmp_type:%d  \n",
                    skb->dev->name,  icmph->type);
            return 0;
};

"""
# initialize BPF
b = BPF(text=bpf_text)
b.attach_kprobe(event="icmp_rcv", fn_name="icmp_rcv_cb")
#end format output
while 1:
    # Read messages from kernel pipe
    (task, pid, cpu, flags, ts, msg) = b.trace_fields()
    print("task:%s pid: %d %s " % (task, pid, msg))
#b.trace_print()

上述python脚本作用是在内核协议栈icm_rcv加入hook。

执行结果为

root@test: ./icmp.py
task:<idle> pid: 0 devnamenp1s0 ----- icmp_type:8
task:<idle> pid: 0 devname: ----- icmp_type:0
task:<idle> pid: 0 devnamenp1s0 ----- icmp_type:8
task:<idle> pid: 0 devname: ----- icmp_type:0
task:<idle> pid: 0 devnamenp1s0 ----- icmp_type:8
task:<idle> pid: 0 devname: ----- icmp_type:0
task:ksoftirqd/1 pid: 16 devnamenp1s0 ----- icmp_type:8
task:<idle> pid: 0 devname: ----- icmp_type:0
task:<idle> pid: 0 devnamenp1s0 ----- icmp_type:8
task:<idle> pid: 0 devname: ----- icmp_type:0
^CTraceback (most recent call last):
  File "./icmp.py", line 35, in <module>
    (task, pid, cpu, flags, ts, msg) = b.trace_fields()
  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 1129, in trace_fields
    line = self.trace_readline(nonblocking)
  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 1161, in trace_readline
    line = trace.readline(1024).rstrip()
KeyboardInterrupt

#!/usr/bin/python
#
# killsnoop Trace signals issued by the kill() syscall.
#           For Linux, uses BCC, eBPF. Embedded C.
#BPF_PERF_OUTPUT

from __future__ import print_function
from bcc import BPF
from bcc.utils import ArgString, printb
import argparse
from time import strftime

# arguments
examples = """examples:
    ./killsnoop           # trace all kill() signals
    ./killsnoop -x        # only show failed kills
    ./killsnoop -p 181    # only trace PID 181
"""
parser = argparse.ArgumentParser(
    description="Trace signals issued by the kill() syscall",
    formatter_class=argparse.RawDescriptionHelpFormatter,
    epilog=examples)
parser.add_argument("-x", "--failed", action="store_true",
    help="only show failed kill syscalls")
parser.add_argument("-p", "--pid",
    help="trace this PID only")
parser.add_argument("--ebpf", action="store_true",
    help=argparse.SUPPRESS)
args = parser.parse_args()
debug = 0

# define BPF program
bpf_text = """
#include <uapi/linux/ptrace.h>
#include <linux/sched.h>

struct val_t {
   u64 pid;
   int sig;
   int tpid;
   char comm[TASK_COMM_LEN];
};

struct data_t {
   u64 pid;
   int tpid;
   int sig;
   int ret;
   char comm[TASK_COMM_LEN];
};

BPF_HASH(infotmp, u32, struct val_t);
BPF_PERF_OUTPUT(events);

int syscall__kill(struct pt_regs *ctx, int tpid, int sig)
{
    u32 pid = bpf_get_current_pid_tgid();
    FILTER

    struct val_t val = {.pid = pid};
    if (bpf_get_current_comm(&val.comm, sizeof(val.comm)) == 0) {
        val.tpid = tpid;
        val.sig = sig;
        infotmp.update(&pid, &val);
    }

    return 0;
};

int do_ret_sys_kill(struct pt_regs *ctx)
{
    struct data_t data = {};
    struct val_t *valp;
    u32 pid = bpf_get_current_pid_tgid();

    valp = infotmp.lookup(&pid);
    if (valp == 0) {
        // missed entry
        return 0;
    }

    bpf_probe_read(&data.comm, sizeof(data.comm), valp->comm);
    data.pid = pid;
    data.tpid = valp->tpid;
    data.ret = PT_REGS_RC(ctx);
    data.sig = valp->sig;

    events.perf_submit(ctx, &data, sizeof(data));
    infotmp.delete(&pid);

    return 0;
}
"""
if args.pid:
    bpf_text = bpf_text.replace('FILTER',
        'if (pid != %s) { return 0; }' % args.pid)
else:
    bpf_text = bpf_text.replace('FILTER', '')
if debug or args.ebpf:
    print(bpf_text)
    if args.ebpf:
        exit()

# initialize BPF
b = BPF(text=bpf_text)
kill_fnname = b.get_syscall_fnname("kill")
b.attach_kprobe(event=kill_fnname, fn_name="syscall__kill")
b.attach_kretprobe(event=kill_fnname, fn_name="do_ret_sys_kill")

# header
print("%-9s %-6s %-16s %-4s %-6s %s" % (
    "TIME", "PID", "COMM", "SIG", "TPID", "RESULT"))

# process event
def print_event(cpu, data, size):
    event = b["events"].event(data)

    if (args.failed and (event.ret >= 0)):
        return

    printb(b"%-9s %-6d %-16s %-4d %-6d %d" % (strftime("%H:%M:%S").encode('ascii'),
        event.pid, event.comm, event.sig, event.tpid, event.ret))

# loop with callback to print_event
b["events"].open_perf_buffer(print_event)
while 1:
    try:
        b.perf_buffer_poll()
    except KeyboardInterrupt:
        exit()

根据man bpf可以得到如下消息：

Extended BPF Design/Architecture
       eBPF maps are a generic data structure for storage of different data types.  Data types are generally treated as binary blobs, so a user just specifies the size of the key and the size of the value at map-creation time.  In other words, a key/value  for  a
       given map can have an arbitrary structure.

       A user process can create multiple maps (with key/value-pairs being opaque bytes of data) and access them via file descriptors.  Different eBPF programs can access the same maps in parallel.  It's up to the user process and eBPF program to decide what they
       store inside maps.

       There's one special map type, called a program array.  This type of map stores file descriptors referring to other eBPF programs.  When a lookup in the map is performed, the program flow is redirected in-place to the beginning of another eBPF  program  and
       does not return back to the calling program.  The level of nesting has a fixed limit of 32, so that infinite loops cannot be crafted.  At runtime, the program file descriptors stored in the map can be modified, so program functionality can be altered based
       on specific requirements.  All programs referred to in a program-array map must have been previously loaded into the kernel via bpf().  If a map lookup fails, the current program continues its  execution.   See  BPF_MAP_TYPE_PROG_ARRAY  below  for  further
       details.

       Generally,  eBPF  programs  are  loaded  by the user process and automatically unloaded when the process exits.  In some cases, for example, tc-bpf(8), the program will continue to stay alive inside the kernel even after the process that loaded the program
       exits.  In that case, the tc subsystem holds a reference to the eBPF program after the file descriptor has been closed by the user-space program.  Thus, whether a specific program continues to live inside the kernel depends on how it is further attached to a given kernel subsystem after it was loaded via bpf().

       Each eBPF program is a set of instructions that is safe to run until its completion.  An in-kernel verifier statically determines that the eBPF program terminates and is safe to execute.  During verification, the kernel increments reference counts for each
       of the maps that the eBPF program uses, so that the attached maps can't be removed until the program is unloaded.

       eBPF programs can be attached to different events.  These events can be the arrival of network packets, tracing events, classification events by network queueing  disciplines (for eBPF programs attached to a tc(8) classifier), and other types that  may  be
       added in the future.  A new event triggers execution of the eBPF program, which may store information about the event in eBPF maps.  Beyond storing data, eBPF programs may call a fixed set of in-kernel helper functions.

       The same eBPF program can be attached to multiple events and different eBPF programs can access the same map:

           tracing     tracing    tracing    packet      packet     packet
           event A     event B    event C    on eth0     on eth1    on eth2
            |             |         |          |           |          ^
            |             |         |          |           v          |
            --> tracing <--     tracing      socket    tc ingress   tc egress
                 prog_1          prog_2      prog_3    classifier    action
                 |  |              |           |         prog_4      prog_5
              |---  -----|  |------|          map_3        |           |
            map_1       map_2                              --| map_4 |--
Extended BPF Design/Architecture
       eBPF maps are a generic data structure for storage of different data types.  Data types are generally treated as binary blobs, so a user just specifies the size of the key and the size of the value at map-creation time.  In other words, a key/value  for  a
       given map can have an arbitrary structure.

       A user process can create multiple maps (with key/value-pairs being opaque bytes of data) and access them via file descriptors.  Different eBPF programs can access the same maps in parallel.  It's up to the user process and eBPF program to decide what they
       store inside maps.

       There's one special map type, called a program array.  This type of map stores file descriptors referring to other eBPF programs.  When a lookup in the map is performed, the program flow is redirected in-place to the beginning of another eBPF  program  and
       does not return back to the calling program.  The level of nesting has a fixed limit of 32, so that infinite loops cannot be crafted.  At runtime, the program file descriptors stored in the map can be modified, so program functionality can be altered based
       on specific requirements.  All programs referred to in a program-array map must have been previously loaded into the kernel via bpf().  If a map lookup fails, the current program continues its  execution.   See  BPF_MAP_TYPE_PROG_ARRAY  below  for  further
       details.

       Generally,  eBPF  programs  are  loaded  by the user process and automatically unloaded when the process exits.  In some cases, for example, tc-bpf(8), the program will continue to stay alive inside the kernel even after the process that loaded the program
       exits.  In that case, the tc subsystem holds a reference to the eBPF program after the file descriptor has been closed by the user-space program.  Thus, whether a specific program continues to live inside the kernel depends on how it is further attached to
       a given kernel subsystem after it was loaded via bpf().

       Each eBPF program is a set of instructions that is safe to run until its completion.  An in-kernel verifier statically determines that the eBPF program terminates and is safe to execute.  During verification, the kernel increments reference counts for each
       of the maps that the eBPF program uses, so that the attached maps can't be removed until the program is unloaded.

       eBPF programs can be attached to different events.  These events can be the arrival of network packets, tracing events, classification events by network queueing  disciplines (for eBPF programs attached to a tc(8) classifier), and other types that  may  be
       added in the future.  A new event triggers execution of the eBPF program, which may store information about the event in eBPF maps.  Beyond storing data, eBPF programs may call a fixed set of in-kernel helper functions.

       The same eBPF program can be attached to multiple events and different eBPF programs can access the same map:

           tracing     tracing    tracing    packet      packet     packet
           event A     event B    event C    on eth0     on eth1    on eth2
            |             |         |          |           |          ^
            |             |         |          |           v          |
            --> tracing <--     tracing      socket    tc ingress   tc egress
                 prog_1          prog_2      prog_3    classifier    action
                 |  |              |           |         prog_4      prog_5
              |---  -----|  |------|          map_3        |           |
            map_1       map_2                              --| map_4 |--
Arguments
       The  operation  to be performed by the bpf() system call is determined by the cmd argument.  Each operation takes an accompanying argument, provided via attr, which is a pointer to a union of type bpf_attr (see below).  The size argument is the size of the
       union pointed to by attr.

       The value provided in cmd is one of the following:

       BPF_MAP_CREATE
              Create a map and return a file descriptor that refers to the map.

       BPF_MAP_LOOKUP_ELEM
              Look up an element by key in a specified map and return its value.

       BPF_MAP_UPDATE_ELEM
              Create or update an element (key/value pair) in a specified map.

       BPF_MAP_DELETE_ELEM
              Look up and delete an element by key in a specified map.

       BPF_MAP_GET_NEXT_KEY
              Look up an element by key in a specified map and return the key of the next element.

       BPF_PROG_LOAD
              Verify and load an eBPF program, returning a new file descriptor associated with the program.

       The bpf_attr union consists of various anonymous structures that are used by different bpf() commands:

           union bpf_attr {
               struct {    /* Used by BPF_MAP_CREATE */
                   __u32         map_type;
                   __u32         key_size;    /* size of key in bytes */
                   __u32         value_size;  /* size of value in bytes */
                   __u32         max_entries; /* maximum number of entries
                                                 in a map */
               };

               struct {    /* Used by BPF_MAP_*_ELEM and BPF_MAP_GET_NEXT_KEY
                              commands */
                   __u32         map_fd;
                   __aligned_u64 key;
                   union {
                       __aligned_u64 value;
                       __aligned_u64 next_key;
                   };
                   __u64         flags;
               };

               struct {    /* Used by BPF_PROG_LOAD */
                   __u32         prog_type;
                   __u32         insn_cnt;
                   __aligned_u64 insns;      /* 'const struct bpf_insn *' */
                   __aligned_u64 license;    /* 'const char *' */
                   __u32         log_level;  /* verbosity level of verifier */
                   __u32         log_size;   /* size of user buffer */
                   __aligned_u64 log_buf;    /* user supplied 'char *'
                                                buffer */
                   __u32         kern_version;
                                             /* checked when prog_type=kprobe
                                                (since Linux 4.1) */
               };
           } __attribute__((aligned(8)));

   eBPF maps
       Maps are a generic data structure for storage of different types of data.  They allow sharing of data between eBPF kernel programs, and also between kernel and user-space applications.

       Each map type has the following attributes:

       *  type
       *  maximum number of elements
       *  key size in bytes
       *  value size in bytes

       The following wrapper functions demonstrate how various bpf() commands can be used to access the maps.  The functions use the cmd argument to invoke different operations.

       BPF_MAP_CREATE
              The BPF_MAP_CREATE command creates a new map, returning a new file descriptor that refers to the map.

                  int
                  bpf_create_map(enum bpf_map_type map_type,
                                 unsigned int key_size,
                                 unsigned int value_size,
                                 unsigned int max_entries)
                  {
                      union bpf_attr attr = {
                          .map_type    = map_type,
                          .key_size    = key_size,
                          .value_size  = value_size,
                          .max_entries = max_entries
                      };

                      return bpf(BPF_MAP_CREATE, &attr, sizeof(attr));
                  }

              The new map has the type specified by map_type, and attributes as specified in key_size, value_size, and max_entries.  On success, this operation returns a file descriptor.  On error, -1 is returned and errno is set to EINVAL, EPERM, or ENOMEM.

              The key_size and value_size attributes will be used by the verifier during program loading to check that the program is calling bpf_map_*_elem() helper functions with a correctly initialized key and to check that the program doesn't access  the  map
              element value beyond the specified value_size.  For example, when a map is created with a key_size of 8 and the eBPF program calls

                  bpf_map_lookup_elem(map_fd, fp - 4)

              the program will be rejected, since the in-kernel helper function

                  bpf_map_lookup_elem(map_fd, void *key)

              expects to read 8 bytes from the location pointed to by key, but the fp - 4 (where fp is the top of the stack) starting address will cause out-of-bounds stack access.

              Similarly, when a map is created with a value_size of 1 and the eBPF program contains

                  value = bpf_map_lookup_elem(...);
                  *(u32 *) value = 1;

              the program will be rejected, since it accesses the value pointer beyond the specified 1 byte value_size limit.

              Currently, the following values are supported for map_type:

                  enum bpf_map_type {
                      BPF_MAP_TYPE_UNSPEC,  /* Reserve 0 as invalid map type */
                      BPF_MAP_TYPE_HASH,
                      BPF_MAP_TYPE_ARRAY,
                      BPF_MAP_TYPE_PROG_ARRAY,
                  };

              map_type  selects  one  of  the  available map implementations in the kernel.  For all map types, eBPF programs access maps with the same bpf_map_lookup_elem() and bpf_map_update_elem() helper functions.  Further details of the various map types are
              given below.

       BPF_MAP_LOOKUP_ELEM
              The BPF_MAP_LOOKUP_ELEM command looks up an element with a given key in the map referred to by the file descriptor fd.

                  int
                  bpf_lookup_elem(int fd, const void *key, void *value)
                  {
                      union bpf_attr attr = {
                          .map_fd = fd,
                          .key    = ptr_to_u64(key),
                          .value  = ptr_to_u64(value),
                      };

                      return bpf(BPF_MAP_LOOKUP_ELEM, &attr, sizeof(attr));
                  }

              If an element is found, the operation returns zero and stores the element's value into value, which must point to a buffer of value_size bytes.

              If no element is found, the operation returns -1 and sets errno to ENOENT.

       BPF_MAP_UPDATE_ELEM
              The BPF_MAP_UPDATE_ELEM command creates or updates an element with a given key/value in the map referred to by the file descriptor fd.

                  int
                  bpf_update_elem(int fd, const void *key, const void *value,
                                  uint64_t flags)
                  {
                      union bpf_attr attr = {
                          .map_fd = fd,
                          .key    = ptr_to_u64(key),
                          .value  = ptr_to_u64(value),
                          .flags  = flags,
                      };

                      return bpf(BPF_MAP_UPDATE_ELEM, &attr, sizeof(attr));
                  }

              The flags argument should be specified as one of the following:

              BPF_ANY
                     Create a new element or update an existing element.

              BPF_NOEXIST
                     Create a new element only if it did not exist.

              BPF_EXIST
                     Update an existing element.

              On success, the operation returns zero.  On error, -1 is returned and errno is set to EINVAL, EPERM, ENOMEM, or E2BIG.  E2BIG indicates that the number of elements in the map reached the max_entries limit specified at map creation time.  EEXIST will
              be returned if flags specifies BPF_NOEXIST and the element with key already exists in the map.  ENOENT will be returned if flags specifies BPF_EXIST and the element with key doesn't exist in the map.

       BPF_MAP_DELETE_ELEM
              The BPF_MAP_DELETE_ELEM command deleted the element whose key is key from the map referred to by the file descriptor fd.

                  int
                  bpf_delete_elem(int fd, const void *key)
                  {
                      union bpf_attr attr = {
                          .map_fd = fd,
                          .key    = ptr_to_u64(key),
                      };

                      return bpf(BPF_MAP_DELETE_ELEM, &attr, sizeof(attr));
                  }

              On success, zero is returned.  If the element is not found, -1 is returned and errno is set to ENOENT.

       BPF_MAP_GET_NEXT_KEY
              The BPF_MAP_GET_NEXT_KEY command looks up an element by key in the map referred to by the file descriptor fd and sets the next_key pointer to the key of the next element.

                  int
                  bpf_get_next_key(int fd, const void *key, void *next_key)
                  {
                      union bpf_attr attr = {
                          .map_fd   = fd,
                          .key      = ptr_to_u64(key),
                          .next_key = ptr_to_u64(next_key),
                      };

                      return bpf(BPF_MAP_GET_NEXT_KEY, &attr, sizeof(attr));
                  }

              If key is found, the operation returns zero and sets the next_key pointer to the key of the next element.  If key is not found, the operation returns zero and sets the next_key pointer to the key of the first element.  If key is the last element, -1
              is returned and errno is set to ENOENT.  Other possible errno values are ENOMEM, EFAULT, EPERM, and EINVAL.  This method can be used to iterate over all elements in the map.

       close(map_fd)
              Delete the map referred to by the file descriptor map_fd.  When the user-space program that created a map exits, all maps will be deleted automatically (but see NOTES).

   eBPF map types
       The following map types are supported:

       BPF_MAP_TYPE_HASH
              Hash-table maps have the following characteristics:

              *  Maps are created and destroyed by user-space programs.  Both user-space and eBPF programs can perform lookup, update, and delete operations.

              *  The kernel takes care of allocating and freeing key/value pairs.

              *  The map_update_elem() helper with fail to insert new element when the max_entries limit is reached.  (This ensures that eBPF programs cannot exhaust memory.)

              *  map_update_elem() replaces existing elements atomically.

              Hash-table maps are optimized for speed of lookup.

       BPF_MAP_TYPE_ARRAY
              Array maps have the following characteristics:

              *  Optimized for fastest possible lookup.  In the future the verifier/JIT compiler may recognize lookup() operations that employ a constant key and optimize it into constant pointer.  It is possible to optimize a non-constant key into direct pointer
                 arithmetic  as  well,  since  pointers  and  value_size are constant for the life of the eBPF program.  In other words, array_map_lookup_elem() may be 'inlined' by the verifier/JIT compiler while preserving concurrent access to this map from user
                 space.

              *  All array elements pre-allocated and zero initialized at init time

              *  The key is an array index, and must be exactly four bytes.

              *  map_delete_elem() fails with the error EINVAL, since elements cannot be deleted.

              *  map_update_elem() replaces elements in a nonatomic fashion; for atomic updates, a hash-table map should be used instead.  There is however one special case that can also be used with arrays: the atomic built-in __sync_fetch_and_add() can be  used
                 on  32  and 64 bit atomic counters.  For example, it can be applied on the whole value itself if it represents a single counter, or in case of a structure containing multiple counters, it could be used on individual counters.  This is quite often
                 useful for aggregation and accounting of events.

              Among the uses for array maps are the following:

              *  As "global" eBPF variables: an array of 1 element whose key is (index) 0 and where the value is a collection of 'global' variables which eBPF programs can use to keep state between events.

              *  Aggregation of tracing events into a fixed set of buckets.

              *  Accounting of networking events, for example, number of packets and packet sizes.

       BPF_MAP_TYPE_PROG_ARRAY (since Linux 4.2)
              A program array map is a special kind of array map whose map values contain only file descriptors referring to other eBPF programs.  Thus, both the key_size and value_size must be exactly four bytes.   This  map  is  used  in  conjunction  with  the
              bpf_tail_call() helper.

              This means that an eBPF program with a program array map attached to it can call from kernel side into

                  void bpf_tail_call(void *context, void *prog_map, unsigned int index);

              and  therefore  replace  its  own program flow with the one from the program at the given program array slot, if present.  This can be regarded as kind of a jump table to a different eBPF program.  The invoked program will then reuse the same stack.
              When a jump into the new program has been performed, it won't return to the old program anymore.

              If no eBPF program is found at the given index of the program array (because the map slot doesn't contain a valid program file descriptor, the specified lookup index/key is out of bounds, or the limit of 32 nested calls has been  exceed),  execution
              continues with the current eBPF program.  This can be used as a fall-through for default cases.

              A  program  array  map is useful, for example, in tracing or networking, to handle individual system calls or protocols in their own subprograms and use their identifiers as an individual map index.  This approach may result in performance benefits,
              and also makes it possible to overcome the maximum instruction limit of a single eBPF program.  In dynamic environments, a user-space daemon might atomically replace individual subprograms at run-time with newer versions  to  alter  overall  program
              behavior, for instance, if global policies change.

   eBPF programs
       The BPF_PROG_LOAD command is used to load an eBPF program into the kernel.  The return value for this command is a new file descriptor associated with this eBPF program.

           char bpf_log_buf[LOG_BUF_SIZE];

           int
           bpf_prog_load(enum bpf_prog_type type,
                         const struct bpf_insn *insns, int insn_cnt,
                         const char *license)
           {
               union bpf_attr attr = {
                   .prog_type = type,
                   .insns     = ptr_to_u64(insns),
                   .insn_cnt  = insn_cnt,
                   .license   = ptr_to_u64(license),
                   .log_buf   = ptr_to_u64(bpf_log_buf),
                   .log_size  = LOG_BUF_SIZE,
                   .log_level = 1,
               };

               return bpf(BPF_PROG_LOAD, &attr, sizeof(attr));
           }

       prog_type is one of the available program types:

           enum bpf_prog_type {
               BPF_PROG_TYPE_UNSPEC,        /* Reserve 0 as invalid
                                               program type */
               BPF_PROG_TYPE_SOCKET_FILTER,
               BPF_PROG_TYPE_KPROBE,
               BPF_PROG_TYPE_SCHED_CLS,
               BPF_PROG_TYPE_SCHED_ACT,
           };

       For further details of eBPF program types, see below.

       The remaining fields of bpf_attr are set as follows:

       *  insns is an array of struct bpf_insn instructions.

       *  insn_cnt is the number of instructions in the program referred to by insns.

       *  license is a license string, which must be GPL compatible to call helper functions marked gpl_only.  (The licensing rules are the same as for kernel modules, so that also dual licenses, such as "Dual BSD/GPL", may be used.)

       *  log_buf  is  a pointer to a caller-allocated buffer in which the in-kernel verifier can store the verification log.  This log is a multi-line string that can be checked by the program author in order to understand how the verifier came to the conclusion
          that the eBPF program is unsafe.  The format of the output can change at any time as the verifier evolves.

       *  log_size size of the buffer pointed to by log_bug.  If the size of the buffer is not large enough to store all verifier messages, -1 is returned and errno is set to ENOSPC.

       *  log_level verbosity level of the verifier.  A value of zero means that the verifier will not provide a log; in this case, log_buf must be a NULL pointer, and log_size must be zero.

       Applying close(2) to the file descriptor returned by BPF_PROG_LOAD will unload the eBPF program (but see NOTES).

       Maps are accessible from eBPF programs and are used to exchange data between eBPF programs and between eBPF programs and user-space programs.  For example, eBPF programs can process various events (like kprobe, packets) and store their data into a map, and
       user-space  programs  can  then fetch data from the map.  Conversely, user-space programs can use a map as a configuration mechanism, populating the map with values checked by the eBPF program, which then modifies its behavior on the fly according to those
       values.

   eBPF program types
       The eBPF program type (prog_type) determines the subset of kernel helper functions that the program may call.  The program type also determines the program input (context)—the format of struct bpf_context (which is the data blob passed into the  eBPF  pro‐
       gram as the first argument).

       For  example, a tracing program does not have the exact same subset of helper functions as a socket filter program (though they may have some helpers in common).  Similarly, the input (context) for a tracing program is a set of register values, while for a
       socket filter it is a network packet.

       The set of functions available to eBPF programs of a given type may increase in the future.

       The following program types are supported:

       BPF_PROG_TYPE_SOCKET_FILTER (since Linux 3.19)
              Currently, the set of functions for BPF_PROG_TYPE_SOCKET_FILTER is:

                  bpf_map_lookup_elem(map_fd, void *key)
                                      /* look up key in a map_fd */
                  bpf_map_update_elem(map_fd, void *key, void *value)
                                      /* update key/value */
                  bpf_map_delete_elem(map_fd, void *key)
                                      /* delete key in a map_fd */

              The bpf_context argument is a pointer to a struct __sk_buff.

       BPF_PROG_TYPE_KPROBE (since Linux 4.1)
              [To be documented]

       BPF_PROG_TYPE_SCHED_CLS (since Linux 4.1)
              [To be documented]

       BPF_PROG_TYPE_SCHED_ACT (since Linux 4.1)
              [To be documented]

   Events
       Once a program is loaded, it can be attached to an event.  Various kernel subsystems have different ways to do so.

       Since Linux 3.19, the following call will attach the program prog_fd to the socket sockfd, which was created by an earlier call to socket(2):

           setsockopt(sockfd, SOL_SOCKET, SO_ATTACH_BPF,
                      &prog_fd, sizeof(prog_fd));

       Since Linux 4.1, the following call may be used to attach the eBPF program referred to by the file descriptor prog_fd to a perf event file descriptor, event_fd, that was created by a previous call to perf_event_open(2):

           ioctl(event_fd, PERF_EVENT_IOC_SET_BPF, prog_fd);

EXAMPLES
       /* bpf+sockets example:
        * 1. create array map of 256 elements
        * 2. load program that counts number of packets received
        *    r0 = skb->data[ETH_HLEN + offsetof(struct iphdr, protocol)]
        *    map[r0]++
        * 3. attach prog_fd to raw socket via setsockopt()
        * 4. print number of received TCP/UDP packets every second
        */
       int
       main(int argc, char **argv)
       {
           int sock, map_fd, prog_fd, key;
           long long value = 0, tcp_cnt, udp_cnt;

           map_fd = bpf_create_map(BPF_MAP_TYPE_ARRAY, sizeof(key),
                                   sizeof(value), 256);
           if (map_fd < 0) {
               printf("failed to create map '%s' ", strerror(errno));
               /* likely not run as root */
               return 1;
           }

           struct bpf_insn prog[] = {
               BPF_MOV64_REG(BPF_REG_6, BPF_REG_1),        /* r6 = r1 */
               BPF_LD_ABS(BPF_B, ETH_HLEN + offsetof(struct iphdr, protocol)),
                                       /* r0 = ip->proto */
               BPF_STX_MEM(BPF_W, BPF_REG_10, BPF_REG_0, -4),
                                       /* *(u32 *)(fp - 4) = r0 */
               BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),       /* r2 = fp */
               BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4),      /* r2 = r2 - 4 */
               BPF_LD_MAP_FD(BPF_REG_1, map_fd),           /* r1 = map_fd */
               BPF_CALL_FUNC(BPF_FUNC_map_lookup_elem),
                                       /* r0 = map_lookup(r1, r2) */
               BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 2),
                                       /* if (r0 == 0) goto pc+2 */
               BPF_MOV64_IMM(BPF_REG_1, 1),                /* r1 = 1 */
               BPF_XADD(BPF_DW, BPF_REG_0, BPF_REG_1, 0, 0),
                                       /* lock *(u64 *) r0 += r1 */
               BPF_MOV64_IMM(BPF_REG_0, 0),                /* r0 = 0 */
               BPF_EXIT_INSN(),                            /* return r0 */
           };

           prog_fd = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, prog,
                                   sizeof(prog), "GPL");

           sock = open_raw_sock("lo");

           assert(setsockopt(sock, SOL_SOCKET, SO_ATTACH_BPF, &prog_fd,
                             sizeof(prog_fd)) == 0);

           for (;;) {
               key = IPPROTO_TCP;
               assert(bpf_lookup_elem(map_fd, &key, &tcp_cnt) == 0);
               key = IPPROTO_UDP
               assert(bpf_lookup_elem(map_fd, &key, &udp_cnt) == 0);
               printf("TCP %lld UDP %lld packets0, tcp_cnt, udp_cnt);
               sleep(1);
           }

           return 0;
       }

       Some complete working code can be found in the samples/bpf directory in the kernel source tree.

RETURN VALUE
       For a successful call, the return value depends on the operation:

       BPF_MAP_CREATE
              The new file descriptor associated with the eBPF map.

       BPF_PROG_LOAD
              The new file descriptor associated with the eBPF program.

       All other commands
              Zero.

       On error, -1 is returned, and errno is set appropriately.

ERRORS
       EPERM  The call was made without sufficient privilege (without the CAP_SYS_ADMIN capability).

       ENOMEM Cannot allocate sufficient memory.

       EBADF  fd is not an open file descriptor.

       EFAULT One of the pointers (key or value or log_buf or insns) is outside the accessible address space.

       EINVAL The value specified in cmd is not recognized by this kernel.

       EINVAL For BPF_MAP_CREATE, either map_type or attributes are invalid.

       EINVAL For BPF_MAP_*_ELEM commands, some of the fields of union bpf_attr that are not used by this command are not set to zero.

       EINVAL For BPF_PROG_LOAD, indicates an attempt to load an invalid program.  eBPF programs can be deemed invalid due to unrecognized instructions, the use of reserved fields, jumps out of range, infinite loops or calls of unknown functions.

       EACCES For BPF_PROG_LOAD, even though all program instructions are valid, the program has been rejected because it was deemed unsafe.  This may be because it may have accessed a disallowed memory region or an uninitialized  stack/register  or  because  the
              function constraints don't match the actual types or because there was a misaligned memory access.  In this case, it is recommended to call bpf() again with log_level = 1 and examine log_buf for the specific reason provided by the verifier.

       ENOENT For BPF_MAP_LOOKUP_ELEM or BPF_MAP_DELETE_ELEM, indicates that the element with the given key was not found.

       E2BIG  The eBPF program is too large or a map reached the max_entries limit (maximum number of elements).

部分内容参考：https://yq.aliyun.com/articles/591413?spm=a2c4e.11163080.searchblog.71.8f2d2ec1PIuQu6