BCC观测工具的使用

观测工具下载地址：https://github.com/iovisor/bcc

bcc观测工具的安装方式：https://github.com/iovisor/bcc/blob/master/INSTALL.md

工具简介：

BCC makes BPF programs easier to write, with kernel instrumentation in C (and includes a C wrapper around LLVM), and front-ends in Python and lua. It is suited for many tasks, including performance analysis and network traffic control.

BCC 是用于创建高效内核跟踪和操作程序的工具包，包括一些有用的工具和示例。它利用了扩展的BPF（Berkeley Packet Filters），正式名称为 eBPF。

使用要求：

1）Linux 内核版本最低 4.1
2）kernel-devel 版本必须和内核版本一致

[root@yang-01 ~]# rpm -qa|grep -i kernel | grep -i devel
kernel-devel-3.10.0-1127.18.2.el7.x86_64
[root@yang-01 ~]# uname -r
3.10.0-1127.el7.x86_64

3）kernel 配置文件相关参数必须按照下面的要求

[root@yang-01 ~]# grep -i BPF /boot/config-`uname -r`
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_BPF_JIT_ALWAYS_ON=y
CONFIG_NETFILTER_XT_MATCH_BPF=m
CONFIG_NET_CLS_BPF=m 
CONFIG_BPF_JIT=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_BPF_EVENTS=y
CONFIG_BPF_KPROBE_OVERRIDE=y

4）对于 CentOS 建议升级到 3.6 及以上版本
5）使用默认的 Python 2.7 即可，不需要升级 Python 3

yum安装

[root@yang-01 ~]# cat /etc/redhat-release

CentOS Linux release 7.8.2003 (Core)

[root@yang-01 ~]# uname -r

3.10.0-1127.el7.x86_64

  

[root@yang-01 ~]# yum update kernel

Loaded plugins: fastestmirror

Loading mirror speeds from cached hostfile

 * base: mirrors.cn99.com

 * epel: mirrors.yun-idc.com

 * extras: mirrors.aliyun.com

 * updates: mirrors.aliyun.com

Resolving Dependencies

--> Running transaction check

---> Package kernel.x86_64 0:3.10.0-1127.18.2.el7 will be installed

--> Finished Dependency Resolution

 

Dependencies Resolved

 

===========================================================================================================================================================================

 Package                              Arch                                 Version                                             Repository                             Size

===========================================================================================================================================================================

Installing:

 kernel                               x86_64                               3.10.0-1127.18.2.el7                                updates                                50 M

 

Transaction Summary

===========================================================================================================================================================================

Install  1 Package

 

Total download size: 50 M

Installed size: 64 M

Is this ok [y/d/N]: y

Downloading packages:

Delta RPMs disabled because /usr/bin/applydeltarpm not installed.

kernel-3.10.0-1127.18.2.el7.x86_64.rpm                                                                                                              |  50 MB  00:00:08    

Running transaction check

Running transaction test

Transaction test succeeded

Running transaction

  Installing : kernel-3.10.0-1127.18.2.el7.x86_64                                                                                                                      1/1

  Verifying  : kernel-3.10.0-1127.18.2.el7.x86_64                                                                                                                      1/1

 

Installed:

  kernel.x86_64 0:3.10.0-1127.18.2.el7                                                                                                                                    

 

Complete!

[root@yang-01 ~]# yum install bcc -y

Loaded plugins: fastestmirror

Loading mirror speeds from cached hostfile

 * base: mirrors.cn99.com

 * epel: mirrors.yun-idc.com

 * extras: mirrors.aliyun.com

 * updates: mirrors.aliyun.com

Resolving Dependencies

--> Running transaction check

---> Package bcc.x86_64 0:0.10.0-1.el7 will be installed

--> Processing Dependency: bcc-tools = 0.10.0-1.el7 for package: bcc-0.10.0-1.el7.x86_64

--> Processing Dependency: llvm-private >= 6.0.1-0.3 for package: bcc-0.10.0-1.el7.x86_64

--> Processing Dependency: libLLVM-7-rhel.so(LLVM_7)(64bit) for package: bcc-0.10.0-1.el7.x86_64

--> Processing Dependency: libclangSerialization.so.7()(64bit) for package: bcc-0.10.0-1.el7.x86_64

--> Processing Dependency: libclangSema.so.7()(64bit) for package: bcc-0.10.0-1.el7.x86_64

--> Processing Dependency: libclangRewrite.so.7()(64bit) for package: bcc-0.10.0-1.el7.x86_64

--> Processing Dependency: libclangParse.so.7()(64bit) for package: bcc-0.10.0-1.el7.x86_64

--> Processing Dependency: libclangLex.so.7()(64bit) for package: bcc-0.10.0-1.el7.x86_64

--> Processing Dependency: libclangFrontend.so.7()(64bit) for package: bcc-0.10.0-1.el7.x86_64

--> Processing Dependency: libclangEdit.so.7()(64bit) for package: bcc-0.10.0-1.el7.x86_64

--> Processing Dependency: libclangDriver.so.7()(64bit) for package: bcc-0.10.0-1.el7.x86_64

--> Processing Dependency: libclangCodeGen.so.7()(64bit) for package: bcc-0.10.0-1.el7.x86_64

--> Processing Dependency: libclangBasic.so.7()(64bit) for package: bcc-0.10.0-1.el7.x86_64

--> Processing Dependency: libclangAnalysis.so.7()(64bit) for package: bcc-0.10.0-1.el7.x86_64

--> Processing Dependency: libclangAST.so.7()(64bit) for package: bcc-0.10.0-1.el7.x86_64

--> Processing Dependency: libLLVM-7-rhel.so()(64bit) for package: bcc-0.10.0-1.el7.x86_64

--> Running transaction check

---> Package bcc-tools.x86_64 0:0.10.0-1.el7 will be installed

--> Processing Dependency: python-bcc = 0.10.0-1.el7 for package: bcc-tools-0.10.0-1.el7.x86_64

--> Processing Dependency: python-netaddr for package: bcc-tools-0.10.0-1.el7.x86_64

--> Processing Dependency: kernel-devel for package: bcc-tools-0.10.0-1.el7.x86_64

---> Package llvm-private.x86_64 0:7.0.1-1.el7 will be installed

--> Running transaction check

---> Package kernel-devel.x86_64 0:3.10.0-1127.18.2.el7 will be installed

---> Package python-bcc.x86_64 0:0.10.0-1.el7 will be installed

---> Package python-netaddr.noarch 0:0.7.5-9.el7 will be installed

--> Finished Dependency Resolution

 

Dependencies Resolved

 

===========================================================================================================================================================================

 Package                                    Arch                               Version                                           Repository                           Size

===========================================================================================================================================================================

Installing:

 bcc                                        x86_64                             0.10.0-1.el7                                      base                                567 k

Installing for dependencies:

 bcc-tools                                  x86_64                             0.10.0-1.el7                                      base                                394 k

 kernel-devel                               x86_64                             3.10.0-1127.18.2.el7                              updates                              18 M

 llvm-private                               x86_64                             7.0.1-1.el7                                       base                                 23 M

 python-bcc                                 x86_64                             0.10.0-1.el7                                      base                                 74 k

 python-netaddr                             noarch                             0.7.5-9.el7                                       base                                983 k

 

Transaction Summary

===========================================================================================================================================================================

Install  1 Package (+5 Dependent packages)

 

Total download size: 42 M

Installed size: 126 M

Downloading packages:

Delta RPMs disabled because /usr/bin/applydeltarpm not installed.

(1/6): bcc-tools-0.10.0-1.el7.x86_64.rpm                                                                                                            | 394 kB  00:00:05    

(2/6): bcc-0.10.0-1.el7.x86_64.rpm                                                                                                                  | 567 kB  00:00:05    

(3/6): python-bcc-0.10.0-1.el7.x86_64.rpm                                                                                                           |  74 kB  00:00:00    

(4/6): python-netaddr-0.7.5-9.el7.noarch.rpm                                                                                                        | 983 kB  00:00:00    

(5/6): kernel-devel-3.10.0-1127.18.2.el7.x86_64.rpm                                                                                                 |  18 MB  00:00:06    

(6/6): llvm-private-7.0.1-1.el7.x86_64.rpm                                                                                                          |  23 MB  00:00:30    

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Total                                                                                                                                      1.4 MB/s |  42 MB  00:00:30    

Running transaction check

Running transaction test

Transaction test succeeded

Running transaction

  Installing : llvm-private-7.0.1-1.el7.x86_64                                                                                                                         1/6

  Installing : kernel-devel-3.10.0-1127.18.2.el7.x86_64                                                                                                                2/6

  Installing : python-netaddr-0.7.5-9.el7.noarch                                                                                                                       3/6

  Installing : python-bcc-0.10.0-1.el7.x86_64                                                                                                                          4/6

  Installing : bcc-tools-0.10.0-1.el7.x86_64                                                                                                                           5/6

  Installing : bcc-0.10.0-1.el7.x86_64                                                                                                                                 6/6

  Verifying  : python-netaddr-0.7.5-9.el7.noarch                                                                                                                       1/6

  Verifying  : python-bcc-0.10.0-1.el7.x86_64                                                                                                                          2/6

  Verifying  : bcc-0.10.0-1.el7.x86_64                                                                                                                                 3/6

  Verifying  : kernel-devel-3.10.0-1127.18.2.el7.x86_64                                                                                                                4/6

  Verifying  : bcc-tools-0.10.0-1.el7.x86_64                                                                                                                           5/6

  Verifying  : llvm-private-7.0.1-1.el7.x86_64                                                                                                                         6/6

 

Installed:

  bcc.x86_64 0:0.10.0-1.el7                                                                                                                                               

 

Dependency Installed:

  bcc-tools.x86_64 0:0.10.0-1.el7           kernel-devel.x86_64 0:3.10.0-1127.18.2.el7       llvm-private.x86_64 0:7.0.1-1.el7       python-bcc.x86_64 0:0.10.0-1.el7     

  python-netaddr.noarch 0:0.7.5-9.el7     

 

Complete!

 

[root@yang-01 tools]# pwd

/usr/share/bcc/tools

[root@yang-01 tools]# ls

argdist       cachestat     dcstat      fileslower      javagc       mountsnoop      opensnoop  pythoncalls  rubystat     sslsniff    tcpaccept   tplist

bashreadline  cachetop      deadlock    filetop         javaobjnew   mysqld_qslower  perlcalls  pythonflow   runqlat      stackcount  tcpconnect  trace

biolatency    capable       deadlock.c  funccount       javastat     nfsdist         perlflow   pythongc     runqlen      statsnoop   tcpconnlat  ttysnoop

biosnoop      cobjnew       doc         funclatency     javathreads  nfsslower       perlstat   pythonstat   runqslower   syncsnoop   tcpdrop     vfscount

biotop        cpudist       drsnoop     funcslower      killsnoop    nodegc          phpcalls   reset-trace  shmsnoop     syscount    tcplife     vfsstat

bitesize      cpuunclaimed  execsnoop   gethostlatency  lib          nodestat        phpflow    rubycalls    slabratetop  tclcalls    tcpretrans  wakeuptime

bpflist       dbslower      ext4dist    hardirqs        llcstat      offcputime      phpstat    rubyflow     sofdsnoop    tclflow     tcpsubnet   xfsdist

btrfsdist     dbstat        ext4slower  javacalls       mdflush      offwaketime     pidpersec  rubygc       softirqs     tclobjnew   tcptop      xfsslower

btrfsslower   dcsnoop       filelife    javaflow        memleak      oomkill         profile    rubyobjnew   solisten     tclstat     tcptracer

 点击此处展开...

报错举例1

# 报错举例

[root@yang-01 ~]# cachestat 1 3

modprobe: FATAL: Module kheaders not found.

chdir(/lib/modules/3.10.0-1127.el7.x86_64/build): No such file or directory

Traceback (most recent call last):

  File "/usr/share/bcc/tools/cachestat", line 96, in <module>

    b = BPF(text=bpf_text)

  File "/usr/lib/python2.7/site-packages/bcc/__init__.py", line 325, in __init__

    raise Exception("Failed to compile BPF text")

Exception: Failed to compile BPF text

 

# 发现报错，这时需要进行内核版本的查看，确保 kernel-devel 与系统上当前安装的内核版本匹配

[root@yang-01 tools]# rpm -qa|grep -i kernel | grep -i devel

kernel-devel-3.10.0-1127.18.2.el7.x86_64

[root@yang-01 tools]# uname -r

3.10.0-1127.el7.x86_64

[root@yang-01 tools]# yum install "kernel-devel-uname-r == $(uname -r)"

Loaded plugins: fastestmirror

Loading mirror speeds from cached hostfile

 * base: mirrors.cn99.com

 * epel: mirrors.yun-idc.com

 * extras: mirrors.aliyun.com

 * updates: mirrors.aliyun.com

Resolving Dependencies

--> Running transaction check

---> Package kernel-devel.x86_64 0:3.10.0-1127.el7 will be installed

--> Finished Dependency Resolution

 

Dependencies Resolved

 

===========================================================================================================================================================================

 Package                                    Arch                                 Version                                          Repository                          Size

===========================================================================================================================================================================

Installing:

 kernel-devel                               x86_64                               3.10.0-1127.el7                                  base                                18 M

 

Transaction Summary

===========================================================================================================================================================================

Install  1 Package

 

Total download size: 18 M

Installed size: 38 M

Is this ok [y/d/N]: y

Downloading packages:

Delta RPMs disabled because /usr/bin/applydeltarpm not installed.

kernel-devel-3.10.0-1127.el7.x86_64.rpm                                                                                                             |  18 MB  00:00:06    

Running transaction check

Running transaction test

Transaction test succeeded

Running transaction

  Installing : kernel-devel-3.10.0-1127.el7.x86_64                                                                                                                     1/1

  Verifying  : kernel-devel-3.10.0-1127.el7.x86_64                                                                                                                     1/1

 

Installed:

  kernel-devel.x86_64 0:3.10.0-1127.el7                                                                                                                                   

 

Complete!

报错举例2

报错: 在安装kernel-devel显示 No package

       No package kernel-devel-uname-r == 5.6.2-1.el7.elrepo.x86_64 available.

  

解决办法: 启用[elrepo-kerne]存储库,

# vim /etc/yum.repos.d/elrepo.repo

[elrepo-kernel]

name=ELRepo.org Community Enterprise Linux Kernel Repository - el7

baseurl=http://elrepo.org/linux/kernel/el7/$basearch/

        http://mirrors.coreix.net/elrepo/kernel/el7/$basearch/

        http://mirror.rackspace.com/elrepo/kernel/el7/$basearch/

        http://repos.lax-noc.com/elrepo/kernel/el7/$basearch/

        http://mirror.ventraip.net.au/elrepo/kernel/el7/$basearch/

mirrorlist=http://mirrors.elrepo.org/mirrors-elrepo-kernel.el7

# 修改这一行将0改为1

enabled=1

 

然后重新执行:

# sudo yum install "kernel-devel-uname-r == $(uname -r)"

缓存命中查询 命令使用举例

[root@yang-01 tools]# cachestat 1 5

    HITS   MISSES  DIRTIES HITRATIO   BUFFERS_MB  CACHED_MB

    1409        0       77  100.00%            1       2440

    1438        0      101  100.00%            1       2440

    1413        0       68  100.00%            1       2440

    1412        0      117  100.00%            1       2440

    1411        0       72  100.00%            1       2440

第一个参数为间隔秒，第二个参数为输出次数

TOTAL ，表示总的 I/O 次数

MISSES ，表示缓存未命中的次数

HITS ，表示缓存命中的次数

DIRTIES， 表示新增到缓存中的脏页数

BUFFERS_MB 表示 Buffers 的大小，以 MB 为单位

CACHED_MB 表示 Cache 的大小，以 MB 为单位

 

 

  

[root@yang-01 tools]# cachetop

16:24:15 Buffers MB: 1 / Cached MB: 2443 / Sort: HITS / Order: ascending

PID      UID      CMD              HITS     MISSES   DIRTIES  READ_HIT%  WRITE_HIT%

    1653 actionte ustats                  1        0        0     100.0%       0.0%

    1778 actionte umon-mgr                1        0        0     100.0%       0.0%

     903 mysql    jsvc                    1        0        0     100.0%       0.0%

   10909 actionte mysqld                  2        0        0     100.0%       0.0%

   22506 actionte uproxy                  2        0        1      50.0%       0.0%

     869 mysql    jsvc                    2        2        2       0.0%       0.0%

   10788 actionte mysqld                  3        0        0     100.0%       0.0%

    1724 actionte urman-agent             4        2        2      33.3%       0.0%

   21780 actionte uguard-agent            4        2        2      33.3%       0.0%

   12798 actionte ustats                  6        0        0     100.0%       0.0%

   12799 actionte ustats                  6        0        0     100.0%       0.0%

   12800 actionte ustats                  6        0        0     100.0%       0.0%

这个命令的输出和top类似，默认按照缓存的命中次数（HITS）排序，展示了每个进程的缓存命中情况。

这里的 HITS、MISSES 和 DIRTIES ，跟 cachestat 里的含义一样，分别代表间隔时间内的缓存命中次数、未命中次数以及新增到缓存中的脏页数。

而 READ_HIT 和 WRITE_HIT ，分别表示读和写的缓存命中率。

BCC工具

# 查看新的进程。那些会消耗系统资源，但很短暂的进程，它们甚至不会出现在 top(1) 命令或其它工具中的显示之中。这些新进程可以使用 execsnoop 进行检测（或使用行业术语说，可以被追踪traced）。

[root@yang-01 tools]# execsnoop -h

usage: execsnoop [-h] [-t] [-x] [-q] [-n NAME] [-l LINE] [--max-args MAX_ARGS]

 

Trace exec() syscalls

 

optional arguments:

  -h, --help            show this help message and exit

  -t, --timestamp       include timestamp on output

  -x, --fails           include failed exec()s

  -q, --quote           Add quotemarks (") around arguments.

  -n NAME, --name NAME  only print commands matching this name (regex), any

                        arg

  -l LINE, --line LINE  only print commands where arg contains this line

                        (regex)

  --max-args MAX_ARGS   maximum number of arguments parsed and displayed,

                        defaults to 20

 

examples:

    ./execsnoop           # trace all exec() syscalls

    ./execsnoop -x        # include failed exec()s

    ./execsnoop -t        # include timestamps

    ./execsnoop -q        # add "quotemarks" around arguments

    ./execsnoop -n main   # only print command lines containing "main"

    ./execsnoop -l tpkg   # only print command where arguments contains "tpkg"

[root@yang-01 tools]# execsnoop

PCOMM            PID    PPID   RET ARGS

bash             17436  1642     0 /usr/bin/bash --noprofile --norc -c systemctl is-active mysqld_5690.service

bash             17437  1642     0

systemctl        17437  1642     0 /usr/bin/bash --noprofile --norc -c systemctl is-active mysqld_3307.service /usr/bin/systemctl is-active mysqld_3307.service

systemctl        17436  1642     0 /usr/bin/systemctl is-active mysqld_5690.service

bash             17439  1642     0 /usr/bin/bash --noprofile --norc -c systemctl is-active mysqld_3307.service

systemctl        17439  1642     0 /usr/bin/systemctl is-active mysqld_3307.service

bash             17440  1642     0 /usr/bin/bash --noprofile --norc -c systemctl is-active mysqld_5690.service

systemctl        17440  1642     0 /usr/bin/systemctl is-active mysqld_5690.service

bash             17442  1642     0 /usr/bin/bash --noprofile --norc -c systemctl is-active mysqld_3307.service

systemctl        17442  1642     0 /usr/bin/systemctl is-active mysqld_3307.service

bash             17443  1642     0 /usr/bin/bash --noprofile --norc -c systemctl is-active mysqld_5690.service

systemctl        17443  1642     0 /usr/bin/systemctl is-active mysqld_5690.service

bash             17445  1642     0 /usr/bin/bash --noprofile --norc -c systemctl is-active mysqld_3307.service

bash             17446  1642     0 /usr/bin/bash --noprofile --norc -c systemctl is-active mysqld_5690.service

##################################################################################################################################################################

  

  

  

# opensnoop 通过跟踪 open() 系统调用来工作。opensnoop 的一些优点在于它能在系统范围内工作，并且跟踪所有进程的 open() 系统调用。

[root@yang-01 tools]# opensnoop -h

usage: opensnoop [-h] [-T] [-U] [-x] [-p PID] [-t TID] [-u UID] [-d DURATION]

                 [-n NAME] [-e] [-f FLAG_FILTER]

 

Trace open() syscalls

 

optional arguments:

  -h, --help            show this help message and exit

  -T, --timestamp       include timestamp on output

  -U, --print-uid       print UID column

  -x, --failed          only show failed opens

  -p PID, --pid PID     trace this PID only

  -t TID, --tid TID     trace this TID only

  -u UID, --uid UID     trace this UID only

  -d DURATION, --duration DURATION

                        total duration of trace in seconds

  -n NAME, --name NAME  only print process names containing this name

  -e, --extended_fields

                        show extended fields

  -f FLAG_FILTER, --flag_filter FLAG_FILTER

                        filter on flags argument (e.g., O_WRONLY)

 

examples:

    ./opensnoop           # trace all open() syscalls

    ./opensnoop -T        # include timestamps

    ./opensnoop -U        # include UID

    ./opensnoop -x        # only show failed opens

    ./opensnoop -p 181    # only trace PID 181

    ./opensnoop -t 123    # only trace TID 123

    ./opensnoop -u 1000   # only trace UID 1000

    ./opensnoop -d 10     # trace for 10 seconds only

    ./opensnoop -n main   # only print process names containing "main"

    ./opensnoop -e        # show extended fields

    ./opensnoop -f O_WRONLY -f O_RDWR  # only print calls for writing

[root@yang-01 tools]# opensnoop

PID    COMM               FD ERR PATH

1642   ustats             14   0 ./logs

18699  opensnoop          -1   2 /usr/lib64/python2.7/encodings/ascii.so

18699  opensnoop          -1   2 /usr/lib64/python2.7/encodings/asciimodule.so

18699  opensnoop          16   0 /usr/lib64/python2.7/encodings/ascii.py

18699  opensnoop          17   0 /usr/lib64/python2.7/encodings/ascii.pyc

1309   uagent             16   0 /action-dmp/ustats/ustats.pid

1309   uagent             16   0 /proc/1642/cmdline

1309   uagent             16   0 /action-dmp/umon-mgr/umon-mgr.pid

1309   uagent             16   0 /proc/1684/cmdline

1309   uagent             16   0 /action-dmp/urman-agent/urman-agent.pid

1309   uagent             16   0 /proc/1672/cmdline

1309   uagent             16   0 /action-dmp/ucore/ucore.pid

1309   uagent             16   0 /proc/1538/cmdline

1309   uagent             16   0 /action-dmp/uguard-agent/uguard-agent.pid

1309   uagent             16   0 /proc/1664/cmdline

1309   uagent             16   0 /action-dmp/uguard-mgr/uguard-mgr.pid

1309   uagent             16   0 /proc/1636/cmdline

1309   uagent             16   0 /action-dmp/umon/umon.pid

1309   uagent             16   0 /proc/1734/cmdline

1309   uagent             16   0 /action-dmp/udeploy/udeploy.pid

1309   uagent             16   0 /proc/1644/cmdline

1309   uagent             16   0 /action-dmp/umc/umc.pid

1309   uagent             16   0 /proc/1638/cmdline

1309   uagent             16   0 /action-dmp/uproxy/uproxy.pid

1309   uagent             16   0 /proc/22496/cmdline

1309   uagent             16   0 /action-dmp/urman-mgr/urman-mgr.pid

1309   uagent             16   0 /proc/1640/cmdline

1664   uguard-agent       18   0 ./logs

1642   ustats             14   0 /dev/null

1642   ustats             40   0 /dev/null

18703  bash                3   0 /etc/ld.so.cache

18703  bash                3   0 /lib64/libtinfo.so.5

18703  bash                3   0 /lib64/libdl.so.2

18703  bash                3   0 /lib64/libc.so.6

18703  bash               -1   6 /dev/tty

18703  bash                3   0 /usr/lib/locale/locale-archive

18703  bash                3   0 /proc/meminfo

18703  bash                3   0 /usr/lib64/gconv/gconv-modules.cache

18703  bash                3   0 /etc/nsswitch.conf

18703  bash                3   0 /etc/ld.so.cache

18703  bash                3   0 /lib64/libnss_files.so.2

18703  bash                3   0 /etc/passwd

5448   mysqld             -1   2 ./universe/u_delay.frm

18703  systemctl           3   0 /etc/ld.so.cache

18703  systemctl           3   0 /lib64/librt.so.1

18703  systemctl           3   0 /lib64/libselinux.so.1

18703  systemctl           3   0 /lib64/liblzma.so.5

18703  systemctl           3   0 /lib64/liblz4.so.1

18703  systemctl           3   0 /lib64/libgcrypt.so.11

18703  systemctl           3   0 /lib64/libcap.so.2

18703  systemctl           3   0 /lib64/libgcc_s.so.1

18703  systemctl           3   0 /lib64/libpthread.so.0

18703  systemctl           3   0 /lib64/libc.so.6

18703  systemctl           3   0 /lib64/libpcre.so.1

18703  systemctl           3   0 /lib64/libdl.so.2

18703  systemctl           3   0 /lib64/libgpg-error.so.0

18703  systemctl           3   0 /lib64/libattr.so.1

18704  bash                3   0 /etc/ld.so.cache

18704  bash                3   0 /lib64/libtinfo.so.5

##################################################################################################################################################################

  

  

# xfsslower 工具可以跟踪大于 1 毫秒（参数）延迟的常见 XFS 文件系统操作。

# 在 xfsslower 工具运行的时候，通过在 XFS 中动态地检测内核函数实现的，并当它结束的时候解除该检测。

# 这个 bcc 工具也有其它文件系统的版本：ext4slower、btrfsslower、zfsslower 和 nfsslower。

[root@yang-01 tools]# xfsslower -h

usage: xfsslower [-h] [-j] [-p PID] [min_ms]

 

Trace common XFS file operations slower than a threshold

 

positional arguments:

  min_ms             minimum I/O duration to trace, in ms (default 10)

 

optional arguments:

  -h, --help         show this help message and exit

  -j, --csv          just print fields: comma-separated values

  -p PID, --pid PID  trace this PID only

 

examples:

    ./xfsslower             # trace operations slower than 10 ms (default)

    ./xfsslower 1           # trace operations slower than 1 ms

    ./xfsslower -j 1        # ... 1 ms, parsable output (csv)

    ./xfsslower 0           # trace all operations (warning: verbose)

    ./xfsslower -p 185      # trace PID 185 only

[root@yang-01 tools]# xfsslower 1

Tracing XFS operations slower than 1 ms

TIME     COMM           PID    T BYTES   OFF_KB   LAT(ms) FILENAME

17:02:44 umon-mgr       1684   S 0       0           5.93 detail.log

17:02:44 umc            1638   S 0       0           1.05 detail.log

17:02:45 mysqld         10763  W 81920   0           8.12 temp_16.ibt

17:02:45 mysqld         10763  W 16384   80          2.16 temp_16.ibt

17:02:45 ustats         1642   S 0       0           3.12 brief.log

17:02:46 mysqld         10763  W 835584  1024        7.81 ibdata1

17:02:46 mysqld         10763  S 0       0          31.14 ib_logfile0

17:02:46 ucore          1538   S 0       0           5.89 raft.db

17:02:46 ustats         1642   S 0       0           4.04 detail.log

17:02:46 mysqld         10763  S 0       0           3.58 ib_logfile0

17:02:46 ustats         1642   S 0       0           1.49 brief.log

17:02:46 umc            1638   S 0       0          13.70 brief.log

17:02:46 uguard-mgr     1636   S 0       0           9.62 detail.log

17:02:46 ucore          1538   S 0       0           8.87 detail.log

17:02:48 ustats         1642   S 0       0           2.16 brief.log

17:02:49 ustats         1642   S 0       0           1.41 brief.log

17:02:49 uguard-agent   1664   S 0       0          26.30 brief.log

17:02:49 uguard-agent   1664   S 0       0           6.56 detail.log

17:02:49 mysqld         10763  S 0       0           1.89 ib_logfile0

17:02:50 ustats         1642   S 0       0           1.01 brief.log

17:02:50 mysqld         10763  S 0       0           1.18 ib_logfile0

##################################################################################################################################################################

  

  

# 可以用来实现高效的内核摘要统计。从内核层到用户层的数据传输仅仅是“计数”列。 用户级程序生成其余的。

[root@yang-01 tools]# biolatency -h

usage: biolatency [-h] [-T] [-Q] [-m] [-D] [-F] [interval] [count]

 

Summarize block device I/O latency as a histogram

 

positional arguments:

  interval            output interval, in seconds

  count               number of outputs

 

optional arguments:

  -h, --help          show this help message and exit

  -T, --timestamp     include timestamp on output

  -Q, --queued        include OS queued time in I/O time

  -m, --milliseconds  millisecond histogram

  -D, --disks         print a histogram per disk device

  -F, --flags         print a histogram per set of I/O flags

 

examples:

    ./biolatency            # summarize block I/O latency as a histogram

    ./biolatency 1 10       # print 1 second summaries, 10 times

    ./biolatency -mT 1      # 1s summaries, milliseconds, and timestamps

    ./biolatency -Q         # include OS queued time in I/O time

    ./biolatency -D         # show each disk device separately

    ./biolatency -F         # show I/O flags separately

 

 

[root@yang-01 tools]# biolatency

Tracing block device I/O... Hit Ctrl-C to end.

^C

     usecs               : count     distribution

         0 -> 1          : 0        |                                        |

         2 -> 3          : 0        |                                        |

         4 -> 7          : 0        |                                        |

         8 -> 15         : 0        |                                        |

        16 -> 31         : 13       |*                                       |

        32 -> 63         : 306      |*********************************       |

        64 -> 127        : 367      |****************************************|

       128 -> 255        : 254      |***************************             |

       256 -> 511        : 12       |*                                       |

       512 -> 1023       : 1        |                                        |

      1024 -> 2047       : 1        |                                        |

      2048 -> 4095       : 9        |                                        |

      4096 -> 8191       : 10       |*                                       |

      8192 -> 16383      : 9        |                                        |

     16384 -> 32767      : 3        |                                        |

     32768 -> 65535      : 2        |                                        |

##################################################################################################################################################################

  

  

  

# tcplife显示 TCP 会话的生命周期和吞吐量统计

[root@yang-01 tools]# tcplife -h

usage: tcplife [-h] [-T] [-t] [-w] [-s] [-p PID] [-L LOCALPORT]

               [-D REMOTEPORT]

 

Trace the lifespan of TCP sessions and summarize

 

optional arguments:

  -h, --help            show this help message and exit

  -T, --time            include time column on output (HH:MM:SS)

  -t, --timestamp       include timestamp on output (seconds)

  -w, --wide            wide column output (fits IPv6 addresses)

  -s, --csv             comma separated values output

  -p PID, --pid PID     trace this PID only

  -L LOCALPORT, --localport LOCALPORT

                        comma-separated list of local ports to trace.

  -D REMOTEPORT, --remoteport REMOTEPORT

                        comma-separated list of remote ports to trace.

 

examples:

    ./tcplife           # trace all TCP connect()s

    ./tcplife -t        # include time column (HH:MM:SS)

    ./tcplife -w        # wider colums (fit IPv6)

    ./tcplife -stT      # csv output, with times & timestamps

    ./tcplife -p 181    # only trace PID 181

    ./tcplife -L 80     # only trace local port 80

    ./tcplife -L 80,81  # only trace local ports 80 and 81

    ./tcplife -D 80     # only trace remote port 80

 

 

[root@yang-01 tools]# tcplife

PID   COMM       LADDR           LPORT RADDR           RPORT TX_KB RX_KB MS

854   jsvc       ::ffff:127.0.0.1 33222 ::ffff:127.0.0.1 3307      0     0 1.91

10763 mysqld     ::ffff:127.0.0.1 3307  ::ffff:127.0.0.1 33222     0     0 1.91

854   jsvc       ::ffff:127.0.0.1 33224 ::ffff:127.0.0.1 3307      0     0 2.02

10763 mysqld     ::ffff:127.0.0.1 3307  ::ffff:127.0.0.1 33224     0     0 2.03

1638  umc        10.186.60.24    48697 10.186.60.24    5704      0     0 3.01

1636  uguard-mgr ::ffff:10.186.60.24 5704  ::ffff:10.186.60.24 48697     0     0 3.03

854   jsvc       ::ffff:127.0.0.1 33228 ::ffff:127.0.0.1 3307      0     0 1.76

10763 mysqld     ::ffff:127.0.0.1 3307  ::ffff:127.0.0.1 33228     0     0 1.77

1684  umon-mgr   ::1             37504 ::1             8200      0     0 0.03

1684  umon-mgr   127.0.0.1       48056 127.0.0.1       8200      0     0 0.02

854   jsvc       ::ffff:127.0.0.1 33234 ::ffff:127.0.0.1 3307      0     0 2.02

10763 mysqld     ::ffff:127.0.0.1 3307  ::ffff:127.0.0.1 33234     0     0 2.04

1642  ustats     127.0.0.1       33236 127.0.0.1       3307      0     0 3.13

10763 mysqld     ::ffff:127.0.0.1 3307  ::ffff:127.0.0.1 33236     0     0 3.15

1642  ustats     127.0.0.1       45738 127.0.0.1       5690      0     0 3.33

5448  mysqld     ::ffff:127.0.0.1 5690  ::ffff:127.0.0.1 45738     0     0 3.35

854   jsvc       ::ffff:127.0.0.1 33240 ::ffff:127.0.0.1 3307      0     0 1.65

10763 mysqld     ::ffff:127.0.0.1 3307  ::ffff:127.0.0.1 33240     0     0 1.66

####################################################################################################################################################################

  

  

  

# 检测用于名称解析的 gethostbyname(3) 和相关的库调用

[root@yang-01 tools]# gethostlatency -h

usage: gethostlatency [-h] [-p PID]

 

Show latency for getaddrinfo/gethostbyname[2] calls

 

optional arguments:

  -h, --help         show this help message and exit

  -p PID, --pid PID  trace this PID only

 

examples:

    ./gethostlatency           # trace all TCP accept()s

    ./gethostlatency -p 181    # only trace PID 181

  

[root@yang-01 tools]# gethostlatency

TIME      PID    COMM                  LATms HOST

17:37:55  903    jsvc                   0.28 localhost

17:38:03  22079  umon                   0.28 localhost

17:38:08  14308  umon-mgr               0.23 localhost

17:38:16  919    jsvc                   0.17 yang-01

17:38:25  903    jsvc                   0.28 localhost

17:38:37  24347  umon                   0.25 localhost

17:38:38  1780   umon-mgr               0.34 localhost

17:38:38  22085  umon                   0.22 localhost

17:38:43  1780   umon-mgr               0.22 localhost

17:38:45  893    jsvc                   0.08 yang-01

#####################################################################################################################################################################

  

  

  

#  trace 工具由 Sasha Goldshtein 提供，并提供了一些基本的 printf(1) 功能和自定义探针。

[root@yang-01 tools]# trace -h

usage: trace [-h] [-b BUFFER_PAGES] [-p PID] [-L TID] [-v] [-Z STRING_SIZE]

             [-S] [-M MAX_EVENTS] [-t] [-T] [-C] [-B] [-s SYM_FILE_LIST] [-K]

             [-U] [-a] [-I header]

             probe [probe ...]

 

Attach to functions and print trace messages.

 

positional arguments:

  probe                 probe specifier (see examples)

 

optional arguments:

  -h, --help            show this help message and exit

  -b BUFFER_PAGES, --buffer-pages BUFFER_PAGES

                        number of pages to use for perf_events ring buffer

                        (default: 64)

  -p PID, --pid PID     id of the process to trace (optional)

  -L TID, --tid TID     id of the thread to trace (optional)

  -v, --verbose         print resulting BPF program code before executing

  -Z STRING_SIZE, --string-size STRING_SIZE

                        maximum size to read from strings

  -S, --include-self    do not filter trace's own pid from the trace

  -M MAX_EVENTS, --max-events MAX_EVENTS

                        number of events to print before quitting

  -t, --timestamp       print timestamp column (offset from trace start)

  -T, --time            print time column

  -C, --print_cpu       print CPU id

  -B, --bin_cmp         allow to use STRCMP with binary values

  -s SYM_FILE_LIST, --sym_file_list SYM_FILE_LIST

                        coma separated list of symbol files to use for symbol

                        resolution

  -K, --kernel-stack    output kernel stack trace

  -U, --user-stack      output user stack trace

  -a, --address         print virtual address in stacks

  -I header, --include header

                        additional header files to include in the BPF program

                        as either full path, or relative to current working

                        directory, or relative to default kernel header search

                        path

 

EXAMPLES:

 

trace do_sys_open

        Trace the open syscall and print a default trace message when entered

trace 'do_sys_open "%s", arg2'

        Trace the open syscall and print the filename being opened

trace 'sys_read (arg3 > 20000) "read %d bytes", arg3'

        Trace the read syscall and print a message for reads >20000 bytes

trace 'r::do_sys_open "%llx", retval'

        Trace the return from the open syscall and print the return value

trace 'c:open (arg2 == 42) "%s %d", arg1, arg2'

        Trace the open() call from libc only if the flags (arg2) argument is 42

trace 'c:malloc "size = %d", arg1'

        Trace malloc calls and print the size being allocated

trace 'p:c:write (arg1 == 1) "writing %d bytes to STDOUT", arg3'

        Trace the write() call from libc to monitor writes to STDOUT

trace 'r::__kmalloc (retval == 0) "kmalloc failed!"'

        Trace returns from __kmalloc which returned a null pointer

trace 'r:c:malloc (retval) "allocated = %x", retval'

        Trace returns from malloc and print non-NULL allocated buffers

trace 't:block:block_rq_complete "sectors=%d", args->nr_sector'

        Trace the block_rq_complete kernel tracepoint and print # of tx sectors

trace 'u:pthread:pthread_create (arg4 != 0)'

        Trace the USDT probe pthread_create when its 4th argument is non-zero

trace 'p::SyS_nanosleep(struct timespec *ts) "sleep for %lld ns", ts->tv_nsec'

        Trace the nanosleep syscall and print the sleep duration in ns

trace -I 'linux/fs.h' 

      'p::uprobe_register(struct inode *inode) "a_ops = %llx", inode->i_mapping->a_ops'

        Trace the uprobe_register inode mapping ops, and the symbol can be found

        in /proc/kallsyms

trace -I 'kernel/sched/sched.h' 

      'p::__account_cfs_rq_runtime(struct cfs_rq *cfs_rq) "%d", cfs_rq->runtime_remaining'

        Trace the cfs scheduling runqueue remaining runtime. The struct cfs_rq is defined

        in kernel/sched/sched.h which is in kernel source tree and not in kernel-devel

        package.  So this command needs to run at the kernel source tree root directory

        so that the added header file can be found by the compiler.

trace -I 'net/sock.h' 

      'udpv6_sendmsg(struct sock *sk) (sk->sk_dport == 13568)'

        Trace udpv6 sendmsg calls only if socket's destination port is equal

        to 53 (DNS; 13568 in big endian order)

trace -I 'linux/fs_struct.h' 'mntns_install "users = %d", $task->fs->users'

        Trace the number of users accessing the file system of the current task

  

  

  

内核版本低的限制

# bcc 基于 eBPF 开发（需要 Linux 3.15 及更高版本）。bcc 使用的大部分内容都需要 Linux 4.1 及更高版本。

  

[root@yang-01 tools]# dbstat mysql -p 10763 -u

Traceback (most recent call last):

  File "/usr/share/bcc/tools/dbstat", line 88, in <module>

    usdt.enable_probe("query__start", "probe_start")

  File "/usr/lib/python2.7/site-packages/bcc/usdt.py", line 154, in enable_probe

    probe

bcc.usdt.USDTException: failed to enable probe 'query__start'; a possible cause can be that the probe requires a pid to enable

 

[root@yang-01 tools]# dbslower mysql -p 10763 -m 30

Traceback (most recent call last):

  File "/usr/share/bcc/tools/dbslower", line 193, in <module>

    usdt.enable_probe("query__start", "query_start")

  File "/usr/lib/python2.7/site-packages/bcc/usdt.py", line 154, in enable_probe

    probe

bcc.usdt.USDTException: failed to enable probe 'query__start'; a possible cause can be that the probe requires a pid to enable

 

  

# OS内核版本, >= 4.4 存在统计Bug, 部分功能需要>= 4.13

# "bcc.usdt.USDTException: failed to enable probe 'query__start'; a possible cause can be that the probe requires a pid to enable” 需要有Dtrace tracepoint的MySQL, 需要重新编译. https://dev.mysql.com/doc/refman/5.7/en/dba-dtrace-server.html

# 使用bcc需要root权限

[root@yang-01 tools]# biolatency -D 2

Tracing block device I/O... Hit Ctrl-C to end.

 

disk = 'vda'

     usecs               : count     distribution

         0 -> 1          : 0        |                                        |

         2 -> 3          : 0        |                                        |

         4 -> 7          : 0        |                                        |

         8 -> 15         : 0        |                                        |

        16 -> 31         : 0        |                                        |

        32 -> 63         : 63       |***********************************     |

        64 -> 127        : 71       |****************************************|

       128 -> 255        : 46       |*************************               |

       256 -> 511        : 4        |**                                      |

       512 -> 1023       : 0        |                                        |

      1024 -> 2047       : 0        |                                        |

      2048 -> 4095       : 3        |*                                       |

      4096 -> 8191       : 3        |*                                       |

 

disk = ''

     usecs               : count     distribution

         0 -> 1          : 0        |                                        |

         2 -> 3          : 0        |                                        |

         4 -> 7          : 0        |                                        |

         8 -> 15         : 0        |                                        |

        16 -> 31         : 0        |                                        |

        32 -> 63         : 0        |                                        |

        64 -> 127        : 0        |                                        |

       128 -> 255        : 0        |                                        |

       256 -> 511        : 0        |                                        |

       512 -> 1023       : 0        |                                        |

      1024 -> 2047       : 0        |                                        |

      2048 -> 4095       : 1        |****************************************|

 

 

disk = 'vda'

     usecs               : count     distribution

         0 -> 1          : 0        |                                        |

         2 -> 3          : 0        |                                        |

         4 -> 7          : 0        |                                        |

         8 -> 15         : 0        |                                        |

        16 -> 31         : 10       |***                                     |

        32 -> 63         : 84       |********************************        |

        64 -> 127        : 103      |****************************************|

       128 -> 255        : 43       |****************                        |

       256 -> 511        : 4        |*                                       |

       512 -> 1023       : 0        |                                        |

      1024 -> 2047       : 0        |                                        |

      2048 -> 4095       : 7        |**                                      |

      4096 -> 8191       : 17       |******                                  |

      8192 -> 16383      : 1        |                                        |

     16384 -> 32767      : 1        |                                        |

     32768 -> 65535      : 1        |                                        |

     65536 -> 131071     : 1        |                                        |

    131072 -> 262143     : 44       |*****************                       |

 

disk = ''

     usecs               : count     distribution

         0 -> 1          : 0        |                                        |

         2 -> 3          : 0        |                                        |

         4 -> 7          : 0        |                                        |

         8 -> 15         : 0        |                                        |

        16 -> 31         : 0        |                                        |

        32 -> 63         : 0        |                                        |

        64 -> 127        : 0        |                                        |

       128 -> 255        : 0        |                                        |

       256 -> 511        : 0        |                                        |

       512 -> 1023       : 0        |                                        |

      1024 -> 2047       : 0        |                                        |

      2048 -> 4095       : 1        |****************************************|

 

  

  

[root@yang-01 tools]# ps aux | grep mysql | grep opt

actiont+ 10763  3.3 18.0 4570952 1070628 ?     Sl   Jul16 1887:06 /opt/mysql/base/8.0.19/bin/mysqld --defaults-file=/opt/mysql/etc/3307/my.cnf --daemonize --pid-file=/opt/mysql/data/3307/mysqld.pid --user=actiontech-mysql --socket=/opt/mysql/data/3307/mysqld.sock --port=3307

[root@yang-01 tools]# filetop -p 10763 -C 5

Tracing... Output every 5 secs. Hit Ctrl-C to end

 

18:01:40 loadavg: 0.14 0.15 0.20 2/1119 31195

 

TID    COMM             READS  WRITES R_Kb    W_Kb    T FILE

10788  mysqld           0      1      0       832     R ibdata1

10909  mysqld           19     0      117     0       R mysql-relay.000044

10794  mysqld           0      99     0       67      R ib_logfile0

10977  mysqld           0      60     0       4       R mysql-relay.000044

10910  mysqld           0      10     0       4       R mysql-bin.000015

10789  mysqld           0      1      0       0       R ib_logfile0

 

18:01:45 loadavg: 0.21 0.16 0.21 2/1119 31212

 

TID    COMM             READS  WRITES R_Kb    W_Kb    T FILE

10909  mysqld           19     0      116     0       R mysql-relay.000044

10794  mysqld           0      98     0       64      R ib_logfile0

10977  mysqld           0      60     0       4       R mysql-relay.000044

10910  mysqld           0      10     0       4       R mysql-bin.000015

 

18:01:50 loadavg: 0.19 0.16 0.20 2/1119 31226

 

TID    COMM             READS  WRITES R_Kb    W_Kb    T FILE

10909  mysqld           15     0      88      0       R mysql-relay.000044

10794  mysqld           0      106    0       70      R ib_logfile0

19780  mysqld           2      2      8       7       R ibzMa2hy

10977  mysqld           0      60     0       4       R mysql-relay.000044

10910  mysqld           0      10     0       4       R mysql-bin.000015

19780  mysqld           1      0      0       0       R mysql-bin.000009

19780  mysqld           1      0      0       0       R mysql-bin.000012

19780  mysqld           1      0      0       0       R mysql-bin.000014

19780  mysqld           1      0      0       0       R mysql-bin.000010

19780  mysqld           1      0      0       0       R mysql-bin.000013

19780  mysqld           1      0      0       0       R mysql-bin.000011

 

 

 

[root@yang-01 tools]# filelife

TIME     PID    COMM             AGE(s)  FILE

18:03:26 1823   ustats           0.00    disk-read-write-test.log

18:03:26 1830   ustats           0.00    disk-read-write-test.log

18:03:29 1736   umon-mgr         0.01    grafana.db-journal

18:03:41 1823   ustats           0.00    disk-read-write-test.log

18:03:41 1784   ustats           0.01    disk-read-write-test.log