整理自man 8 crash
1、简介
Crash工具可以用来分析一个正在运行的内核,也可以用来分析一个内核的crash dump文件,这里说的是内核代码异常产生的crash dump文件,不是应用层程序运行异常产生的core dump文件,它支持分析由netdump, diskdump, LKCD, kdump, xen‐dump 或者 kvmdump 工具产生的crash dump文件。它整合了SVR4 UNIX crash的工具和GDB调试器,因而具有源码级别调试能力。Crash工具可以用来分析内核的调用堆栈,内核源码的反汇编,内核数据结构和变量的格式化展示等等,另外Crash也可以传递一些GDB命令来执行。在初始化的过程中,
$HOME/.crashrc 文件中的命令和当前目录下的./.crashrc文件中的命令将会被执行。 Crash工具后向兼容,当内核版本变化导致Crash工具更新后后仍然会兼容以前的内核版本。
2、启动命令行
crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS] (用来分析dumpfile文件)
crash [OPTION]... [NAMELIST] (用来分析正在运行的系统)
NAMELIST:这个是未压缩的内核镜像文件(vmlinux)的路径,在用来分析dumpfile文件的时候也可以使用gzip或者bzip2压缩后的vmlinux文件。
MEMORY-IMAGE[@ADDRESS]:由netdump, diskdump, LKCD, kdump, xen‐dump 或者 kvmdump 工具产生的crash dump文件的路径,如果运行Crash工具没有输入这个参数的话,那么Crash工具将会用来分析正在运行的linux内核,分析正在运行的内核需要访问系统的RAM,一般是需要root权限的。分析live system的时候,默认情况下/dev/crash将会被使用,如果这个文件不存在,然后会使用/dev/mem,但是如果kernel被配置为CONFIG_STRICT_DEVMEM,那么/proc/kcore将会被使用,也可以显式的指定/dev/crash、/dev/mem 、 /proc/kcore。
分析live system的时候,不指定NAMELIST场景下,crash也会从一些默认目录查找对应的NAMELIST,如果找到的和正在运行的不匹配,没那么会提示如下错误crash: invalid kernel virtual address:
如果dumpfile里面没有描述文件内容的头信息,即原始RAM的dumpfile文件,那么对应块的物理起始地址需要通过@ADDRESS以16进制指定,这将会在/var/tmp目录下创建临时的ELF信息。这种场景下可以通过MEMORY-IMAGE@ADDRESS指定多个dumofile文件。
可以看到如果要用来分析内核的core dump文件,那么必须输入NAMELIST MEMORY-IMAGE这两个参数。而如果用来分析正在运行的linux系统的话,如果内核的vmlinux没有在固定的位置(如/boot),那么NAMELIST参数也需要指定。
3、相关选项
mapfile:当分析正在运行的内核的时候,如果NAMELIST文件与当前正在运行的内核不一致,或者分析dumpfile的时候,NAMELIST与产生dumpfile的内核不一致,那么原始内核的System.map文件需要在命令行指定。
-h [option]/--help [option]:不带有option的时候,展示crash工具的帮助,如果option是carsh工具的命令名字,那么会展示对应的帮助。另外option也可以是"input"、"output"或者"all"。
-s:静默处理到"crash>" ,不会展示版本信息、初始化过程等。
-i file:在展示"crash>"之前,执行file文件中的命令
-d num:设置debug level,数值越高,crash在初始化和运行的时候就会展示越多的信息
-S:使用 /boot/System.map 作为mapfile
-t:展示系统crash的事件戳并推出
-c tty-device:使用tty-device作为调试信息的控制台
-p page-size:如果处理器的分页大小不能通过dumpfile判断出来,并且不能使用处理器的默认值,那么可以通过这个选项来指定
-o filename:使用MEMORY-IMAGE@ADDRESS形式的原始RAM的dumpfile时候,这个选项可以用来创建一个独立的ELF vmcore,以备将来使用
-x:自动的从一个指定目录加载扩展模块。环境变量CRASH_EXTENSIONS可以用来指定加载目录。
--active:仅跟踪每个cpu上正在运行的task。
--buildinfo:展示crash的build信息
--memory_module modname:使用modname作为crash.ko的替代模块来创建/dev/crash设备。
--memory_device device:使用device代替/dev/crash、/dev/mem或者/proc/kcore。
--smp:指定分析的系统是smp内核
-v:展示crash的版本信息
--cpus number:指定smp系统的cpu数目
--more/--less:指定输出工具
--hex/--dec:指定命令输出的进制
4、crash命令
crash命令分为符号展示、系统状态、实用函数和会话控制四部分。可以通过crash -h <command>来查看命令帮助,在crash会话中也可以通过help command来查看命令的帮助。
alias:创建命令的别名
bt:展示调用堆栈信息,-a选项可以显示所有cpu上的调用堆栈。foreach bt可以显示所有task的调用堆栈
dev:展示字符设备和块设备的分配、I/O端口使用和I/O内存使用情况,这个命令可以用来查看每个设备对应的fops。
dis:反汇编命令。-f可以从函数的开始处开始展示汇编命令到指定的地址,-f可以从指定地址展示到函数的结尾。-l可以展示源代码行。-s可以显示对应的源代码,count可以指定显示的行数。
eval:计算表达式并展示结果。
q/exit:退出crash
foreach:为指定的task重复一个指定的命令。
ipcs:展示System V IPC信息
irq:展示irq信息
kmem:展示内核内存使用
list:展示一个链表的内容
log:展示log_buf内容
mod:展示模块信息,mod -s tcp_debug [pathtofile],命令可以用来加载tcp_debug模块的调试信息。mod -d可以用来删除模块的调试信息
mount:展示当前挂载的文件系统的命令
net:展示网络相关的信息
p:传递参数给gdb的print命令来展示相关信息。
gdb:传递参数给内嵌的GDB模块,当crash和gdb命令名字冲突的时候可以使用。
ps:展示进程状态
rd:读取并展示内存
repeat:无限重复一个命令,可以给定一个命令执行的间隔
runq:展示runq上的task
search:在一段内存范围内搜索给定值
set:设置或者获取上下文。可以根据pid、task_struct等来进行设置。
sig:展示task的signal-handling数据
struct:展示结构体的定义,或者从指定的地址开始解析一个结构体。使用-o参数可以显示结构体的详细偏移信息。
swap:展示swap信息
sys:展示系统信息
task:展示task_struct内容
tree:展示rb树或者radix树的信息
timer:展示定时器信息。
union:与struct类似,但是用于union的展示
whatis:展示结构体、联合体等定义
waitq:展示等待队列的内容
wr:修改一个正在运行的系统的内存,只有使用/dev/mem的时候才能修改。
root@Inspiron:/home/lybxin# crash /doc/ubuntu-compile/vmlinux /var/crash/201610291840/dump.201610291840
crash 7.1.4
Copyright(C)2002-2015 RedHat,Inc.
Copyright(C)2004,2005,2006,2010 IBM Corporation
Copyright(C)1999-2006 Hewlett-PackardCo
Copyright(C)2005,2006,2011,2012 FujitsuLimited
Copyright(C)2006,2007 VA LinuxSystemsJapan K.K.
Copyright(C)2005,2011 NEC Corporation
Copyright(C)1999,2002,2007 SiliconGraphics,Inc.
Copyright(C)1999,2000,2001,2002 MissionCriticalLinux,Inc.
This program is free software, covered by the GNU GeneralPublicLicense,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter"help copying" to see the conditions.
This program has absolutely no warranty. Enter"help warranty"for details.
GNU gdb (GDB)7.6
Copyright(C)2013FreeSoftwareFoundation,Inc.
LicenseGPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type"show copying"
and "show warranty"for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
KERNEL:/doc/ubuntu-compile/vmlinux
DUMPFILE:/var/crash/201610291840/dump.201610291840 [PARTIAL DUMP]
CPUS:4
DATE:SatOct2918:40:252016
UPTIME:01:21:59
LOAD AVERAGE:0.39,0.50,0.50
TASKS:643
NODENAME:Inspiron
RELEASE:4.4.13+
VERSION:#35 SMP Fri Oct 28 23:13:30 CST 2016
MACHINE: x86_64 (2526Mhz)
MEMORY:3.9 GB
PANIC:"general protection fault: 0000 [#1] SMP "
PID:5792
COMMAND:"server.out"
TASK: ffff880013531900 [THREAD_INFO: ffff8800121ac000]
CPU:3
STATE: (PANIC)
crash> bt
PID:5792 TASK: ffff880013531900 CPU:3 COMMAND:"server.out"
#0 [ffff880137d839b0] machine_kexec at ffffffff8105ae6b
#1 [ffff880137d83a10] crash_kexec at ffffffff8110cb12
#2 [ffff880137d83ae0] oops_end at ffffffff81030c29
#3 [ffff880137d83b08] die at ffffffff810310db
#4 [ffff880137d83b38] do_general_protection at ffffffff8102e5d9
#5 [ffff880137d83b60] general_protection at ffffffff81832118
[exception RIP: tcp_v4_rcv+2735]
RIP: ffffffff8178920f RSP: ffff880137d83c18 RFLAGS:00010282
RAX:6601a8c0647c770e RBX: ffff880131d04ae8 RCX:0000000000000006
RDX:0000000000000000 RSI: ffff880013531900 RDI: ffff880137d8dc78
RBP: ffff880137d83c98 R8:000000000000000a R9:0000000000000413
R10:000000000000c5c0 R11:0000000000000413 R12: ffffffff81ef3e80
R13: ffff8800ba0e4690 R14: ffff8800b7cde4fc R15: ffff8800b7cde510
ORIG_RAX: ffffffffffffffff CS:0010 SS:0018
#6 [ffff880137d83c10] tcp_v4_rcv at ffffffff817892a9
#7 [ffff880137d83ca0] ip_local_deliver_finish at ffffffff8175fdc4
#8 [ffff880137d83cc8] ip_local_deliver at ffffffff817600cf
#9 [ffff880137d83d30] ip_rcv_finish at ffffffff8175faa2
#10 [ffff880137d83d60] ip_rcv at ffffffff817603d1
#11 [ffff880137d83dd8] __netif_receive_skb_core at ffffffff81721944
#12 [ffff880137d83e70] __netif_receive_skb at ffffffff81721cb8
#13 [ffff880137d83e90] process_backlog at ffffffff81722ab8
-- MORE -- forward:<SPACE>,<ENTER> or j backward: b or k quit: q...skipping...
PID:5792 TASK: ffff880013531900 CPU:3 COMMAND:"server.out"
#0 [ffff880137d839b0] machine_kexec at ffffffff8105ae6b
#1 [ffff880137d83a10] crash_kexec at ffffffff8110cb12
#2 [ffff880137d83ae0] oops_end at ffffffff81030c29
#3 [ffff880137d83b08] die at ffffffff810310db
#4 [ffff880137d83b38] do_general_protection at ffffffff8102e5d9
#5 [ffff880137d83b60] general_protection at ffffffff81832118
[exception RIP: tcp_v4_rcv+2735]
RIP: ffffffff8178920f RSP: ffff880137d83c18 RFLAGS:00010282
RAX:6601a8c0647c770e RBX: ffff880131d04ae8 RCX:0000000000000006
RDX:0000000000000000 RSI: ffff880013531900 RDI: ffff880137d8dc78
RBP: ffff880137d83c98 R8:000000000000000a R9:0000000000000413
R10:000000000000c5c0 R11:0000000000000413 R12: ffffffff81ef3e80
R13: ffff8800ba0e4690 R14: ffff8800b7cde4fc R15: ffff8800b7cde510
ORIG_RAX: ffffffffffffffff CS:0010 SS:0018
#6 [ffff880137d83c10] tcp_v4_rcv at ffffffff817892a9
#7 [ffff880137d83ca0] ip_local_deliver_finish at ffffffff8175fdc4
#8 [ffff880137d83cc8] ip_local_deliver at ffffffff817600cf
#9 [ffff880137d83d30] ip_rcv_finish at ffffffff8175faa2
#10 [ffff880137d83d60] ip_rcv at ffffffff817603d1
#11 [ffff880137d83dd8] __netif_receive_skb_core at ffffffff81721944
#12 [ffff880137d83e70] __netif_receive_skb at ffffffff81721cb8
#13 [ffff880137d83e90] process_backlog at ffffffff81722ab8
#14 [ffff880137d83ed0] net_rx_action at ffffffff817221fe
-- MORE -- forward:<SPACE>,<ENTER> or j backward: b or k quit: q...skipping...
PID:5792 TASK: ffff880013531900 CPU:3 COMMAND:"server.out"
#0 [ffff880137d839b0] machine_kexec at ffffffff8105ae6b
#1 [ffff880137d83a10] crash_kexec at ffffffff8110cb12
#2 [ffff880137d83ae0] oops_end at ffffffff81030c29
#3 [ffff880137d83b08] die at ffffffff810310db
#4 [ffff880137d83b38] do_general_protection at ffffffff8102e5d9
#5 [ffff880137d83b60] general_protection at ffffffff81832118
[exception RIP: tcp_v4_rcv+2735]
RIP: ffffffff8178920f RSP: ffff880137d83c18 RFLAGS:00010282
RAX:6601a8c0647c770e RBX: ffff880131d04ae8 RCX:0000000000000006
RDX:0000000000000000 RSI: ffff880013531900 RDI: ffff880137d8dc78
RBP: ffff880137d83c98 R8:000000000000000a R9:0000000000000413
R10:000000000000c5c0 R11:0000000000000413 R12: ffffffff81ef3e80
R13: ffff8800ba0e4690 R14: ffff8800b7cde4fc R15: ffff8800b7cde510
ORIG_RAX: ffffffffffffffff CS:0010 SS:0018
#6 [ffff880137d83c10] tcp_v4_rcv at ffffffff817892a9
#7 [ffff880137d83ca0] ip_local_deliver_finish at ffffffff8175fdc4
#8 [ffff880137d83cc8] ip_local_deliver at ffffffff817600cf
#9 [ffff880137d83d30] ip_rcv_finish at ffffffff8175faa2
#10 [ffff880137d83d60] ip_rcv at ffffffff817603d1
#11 [ffff880137d83dd8] __netif_receive_skb_core at ffffffff81721944
#12 [ffff880137d83e70] __netif_receive_skb at ffffffff81721cb8
#13 [ffff880137d83e90] process_backlog at ffffffff81722ab8
#14 [ffff880137d83ed0] net_rx_action at ffffffff817221fe
#15 [ffff880137d83f48] __do_softirq at ffffffff81084a41
-- MORE -- forward:<SPACE>,<ENTER> or j backward: b or k quit: q...skipping...
PID:5792 TASK: ffff880013531900 CPU:3 COMMAND:"server.out"
#0 [ffff880137d839b0] machine_kexec at ffffffff8105ae6b
#1 [ffff880137d83a10] crash_kexec at ffffffff8110cb12
#2 [ffff880137d83ae0] oops_end at ffffffff81030c29
#3 [ffff880137d83b08] die at ffffffff810310db
#4 [ffff880137d83b38] do_general_protection at ffffffff8102e5d9
#5 [ffff880137d83b60] general_protection at ffffffff81832118
[exception RIP: tcp_v4_rcv+2735]
RIP: ffffffff8178920f RSP: ffff880137d83c18 RFLAGS:00010282
RAX:6601a8c0647c770e RBX: ffff880131d04ae8 RCX:0000000000000006
RDX:0000000000000000 RSI: ffff880013531900 RDI: ffff880137d8dc78
RBP: ffff880137d83c98 R8:000000000000000a R9:0000000000000413
R10:000000000000c5c0 R11:0000000000000413 R12: ffffffff81ef3e80
R13: ffff8800ba0e4690 R14: ffff8800b7cde4fc R15: ffff8800b7cde510
ORIG_RAX: ffffffffffffffff CS:0010 SS:0018
#6 [ffff880137d83c10] tcp_v4_rcv at ffffffff817892a9
#7 [ffff880137d83ca0] ip_local_deliver_finish at ffffffff8175fdc4
#8 [ffff880137d83cc8] ip_local_deliver at ffffffff817600cf
#9 [ffff880137d83d30] ip_rcv_finish at ffffffff8175faa2
#10 [ffff880137d83d60] ip_rcv at ffffffff817603d1
#11 [ffff880137d83dd8] __netif_receive_skb_core at ffffffff81721944
#12 [ffff880137d83e70] __netif_receive_skb at ffffffff81721cb8
#13 [ffff880137d83e90] process_backlog at ffffffff81722ab8
#14 [ffff880137d83ed0] net_rx_action at ffffffff817221fe
#15 [ffff880137d83f48] __do_softirq at ffffffff81084a41
#16 [ffff880137d83fb0] do_softirq_own_stack at ffffffff81831d0c
-- MORE -- forward:<SPACE>,<ENTER> or j backward: b or k quit: q...skipping...
PID:5792 TASK: ffff880013531900 CPU:3 COMMAND:"server.out"
#0 [ffff880137d839b0] machine_kexec at ffffffff8105ae6b
#1 [ffff880137d83a10] crash_kexec at ffffffff8110cb12
#2 [ffff880137d83ae0] oops_end at ffffffff81030c29
#3 [ffff880137d83b08] die at ffffffff810310db
#4 [ffff880137d83b38] do_general_protection at ffffffff8102e5d9
#5 [ffff880137d83b60] general_protection at ffffffff81832118
[exception RIP: tcp_v4_rcv+2735]
RIP: ffffffff8178920f RSP: ffff880137d83c18 RFLAGS:00010282
RAX:6601a8c0647c770e RBX: ffff880131d04ae8 RCX:0000000000000006
RDX:0000000000000000 RSI: ffff880013531900 RDI: ffff880137d8dc78
RBP: ffff880137d83c98 R8:000000000000000a R9:0000000000000413
R10:000000000000c5c0 R11:0000000000000413 R12: ffffffff81ef3e80
R13: ffff8800ba0e4690 R14: ffff8800b7cde4fc R15: ffff8800b7cde510
ORIG_RAX: ffffffffffffffff CS:0010 SS:0018
#6 [ffff880137d83c10] tcp_v4_rcv at ffffffff817892a9
#7 [ffff880137d83ca0] ip_local_deliver_finish at ffffffff8175fdc4
#8 [ffff880137d83cc8] ip_local_deliver at ffffffff817600cf
#9 [ffff880137d83d30] ip_rcv_finish at ffffffff8175faa2
#10 [ffff880137d83d60] ip_rcv at ffffffff817603d1
#11 [ffff880137d83dd8] __netif_receive_skb_core at ffffffff81721944
#12 [ffff880137d83e70] __netif_receive_skb at ffffffff81721cb8
#13 [ffff880137d83e90] process_backlog at ffffffff81722ab8
#14 [ffff880137d83ed0] net_rx_action at ffffffff817221fe
#15 [ffff880137d83f48] __do_softirq at ffffffff81084a41
#16 [ffff880137d83fb0] do_softirq_own_stack at ffffffff81831d0c
---<IRQ stack>---
-- MORE -- forward:<SPACE>,<ENTER> or j backward: b or k quit: q...skipping...
PID:5792 TASK: ffff880013531900 CPU:3 COMMAND:"server.out"
#0 [ffff880137d839b0] machine_kexec at ffffffff8105ae6b
#1 [ffff880137d83a10] crash_kexec at ffffffff8110cb12
#2 [ffff880137d83ae0] oops_end at ffffffff81030c29
#3 [ffff880137d83b08] die at ffffffff810310db
#4 [ffff880137d83b38] do_general_protection at ffffffff8102e5d9
#5 [ffff880137d83b60] general_protection at ffffffff81832118
[exception RIP: tcp_v4_rcv+2735]
RIP: ffffffff8178920f RSP: ffff880137d83c18 RFLAGS:00010282
RAX:6601a8c0647c770e RBX: ffff880131d04ae8 RCX:0000000000000006
RDX:0000000000000000 RSI: ffff880013531900 RDI: ffff880137d8dc78
RBP: ffff880137d83c98 R8:000000000000000a R9:0000000000000413
R10:000000000000c5c0 R11:0000000000000413 R12: ffffffff81ef3e80
R13: ffff8800ba0e4690 R14: ffff8800b7cde4fc R15: ffff8800b7cde510
ORIG_RAX: ffffffffffffffff CS:0010 SS:0018
#6 [ffff880137d83c10] tcp_v4_rcv at ffffffff817892a9
#7 [ffff880137d83ca0] ip_local_deliver_finish at ffffffff8175fdc4
#8 [ffff880137d83cc8] ip_local_deliver at ffffffff817600cf
#9 [ffff880137d83d30] ip_rcv_finish at ffffffff8175faa2
#10 [ffff880137d83d60] ip_rcv at ffffffff817603d1
#11 [ffff880137d83dd8] __netif_receive_skb_core at ffffffff81721944
#12 [ffff880137d83e70] __netif_receive_skb at ffffffff81721cb8
#13 [ffff880137d83e90] process_backlog at ffffffff81722ab8
#14 [ffff880137d83ed0] net_rx_action at ffffffff817221fe
#15 [ffff880137d83f48] __do_softirq at ffffffff81084a41
#16 [ffff880137d83fb0] do_softirq_own_stack at ffffffff81831d0c
---<IRQ stack>---
#17 [ffff8800121af888] do_softirq_own_stack at ffffffff81831d0c
-- MORE -- forward:<SPACE>,<ENTER> or j backward: b or k quit: q...skipping...
PID:5792 TASK: ffff880013531900 CPU:3 COMMAND:"server.out"
#0 [ffff880137d839b0] machine_kexec at ffffffff8105ae6b
#1 [ffff880137d83a10] crash_kexec at ffffffff8110cb12
#2 [ffff880137d83ae0] oops_end at ffffffff81030c29
#3 [ffff880137d83b08] die at ffffffff810310db
#4 [ffff880137d83b38] do_general_protection at ffffffff8102e5d9
#5 [ffff880137d83b60] general_protection at ffffffff81832118
[exception RIP: tcp_v4_rcv+2735]
RIP: ffffffff8178920f RSP: ffff880137d83c18 RFLAGS:00010282
RAX:6601a8c0647c770e RBX: ffff880131d04ae8 RCX:0000000000000006
RDX:0000000000000000 RSI: ffff880013531900 RDI: ffff880137d8dc78
RBP: ffff880137d83c98 R8:000000000000000a R9:0000000000000413
R10:000000000000c5c0 R11:0000000000000413 R12: ffffffff81ef3e80
R13: ffff8800ba0e4690 R14: ffff8800b7cde4fc R15: ffff8800b7cde510
ORIG_RAX: ffffffffffffffff CS:0010 SS:0018
#6 [ffff880137d83c10] tcp_v4_rcv at ffffffff817892a9
#7 [ffff880137d83ca0] ip_local_deliver_finish at ffffffff8175fdc4
#8 [ffff880137d83cc8] ip_local_deliver at ffffffff817600cf
#9 [ffff880137d83d30] ip_rcv_finish at ffffffff8175faa2
#10 [ffff880137d83d60] ip_rcv at ffffffff817603d1
#11 [ffff880137d83dd8] __netif_receive_skb_core at ffffffff81721944
#12 [ffff880137d83e70] __netif_receive_skb at ffffffff81721cb8
#13 [ffff880137d83e90] process_backlog at ffffffff81722ab8
#14 [ffff880137d83ed0] net_rx_action at ffffffff817221fe
#15 [ffff880137d83f48] __do_softirq at ffffffff81084a41
#16 [ffff880137d83fb0] do_softirq_own_stack at ffffffff81831d0c
---<IRQ stack>---
#17 [ffff8800121af888] do_softirq_own_stack at ffffffff81831d0c
[exception RIP: unknown or invalid address]
RIP:0000000088c26cce RSP: ffffffff81ef3e80 RFLAGS: ffff880131d04ae8
RAX:0000000000000000 RBX: ffff8800121af8c8 RCX: ffff8800121af940
RDX: ffffffff81764479 RSI:0202020000000001 RDI:0000000000000246
RBP: ffff880132960e00 R8: ffffffff8108450d R9: ffff8800121af8e0
R10:00000000fffffe01 R11: ffffffff81084488 R12:0000000088c26cce
R13:0000000000000001 R14: ffffffff811ad08c R15: ffffffff81724156
ORIG_RAX:0100007f00000000 CS:2000200 SS: ffff8800b81b2d00
bt: WARNING: possibly bogus exception frame
#18 [ffff8800121af948] ip_finish_output at ffffffff81765406
#19 [ffff8800121af988] ip_output at ffffffff81765dfe
#20 [ffff8800121af9f8] ip_local_out at ffffffff817655c5
#21 [ffff8800121afa20] ip_queue_xmit at ffffffff817658e4
#22 [ffff8800121afa60] tcp_transmit_skb at ffffffff8177f25f
#23 [ffff8800121afaf8] tcp_write_xmit at ffffffff8177f925
#24 [ffff8800121afb78] __tcp_push_pending_frames at ffffffff81780c43
#25 [ffff8800121afba0] tcp_send_fin at ffffffff817827c8
#26 [ffff8800121afbd0] tcp_close at ffffffff817729e2
#27 [ffff8800121afc00] inet_release at ffffffff8179de0c
#28 [ffff8800121afc20] sock_release at ffffffff8170399f
#29 [ffff8800121afc40] sock_close at ffffffff81703a12
#30 [ffff8800121afc50] __fput at ffffffff8120da14
#31 [ffff8800121afc98] ____fput at ffffffff8120db8e
#32 [ffff8800121afca8] task_work_run at ffffffff8109da8f
#33 [ffff8800121afcd8] do_exit at ffffffff81082c37
#34 [ffff8800121afd48] do_group_exit at ffffffff810834c3
#35 [ffff8800121afd78] get_signal at ffffffff8108f672
#36 [ffff8800121afdf8] do_signal at ffffffff8102d537
#37 [ffff8800121afef8] exit_to_usermode_loop at ffffffff8100320c
#38 [ffff8800121aff30] syscall_return_slowpath at ffffffff81003c6e
#39 [ffff8800121aff50] int_ret_from_sys_call at ffffffff81830150
RIP:00007f0aaf3f40fa RSP:00007f0aaf2ddd40 RFLAGS:00000293
RAX: fffffffffffffdfc RBX:0000000000000000 RCX:00007f0aaf3f40fa
RDX:00007f0aaf2ddd80 RSI:0000000000000000 RDI:0000000000000001
RBP:00007f0aaf2ddd80 R8:0000000000000000 R9:0000000000000007
R10:0000000000000000 R11:0000000000000293 R12:0000000000000000
R13:00007ffeba0f765f R14:00007f0aaf2de9c0 R15:0000000000000000
ORIG_RAX:00000000000000e6 CS:0033 SS:002b
crash>
crash> dis -l tcp_v4_rcv+2735
/home/lybxin/MyRes/linux-source-4.4.0/net/ipv4/tcp_ipv4.c:1663
0xffffffff8178920f<tcp_v4_rcv+2735>: mov 0x5d0(%rax),%rdi