web服务器卡死，登陆到后台查看问题； ps aux执行的时候发现卡死，
重新ssh 登陆 strace ps 发现如下结果：
使用gdb 调试也是卡死！
使用top -b 查看所有的进程，发现 之前的ps 的进程为D状态， 同时web服务器 部分线程进程为D状态；
dmesg 查看结果发现：
[20761.085669] INFO: task apache2:7135 blocked for more than 120 seconds. [20761.085675] Tainted: G W O #4 [20761.085677] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [20761.085679] apache2 D ffffffc000086ef8 0 7135 4035 0x00000000 [20761.085683] Call trace: [20761.085736] [<ffffffc000086ef8>] __switch_to+0xa0/0xb8 [20761.085767] [<ffffffc0009ff10c>] __schedule+0x24c/0x704 [20761.085769] [<ffffffc0009ff5fc>] schedule+0x38/0x90 [20761.085781] [<ffffffc000a0260c>] rwsem_down_write_failed+0x1d8/0x310 [20761.085783] [<ffffffc000a00928>] down_write+0x5c/0x74 [20761.085795] [<ffffffc0002188d4>] SyS_mprotect+0xb0/0x204 [20761.085797] [<ffffffc000085c74>] el0_svc_naked+0x24/0x28 [20761.085799] INFO: task apache2:7138 blocked for more than 120 seconds. [20761.085800] Tainted: G W O YUN #4 [20761.085801] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [20761.085802] apache2 D ffffffc000086ef8 0 7138 4035 0x00000000 [20761.085805] Call trace: [20761.085807] [<ffffffc000086ef8>] __switch_to+0xa0/0xb8 [20761.085809] [<ffffffc0009ff10c>] __schedule+0x24c/0x704 [20761.085811] [<ffffffc0009ff5fc>] schedule+0x38/0x90 [20761.085813] [<ffffffc000a0260c>] rwsem_down_write_failed+0x1d8/0x310 [20761.085815] [<ffffffc000a00928>] down_write+0x5c/0x74 [20761.085818] [<ffffffc000249344>] split_huge_page_to_list+0x64/0x7e4 [20761.085819] [<ffffffc00024a570>] __split_huge_page_pmd+0x120/0x354 [20761.085821] [<ffffffc00020dc88>] unmap_single_vma+0x178/0x644 [20761.085823] [<ffffffc00020ec48>] zap_page_range+0xa8/0x114 [20761.085825] [<ffffffc000221150>] SyS_madvise+0x2f4/0x520 [20761.085827] [<ffffffc000085c74>] el0_svc_naked+0x24/0x28 [20761.085828] INFO: task apache2:7158 blocked for more than 120 seconds. [20761.085830] Tainted: G W O server.YUN #4 [20761.085831] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [20761.085832] apache2 D ffffffc000086ef8 0 7158 4035 0x00000008 [20761.085834] Call trace: [20761.085836] [<ffffffc000086ef8>] __switch_to+0xa0/0xb8 [20761.085838] [<ffffffc0009ff10c>] __schedule+0x24c/0x704 [20761.085840] [<ffffffc0009ff5fc>] schedule+0x38/0x90 [20761.085842] [<ffffffc000a0260c>] rwsem_down_write_failed+0x1d8/0x310 [20761.085843] [<ffffffc000a00928>] down_write+0x5c/0x74 [20761.085845] [<ffffffc000249344>] split_huge_page_to_list+0x64/0x7e4 [20761.085847] [<ffffffc00024a570>] __split_huge_page_pmd+0x120/0x354 [20761.085849] [<ffffffc00020dc88>] unmap_single_vma+0x178/0x644 [20761.085850] [<ffffffc00020ec48>] zap_page_range+0xa8/0x114 [20761.085852] [<ffffffc000221150>] SyS_madvise+0x2f4/0x520 [20761.085854] [<ffffffc000085c74>] el0_svc_naked+0x24/0x28 [20761.085856] INFO: task ps:17403 blocked for more than 120 seconds. [20761.085857] Tainted: G W O #4
查看内核代码只接原因为：fs/proc/base.c 文件中的proc_pid_cmdline_read 函数执行如下代码发生获取信号量失败而导致休眠
down_read(&mm->mmap_sem); arg_start = mm->arg_start; arg_end = mm->arg_end; env_start = mm->env_start; env_end = mm->env_end; up_read(&mm->mmap_sem);
void __sched down_read(struct rw_semaphore *sem) { might_sleep(); rwsem_acquire_read(&sem->dep_map, 0, 0, _RET_IP_); LOCK_CONTENDED(sem, __down_read_trylock, __down_read); } /* * lock for reading */ static inline void __down_read(struct rw_semaphore *sem) { if (unlikely(atomic_long_inc_return_acquire((atomic_long_t *)&sem->count) <= 0)) rwsem_down_read_failed(sem); }
那是什么进程获取此sem没有释放呢？
目前怎样查看？------>首先需要获取内核的堆栈
同时目前google 结果发现：内核有相关patch对此进行修改；见内核patch
