第一步： 查看消耗CPU最高的进程PID
[lolaage@web2 tomcat-ns]$ top
top - 13:23:32 up 42 days, 19:11, 3 users, load average: 1.01, 0.86, 0.78
Tasks: 153 total, 2 running, 151 sleeping, 0 stopped, 0 zombie
%Cpu(s): 16.3 us, 0.8 sy, 0.0 ni, 82.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 16267172 total, 1135628 free, 6554372 used, 8577172 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 9275688 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23496 lolaage 20 0 9985736 1.978g 11148 S 104.8 12.7 32:17.81 java
5081 lolaage 20 0 1289012 328804 2012 S 9.7 2.0 2562:11 srs
1958 lolaage 20 0 10.631g 2.809g 7520 S 8.9 18.1 2332:34 java
1 root 20 0 43800 3700 1932 S 0.0 0.0 6:29.39 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:01.03 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 2:12.16 ksoftirqd/0
第二步：查看消耗CPU最高的线程TID
[lolaage@video1 mpmt-socket]$ ps -mp 23496 -o THREAD,tid,time
USER %CPU PRI SCNT WCHAN USER SYSTEM TID TIME
lolaage 51.6 - - - - - - 00:32:22
lolaage 0.0 19 - futex_ - - 23496 00:00:00
lolaage 0.3 19 - futex_ - - 23508 00:00:14
lolaage 0.0 19 - ep_pol - - 23530 00:00:00
lolaage 0.0 19 - futex_ - - 23539 00:00:00
lolaage 4.5 19 - futex_ - - 23541 00:02:51
lolaage 4.5 19 - futex_ - - 23543 00:02:51
lolaage 4.5 19 - futex_ - - 23545 00:02:52
lolaage 4.5 19 - futex_ - - 23546 00:02:51
lolaage 4.5 19 - futex_ - - 23547 00:02:51
lolaage 4.6 19 - - - - 23548 00:02:55
lolaage 5.3 19 - futex_ - - 23549 00:03:19
lolaage 5.3 19 - futex_ - - 23550 00:03:20
lolaage 4.5 19 - futex_ - - 23551 00:02:50
lolaage 4.5 19 - futex_ - - 23552 00:02:51
lolaage 0.0 19 - futex_ - - 23553 00:00:00
第三步：把线程ID转为16进制
[lolaage@video1 mpmt-socket]$ printf "%x " 23549
5bfd
第四步：查看堆栈信息
[lolaage@video1 mpmt-socket]$ jstack 23496 |grep 5bfd -A 30
"startQuertz_Worker-7" #25 prio=5 os_prio=0 tid=0x00007fb5fcdf8800 nid=0x5bfd in Object.wait() [0x00007fb5f6629000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
- locked <0x00000006c8d35208> (a java.lang.Object)
"startQuertz_Worker-6" #24 prio=5 os_prio=0 tid=0x00007fb5fcdf6800 nid=0x5bfc in Object.wait() [0x00007fb5f672a000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
- locked <0x00000006c8d15290> (a java.lang.Object)
"startQuertz_Worker-5" #23 prio=5 os_prio=0 tid=0x00007fb5fcdf4800 nid=0x5bfb in Object.wait() [0x00007fb5f682b000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
- locked <0x00000006c8d14fa8> (a java.lang.Object)
"startQuertz_Worker-4" #22 prio=5 os_prio=0 tid=0x00007fb5fcdf2800 nid=0x5bfa in Object.wait() [0x00007fb5f692c000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
- locked <0x00000006c8d14cc0> (a java.lang.Object)
经检查发现相关的程序代码中使用了locked导致的