100 CPU Diagnosis(转) 规格严格

On the Java Posse mailing list, someone posted a question about a JVM that was eating CPU like candy from a baby.

Here’s the problem from the mailing list:

I found a java process whose cpu is almost 100% at my notebook.

Here is the top output:

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM TIME+
6150 root      20   0 1411m 612m  12m R 98.8 15.7  45:49.12

and the jstack output for thread 6150(0×1806):

“VM Thread” prio=10 tid=0x0000000040ca9000 nid=0×1806 runnable

the last is jstat -gc output:

S0C    S1C    S0U    S1U      EC       EU        OC         OU       PC     PU    YGC     YGCT    FGC    FGCT     GCT
64.0   64.0   0.0    0.0   332992.0   0.0     666304.0   73192.5   83968.0 83967.9   6893   17.576 6882  2705.923 2723.499

Before you read further, take a look at this and see if you can tell what the problem is…

Now for the solution. Kirk Pepperdine said this:

YGC (Young gen count) is 6893 and FGCT is 6882 which is a completely ridiculous full GC to GC ratio. As for the times, YGCT is 17.56 where as FGCT is 2705.923 giving a 99.9% Full GC to GC ratio. A normal FGC:GC ratio should be less than 5% and less than 1% is desirable.

Second observation
PC is perm gen configured @ 83968 and PU perm gen used at 83967.9
Diagnosis: Frequent Full GC’s due to Perm Gen being full.
Solution: Increase the size of perm gen using -XX:MaxPermSize.

Awesome work, Mr. Pepperdine. Kirk is a frequent speaker at conferences, and he does a lot of performance work. If I needed a performance guru, Kirk would be the first person I’d think of.