zoukankan      html  css  js  c++  java

  • JVM最多能创建多少个线程: unable to create new native thread

    转载自：http://www.rigongyizu.com/jvm-max-threads/

    有应用报出这样的异常“java.lang.OutOfMemoryError: unable to create new native thread”。甚至机器上执行shell命令也会报”-bash: fork: Resource temporarily unavailable”异常。机器上的其他应用如hadoop也会受影响：

    1 2013-08-21 20:15:48,496 FATAL org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread
    2 java.lang.OutOfMemoryError: unable to create new native thread
    3         at java.lang.Thread.start0(Native Method)
    4         at java.lang.Thread.start(Thread.java:640)
    5         at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.
    6 ResourceLocalizationService$LocalizerTracker.handle(ResourceLocalizationService.java:524)
    7         at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.
    8 ResourceLocalizationService$LocalizerTracker.handle(ResourceLocalizationService.java:456)
    9         at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:128)
    10         at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:77)
    11         at java.lang.Thread.run(Thread.java:662)
    12 2013-08-21 20:15:48,497 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Exiting, bbye..

    一看以为内存不够导致无法创建新的线程，但是观察机器上的内存还有空闲，猜测是哪个地方对线程创建有限制。

    首先需要排除操作系统对线程创建数的限制，参考：《JVM中可生成的最大Thread数量》一文，设置操作系统可以支持创建10万个线程：

    1 echo "100000" > /proc/sys/kernel/threads-max
    2 echo "100000" > /proc/sys/kernel/pid_max     （默认32768）
    3 echo "200000" > /proc/sys/vm/max_map_count   （默认65530）
    4 ulimit -u unlimited   (设置max user processes的值)

    当前测试环境为：

    1 [admin@bufer108081.tbc ~]$ uname -a
    2 Linux bufer108081.tbc 2.6.32-220.23.2.ali927.el5.x86_64 #1 SMP Mon Jan 28 14:57:06 CST 2013 x86_64 x86_64 x86_64 GNU/Linux
    3 [admin@bufer108081.tbc ~]$ cat /etc/redhat-release
    4 Red Hat Enterprise Linux Server release 5.7 (Tikanga)
    5 [admin@bufer108081.tbc ~]$ java -version
    6 java version "1.7.0_51"
    7 Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
    8 OpenJDK (Alibaba) 64-Bit Server VM (build 24.45-b08-internal, mixed mode)
    9 [admin@bufer108081.tbc ~]$ ulimit -a
    10 core file size          (blocks, -c) 0
    11 data seg size           (kbytes, -d) unlimited
    12 scheduling priority             (-e) 0
    13 file size               (blocks, -f) unlimited
    14 pending signals                 (-i) 387068
    15 max locked memory       (kbytes, -l) 64
    16 max memory size         (kbytes, -m) unlimited
    17 open files                      (-n) 131072
    18 pipe size            (512 bytes, -p) 8
    19 POSIX message queues     (bytes, -q) 819200
    20 real-time priority              (-r) 0
    21 stack size              (kbytes, -s) 10240
    22 cpu time               (seconds, -t) unlimited
    23 max user processes              (-u) unlimited
    24 virtual memory          (kbytes, -v) unlimited
    25 file locks                      (-x) unlimited
    26 [admin@bufer108081.tbc ~/dev/baoniu]$ free -g
    27              total       used       free     shared    buffers     cached
    28 Mem:            47         31         15          0          3         25
    29 -/+ buffers/cache:          3         44
    30 Swap:            0          0          0

    测试程序见本文最后面。测试结果：突破了网上所说的32000个线程数，成功创建了 10万个线程。 （由于/proc/sys/kernel/pid_max默认为32768，所以网上很多测试程序测试JVM只能创建32000个线程。）

    1 [admin@bufer108081.tbc ~/dev/baoniu]$ java -Xss128k MaxThreadsMain
    2 The stack size specified is too small, Specify at least 228k
    3 Error: Could not create the Java Virtual Machine.
    4 Error: A fatal exception has occurred. Program will exit.
    5 [admin@bufer108081.tbc ~/dev/baoniu]$ java -Xss228k MaxThreadsMain
    6 4,000 threads: Time to create 4,000 threads was 0.846 seconds
    7 8,000 threads: Time to create 4,000 threads was 2.425 seconds
    8 12,000 threads: Time to create 4,000 threads was 4.813 seconds
    9 16,000 threads: Time to create 4,000 threads was 7.229 seconds
    10 20,000 threads: Time to create 4,000 threads was 10.443 seconds
    11 24,000 threads: Time to create 4,000 threads was 14.480 seconds
    12 28,000 threads: Time to create 4,000 threads was 19.709 seconds
    13 32,000 threads: Time to create 4,000 threads was 24.742 seconds
    14 36,000 threads: Time to create 4,000 threads was 31.181 seconds
    15 40,000 threads: Time to create 4,000 threads was 36.629 seconds
    16 44,000 threads: Time to create 4,000 threads was 42.796 seconds
    17 48,000 threads: Time to create 4,000 threads was 48.659 seconds
    18 52,000 threads: Time to create 4,000 threads was 55.030 seconds
    19 56,000 threads: Time to create 4,000 threads was 60.130 seconds
    20 60,000 threads: Time to create 4,000 threads was 67.419 seconds
    21 64,000 threads: Time to create 4,000 threads was 73.507 seconds
    22 68,000 threads: Time to create 4,000 threads was 79.416 seconds
    23 72,000 threads: Time to create 4,000 threads was 85.261 seconds
    24 76,000 threads: Time to create 4,000 threads was 92.201 seconds
    25 80,000 threads: Time to create 4,000 threads was 98.087 seconds
    26 84,000 threads: Time to create 4,000 threads was 108.263 seconds
    27 88,000 threads: Time to create 4,000 threads was 114.840 seconds
    28 92,000 threads: Time to create 4,000 threads was 121.841 seconds
    29 96,000 threads: Time to create 4,000 threads was 127.714 seconds
    30 After creating 99,410 threads, java.lang.OutOfMemoryError: unable to create new native thread
    31         at java.lang.Thread.start0(Native Method)
    32         at java.lang.Thread.start(Thread.java:713)
    33         at MaxThreadsMain.addThread(MaxThreadsMain.java:43)
    34         at MaxThreadsMain.main(MaxThreadsMain.java:13)

    创建9W多个线程后，进程占用内存：VIRT=40.5g RES=4.7g，用free -g查看系统还有9G的空闲(free)内存。

    JVM最多能启动的线程数参照公式：

     

    1 (MaxProcessMemory  - JVMMemory – ReservedOsMemory) / (ThreadStackSize) = Number of threads
    • MaxProcessMemory : 进程的最大寻址空间
    • JVMMemory : JVM内存
    • ReservedOsMemory : 保留的操作系统内存，如Native heap，JNI之类，一般100多M
    • ThreadStackSize : 线程栈的大小，jvm启动时由Xss指定

    MaxProcessMemory：如32位的linux默认每个进程最多申请3G的地址空间，64位的操作系统可以支持到46位（64TB）的物理地址空间和47位（128T）的进程虚拟地址空间（linux 64位CPU内存限制）。

    JVM内存：由Heap区和Perm区组成。通过-Xms和-Xmx可以指定heap区大小，通过-XX:PermSize和-XX:MaxPermSize指定perm区的大小(默认从32MB 到64MB，和JVM版本有关)。

    线程栈ThreadStackSize：

    Java程序中，每个线程都有自己的Stack Space。这个Stack Space的空间是独立分配的，与-Xmx和-Xms指定的堆大小无关。Stack Space用来做方法的递归调用时压入Stack Frame。所以当递归调用太深的时候，就有可能耗尽Stack Space，爆出StackOverflow的错误。对于32位JVM，缺省值为256KB，对于64位JVM，缺省值为512KB。最大值根据平台和特定机器配置的不同而不同。如果超过最大值，那么将报告java/lang/OutOfMemoryError消息。

    可见，减少Xss指定的线程栈大小能够启动更多的线程，但是线程总数也受到系统空闲内存和操作系统的限制。

    总结下影响Java线程数量的因素：

    • Java虚拟机本身：-Xms，-Xmx，-Xss；
    • 系统限制： /proc/sys/kernel/pid_max， /proc/sys/kernel/thread-max， max_user_process（ulimit -u）， /proc/sys/vm/max_map_count。

    ps: 最后发现是这台机器上有个应用代码问题创建了过多的线程，达到系统限制，而影响了YARN和其他应用。一般来说，单机线程数过多可以考虑使用线程池或者更多的服务器。

    附测试程序：

    1 import java.util.ArrayList;
    2 import java.util.List;
    3  
    4 public class MaxThreadsMain {
    5  
    6   public static final int BATCH_SIZE = 4000;
    7  
    8   public static void main(String... args) throws InterruptedException {
    9     List<Thread> threads = new ArrayList<Thread>();
    10     try {
    11       for (int i = 0; i <= 100 * 1000; i += BATCH_SIZE) {
    12         long start = System.currentTimeMillis();
    13         addThread(threads, BATCH_SIZE);
    14         long end = System.currentTimeMillis();
    15         Thread.sleep(1000);
    16         long delay = end - start;
    17         System.out.printf("%,d threads: Time to create %,d threads was %.3f seconds %n", threads.size(), BATCH_SIZE, delay / 1e3);
    18       }
    19     } catch (Throwable e) {
    20       System.err.printf("After creating %,d threads, ", threads.size());
    21       e.printStackTrace();
    22     }
    23  
    24   }
    25  
    26   private static void addThread(List<Thread> threads, int num) {
    27     for (int i = 0; i < num; i++) {
    28       Thread t = new Thread(new Runnable() {
    29         @Override
    30         public void run() {
    31           try {
    32             while (!Thread.interrupted()) {
    33               Thread.sleep(1000);
    34             }
    35           } catch (InterruptedException ignored) {
    36             //
    37           }
    38         }
    39       });
    40       t.setDaemon(true);
    41       t.setPriority(Thread.MIN_PRIORITY);
    42       threads.add(t);
    43       t.start();
    44     }
    45   }
    46 }

     

    附：

    [html] view plain copy print?
    1. MySQL Performance: Hitting Error "Can't Create A New Thread (Errno 11)" On A High Number Of Connections:  
    2. Http://Dimitrik.Free.Fr/Blog/Archives/2010/11/Mysql-Performance-Hitting-Error-Cant-Create-A-New-Thread-Errno-11-On-A-High-Number-Of-Connections.Html  
    3.   
    4.   
    5.   $ Ulimit -U  
    6.   1024  
    7. It Explains 1000 Sessions Limitation ;-)  
    8.   
    9.   
    10. Adding Few Lines More To My "/Etc/Security/Limits.Conf" File:  
    11.   
    12.   
    13.   # Cat /Etc/Security/Limits.Conf  
    14.   Mysql     Soft    Nofile  10240  
    15.   Mysql     Hard    Nofile  40960  
    16.   Mysql     Soft    Nproc   10240  
    17.   Mysql     Hard    Nproc   40960  
    18. Fixed My Issue! :-))  
    MySQL Performance: Hitting Error "Can't Create A New Thread (Errno 11)" On A High Number Of Connections:
Http://Dimitrik.Free.Fr/Blog/Archives/2010/11/Mysql-Performance-Hitting-Error-Cant-Create-A-New-Thread-Errno-11-On-A-High-Number-Of-Connections.Html


  $ Ulimit -U
  1024
It Explains 1000 Sessions Limitation ;-)


Adding Few Lines More To My "/Etc/Security/Limits.Conf" File:


  # Cat /Etc/Security/Limits.Conf
  Mysql 	Soft 	Nofile 	10240
  Mysql 	Hard    Nofile 	40960
  Mysql 	Soft 	Nproc 	10240
  Mysql 	Hard    Nproc 	40960
Fixed My Issue! :-))
  • 相关阅读:
    网站安全编程 黑客入侵 脚本黑客 高级语法入侵 C/C++ C# PHP JSP 编程
    【算法导论】贪心算法，递归算法，动态规划算法总结
    cocoa2dx tiled map添加tile翻转功能
    8月30日上海ORACLE大会演讲PPT下载
    【算法导论】双调欧几里得旅行商问题
    Codeforces Round #501 (Div. 3) B. Obtaining the String (思维,字符串)
    Codeforces Round #498 (Div. 3) D. Two Strings Swaps (思维)
    Educational Codeforces Round 89 (Rated for Div. 2) B. Shuffle (数学,区间)
    洛谷 P1379 八数码难题 (BFS)
    Educational Codeforces Round 89 (Rated for Div. 2) A. Shovels and Swords (贪心)
  • 原文地址：https://www.cnblogs.com/firstdream/p/7802290.html
Copyright © 2011-2022 走看看