zoukankan      html  css  js  c++  java
  • hdfs OutOfMemoryError

    大量推送本地文件到hdfs如下

    hadoop fs -put ${local_path} ${hdfs_path}报错。

    Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.util.Arrays.copyOfRange(Arrays.java:3664)
        at java.lang.StringBuffer.toString(StringBuffer.java:669)
        at java.net.URI.toString(URI.java:1945)
        at java.net.URI.<init>(URI.java:742)
        at org.apache.hadoop.fs.Path.initialize(Path.java:202)
        at org.apache.hadoop.fs.Path.<init>(Path.java:196)
        at org.apache.hadoop.fs.Path.getPathWithoutSchemeAndAuthority(Path.java:80)
        at org.apache.hadoop.fs.shell.CommandWithDestination.checkPathsForReservedRaw(CommandWithDestination.java:355)
        at org.apache.hadoop.fs.shell.CommandWithDestination.copyFileToTarget(CommandWithDestination.java:322)
        at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:263)
        at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:248)
        at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317)
        at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:373)
        at org.apache.hadoop.fs.shell.CommandWithDestination.recursePath(CommandWithDestination.java:291)
        at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:319)
        at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:373)
        at org.apache.hadoop.fs.shell.CommandWithDestination.recursePath(CommandWithDestination.java:291)
        at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:319)
        at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:373)
        at org.apache.hadoop.fs.shell.CommandWithDestination.recursePath(CommandWithDestination.java:291)
        at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:319)
        at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:373)
        at org.apache.hadoop.fs.shell.CommandWithDestination.recursePath(CommandWithDestination.java:291)
        at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:319)
        at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:373)
        at org.apache.hadoop.fs.shell.CommandWithDestination.recursePath(CommandWithDestination.java:291)
        at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:319)
        at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:373)
        at org.apache.hadoop.fs.shell.CommandWithDestination.recursePath(CommandWithDestination.java:291)
        at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:319)
        at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:373)
        at org.apache.hadoop.fs.shell.CommandWithDestination.recursePath(CommandWithDestination.java:291)

    在hadoop命令下发现:

    hadoop-2.7.3/bin/hadoop:    exec "$JAVA" $JAVA_HEAP_MAX $HADOOP_OPTS $CLASS "$@"

    查找JAVA_HEAP_MAX

    发现

    hadoop-2.7.3/libexec/hadoop-config.sh:  JAVA_HEAP_MAX="-Xmx""$HADOOP_HEAPSIZE""m"

    继续查找HADOOP_HEAPSIZE

    发现

    hadoop-2.7.3/libexec/hadoop-config.sh:  JAVA_HEAP_MAX="-Xmx""$HADOOP_HEAPSIZE""m"

    以及

    hadoop-2.7.3/share/doc/hadoop/hadoop-project-dist/hadoop-common/ClusterSetup.html:<li><tt>HADOOP_HEAPSIZE</tt> / <tt>YARN_HEAPSIZE</tt> - The maximum amount of heapsize to use, in MB e.g. if the varibale is set to 1000 the heap will be set to 1000MB. This is used to configure the heap size for the daemon. By default, the value is 1000. If you want to configure the values separately for each deamon you can use.</li>
    <li>
        <tt>HADOOP_HEAPSIZE</tt>
        / <tt>YARN_HEAPSIZE</tt> - The maximum amount of heapsize to use, in MB e.g. if the varibale is set to 1000 the heap
        will be set to 1000MB. This is used to configure the heap size for the daemon. By default, the value is 1000. If you
        want to configure the values separately for each deamon you can use.
    </li>

    调整JVM堆的最大值:

    export HADOOP_HEAPSIZE=100000
  • 相关阅读:
    CentOS7 安装Kafka
    使用ZeroTier搭建大局域网利用VNC远程桌面
    IntelliJ常用配置备忘
    docker-compose部署kafka
    使用Mybatis执行sql脚本
    jQuery实现三级联动菜单(鼠标悬停联动)
    phonegap 使用极光推送实现消息推送
    如何用HTML5+PhoneGap写个Path项目
    如何脱离SDK,使用DW5.5和phonegap以及JQMobile搭建开发环境
    JSON和JSONP有哪些区别,PhoneGap跨域请求如何实现
  • 原文地址:https://www.cnblogs.com/suanec/p/9878612.html
Copyright © 2011-2022 走看看