zoukankan      html  css  js  c++  java
  • INFO hdfs.DFSClient: Exception in createBlockOutputStream java.net解决办法

    自己安装好Hadoop2.7.x之后,发现dfs中的/bin/hadoop fs -put命令不能够使用,报错如下:

    [hadoop@master bin]$ ./hadoop fs -put ../logs/123 /wc/mytemp
    16/07/27 01:29:26 INFO hdfs.DFSClient: Exception in createBlockOutputStream
    java.net.NoRouteToHostException: No route to host
            at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
            at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
            at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
            at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
            at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1537)
            at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1313)
            at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1266)
            at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449)
    16/07/27 01:29:26 INFO hdfs.DFSClient: Abandoning BP-555863411-172.16.95.100-1469590594354:blk_1073741825_1001
    16/07/27 01:29:26 INFO hdfs.DFSClient: Excluding datanode DatanodeInfoWithStorage[172.16.95.101:50010,DS-ee00e1f8-5143-4f06-9ef8-b0f862fce649,DISK]
    16/07/27 01:29:26 INFO hdfs.DFSClient: Exception in createBlockOutputStream
    java.net.NoRouteToHostException: No route to host
            at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
            at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
            at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
            at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
            at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1537)
            at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1313)
            at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1266)
            at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449)
    16/07/27 01:29:26 INFO hdfs.DFSClient: Abandoning BP-555863411-172.16.95.100-1469590594354:blk_1073741826_1002
    16/07/27 01:29:26 INFO hdfs.DFSClient: Excluding datanode DatanodeInfoWithStorage[172.16.95.102:50010,DS-eea51eda-0a07-4583-9eee-acd7fc645859,DISK]
    16/07/27 01:29:26 WARN hdfs.DFSClient: DataStreamer Exception
    org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /wc/mytemp/123._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1).  There are 2 datanode(s) running and 2 node(s) are excluded in this operation.
            at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1547)
            at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3107)
            at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3031)
            at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:724)
            at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492)
            at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
            at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
            at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
            at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
            at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
            at java.security.AccessController.doPrivileged(Native Method)
            at javax.security.auth.Subject.doAs(Subject.java:422)
            at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
            at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)

            at org.apache.hadoop.ipc.Client.call(Client.java:1475)
            at org.apache.hadoop.ipc.Client.call(Client.java:1412)
            at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
            at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
            at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:418)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:497)
            at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
            at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
            at com.sun.proxy.$Proxy10.addBlock(Unknown Source)
            at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1459)
            at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1255)
            at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449)
    put: File /wc/mytemp/123._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1).  There are 2 datanode(s) running and 2 node(s) are excluded in this operation.
    [hadoop@master bin]$ service firewall
    The service command supports only basic LSB actions (start, stop, restart, try-restart, reload, force-reload, status). For other actions, please try to use systemctl.

    到网上找资料,好不统一找到几个解决的办法,感觉值的参考的是该文:hadoop常见错误解决办法 ,在其第14条,他给出的解决办法如下:

    I have resolved the issue:
    What i did:
    1) '/etc/init.d/iptables stop' -->stopped firewall
    2) SELINUX=disabled in '/etc/selinux/config' file.-->disabled selinux
    I worked for me after these two changes

    需要补充的地方:

      (1)此处CentOS7中,防火墙的方式改变,关闭防火墙改成service firewalld stop

      (2)不仅是master中的firewall要关闭,所有的从节点(slave)中的防火墙也要关闭。

      (3)检查所有的主机,/etc/selinux/config下的SELINUX,设置SELINUX=disable。

      (4)建议关闭防火墙之后,再使用systemctl disable firewalld.service #禁止firewall开机启动

    然后在进行put的时候,就成功了。截图如下:

      

    另外一个人博主给出的解决方法如下:hadoop报错:hdfs.DFSClient: Exception in createBlockOutputStream, 大家可以尝试一下,可以这样说,新手一般都会挂掉,因为格式化namenode的时候,slave中的文件会和namenode不一致,造成DataNode节点启动失败,解决办法就是删除slave和master中所有的data节点,然后重新执行namenode格式化操作。(不只是自己水平有限还是问题不一样,这种方法初步判定为坑人)

  • 相关阅读:
    为什么DIY报价----走出软件作坊:三五个人十来条枪 如何成为开发正规军(十二)[转]
    物以类聚,人以群分--走出软件作坊:三五个人十来条枪 如何成为开发正规军(十一)[转]
    将服务费用DIY到底----走出软件作坊:三五个人十来条枪 如何成为开发正规军(十)[转]
    实施费用也能DIY--走出软件作坊:三五个人十来条枪 如何成为开发正规军(九)[转]
    去掉iphone手机滑动默认行为
    获取json对象长度的问题
    手机淘宝用JS来动态写meta标签(1像素边框处理方法)
    移动端多行文字截断
    移动端初始化页面
    JavaScript 高级程序设计 02-变量、数据类型
  • 原文地址:https://www.cnblogs.com/xiangyangzhu/p/5710835.html
Copyright © 2011-2022 走看看