zoukankan      html  css  js  c++  java
  • Caused by: java.io.IOException: Filesystem closed的处理

    org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename output from: hdfs://nameservice/user/hive/warehouse/om_dw.db/mac_wifi_day_data/tid=CYJOY/.hive-staging_hive_2016-01-20_10-19-09_200_1283758166994658237-1/_task_tmp.-ext-10002/c_date=2014-10-06/_tmp.000151_0 to: hdfs://nameservice/user/hive/warehouse/om_dw.db/mac_wifi_day_data/tid=CYJOY/.hive-staging_hive_2016-01-20_10-19-09_200_1283758166994658237-1/_tmp.-ext-10002/c_date=2014-10-06/000151_0
    at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:242)
    at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$200(FileSinkOperator.java:143)
    at org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1051)
    at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:617)
    at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:631)
    at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:631)
    at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:631)
    at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:192)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
    Caused by: java.io.IOException: Filesystem closed
    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808)
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2113)
    at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
    at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
    at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:218)
    ... 15 more


    原因:
    多个datanode在getFileSystem过程中,由于Configuration一样,会得到同一个FileSystem。如果有一个datanode在使用完关闭连接,其它的datanode在访问就会出现上述异常


    解决办法:
    在hdfs core-site.xml里把fs.hdfs.impl.disable.cache设置为true

  • 相关阅读:
    复杂json后端解析出现第二层无数据的问题
    idea启动springboot项目报Error running 'ServiceStarter': Command line is too long. Shorten command line for ServiceStarter or also for Application
    docker强制关闭命令
    scala下实现actor多线程基础
    orcale数据库分配用户
    多线程实现互相通信
    从一份配置清单详解 Nginx 服务器配置
    PostgreSQL CentOS 7 安装配置
    .net core session部分浏览器或移动客户端不可用
    VS2019 远程调试
  • 原文地址:https://www.cnblogs.com/dailidong/p/7571137.html
Copyright © 2011-2022 走看看