zoukankan      html  css  js  c++  java
  • Hadoop 读取文件API报错

    Exception in thread "main" org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1124468226-10.0.2.15-1429879726015:blk_1073742186_1370 file=/user/testdir/yarn-site.xml
        at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:889)
        at org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:998)
        at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1333)
        at org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:78)
        at org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:116)
        at com.hdfs.file.ReadFile.main(ReadFile.java:24)、

    代码如下,在windows机器上运行,报上面的错误:

    package com.hdfs.file;
    
    import org.apache.hadoop.conf.Configuration;
    import org.apache.hadoop.fs.FSDataInputStream;
    import org.apache.hadoop.fs.FileStatus;
    import org.apache.hadoop.fs.FileSystem;
    import org.apache.hadoop.fs.Path;
    
    public class ReadFile {
    
        /**
         * @param args
         * @throws Exception 
         */
        public static void main(String[] args) throws Exception {
            // TODO Auto-generated method stub
            Configuration conf=new Configuration();
            FileSystem fs=FileSystem.get(conf);
            Path path=new Path("hdfs://sandbox.hortonworks.com:8020/user/testdir/yarn-site.xml");
            if(fs.exists(path)){
                FSDataInputStream fsIn=fs.open(path);
                FileStatus status=fs.getFileStatus(path);
                byte[] buffer=new byte[Integer.parseInt(String.valueOf(status.getLen()))];
                fsIn.readFully(0,buffer);
                fsIn.close();
                fs.close();
                System.out.println("读取完成!");
                System.out.println(new String(buffer));
            }else{
                throw new Exception("the file is not found!");
            }
        }
    
    }

     

    将代码打成Jar包,放到hadoop集群上去运行,一切正常:

    image

    image

    image

    image

  • 相关阅读:
    poj 2754 Similarity of necklaces 2 转换成多重背包,单调队列优化/ 二进制优化
    new和delete2
    new和delete1
    new和delete4
    new和delete3
    new(placement new)
    用例图中的Actor(参与者)一定是人吗?
    二维数组的函数参数传递
    二维指针动态创建二维数组(C/C++)
    OOD的五项基本原则——SOLID
  • 原文地址:https://www.cnblogs.com/justinzhang/p/4844341.html
Copyright © 2011-2022 走看看