zoukankan      html  css  js  c++  java
  • maven项目测试HDFS读取文件

    1、读取文件

       /**
         * 测试读取文件
         * @throws IOException
         */
        @Test
        public void testSave() throws IOException {
            Configuration conf = new Configuration();
            FileSystem fs = FileSystem.get(conf);
            FSDataInputStream fis = fs.open(new Path("hdfs://s150:8020/usr/xiaoqiu/hadoop/mydir1/hello1.txt"));
            ByteArrayOutputStream baos = new ByteArrayOutputStream();
            IOUtils.copyBytes(fis,baos,1024);
            fis.close();
            System.out.println(new String(baos.toByteArray()));
        }

    2、写入文件

        /**
         * 测试写入文件
         */
        @Test
        public void testWrite () throws IOException {
            Configuration conf = new Configuration();
            FileSystem fs = FileSystem.get(conf);
            FSDataOutputStream fos =  fs.create(new Path("hdfs://s150:8020/usr/xiaoqiu/hadoop/mydir1/hello1.txt"));
            fos.write("hello world".getBytes());
            fos.close();
        }

    3、定制副本数和blocksize

     /**
         * 定制副本数和blocksize
         */
        @Test
        public void testWrite2 () throws IOException {
            Configuration conf = new Configuration();
            FileSystem fs = FileSystem.get(conf);
            FSDataOutputStream fos = fs.create(new Path("hdfs://s150:8020/usr/xiaoqiu/hadoop/mydir1/hello1.txt"),
                    true,
                    1024,
                    (short)2,
                    1024);
            fos.write("how are you ".getBytes());
            fos.close();
    
        }

    修改最小区域块大小
    [xiaoqiu@s150 /soft/hadoop/etc/hadoop]$ cat hdfs-site.xml
    <?xml version="1.0" encoding="UTF-8"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
    
    <configuration>
    <property>
      <name>dfs.replication</name>
      <value>2</value>
    </property>
    <property>
       <name>dfs.namenode.fs-limits.min-block-size</name>
       <value>1024</value>
    </property>
    </configuration>
    修改了配置文件记得重启Hadoop



































    欢迎关注我的公众号:小秋的博客 CSDN博客:https://blog.csdn.net/xiaoqiu_cr github:https://github.com/crr121 联系邮箱:rongchen633@gmail.com 有什么问题可以给我留言噢~
  • 相关阅读:
    oracle中Blob和Clob类型的区别
    为什么要分库分表
    Enable file editing in Visual Studio's debug mode
    SQL Server Dead Lock Log
    Debug .NET Framework Source
    SQL Server text field里面有换行符的时候copy到excel数据会散乱
    诊断和修复Web测试记录器(Web Test Recorder)问题
    Can't load Microsoft.ReportViewer.ProcessingObjectModel.dll
    'telnet' is not recognized as an internal or external command
    Linq to XML
  • 原文地址:https://www.cnblogs.com/flyingcr/p/10327007.html
Copyright © 2011-2022 走看看