zoukankan      html  css  js  c++  java
  • 马士兵hadoop2.7.3_使用java访问hdfs

    • 通过java程序访问hdfs:
      • hdfs系统会把用到的数据存储在core-site.xml中由hadoop.tmp.dir指定,而这个值默认位于/tmp/hadoop-${user.name}下面, 由于/tmp目录在系统重启时候会被删除,所以应该修改目录位置。 修改core-site.xml(在所有站点上都修改)
        1
        2
        3
        4
        5
        <property>
            <name>hadoop.tmp.dir</name>
            <value>/var/hadoop</value>
        </property>
      • hdfs namenode -format
      • windows上的权限系统和linux上的权限系统,测试期间为了简单起见可以关闭权限检查 在namenode的hdfs-site.xml上,添加配置:
        1
        2
        3
        4
        5
        <property>
            <name>dfs.permissions.enabled</name>
            <value>false</value>
        </property>
        重新启动namenode: hadoop-daemon.sh stop namenode, hadoop-daemon.sh start namenode
    • 从HDFS中读取文件
      URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory());
      InputStream in = new URL("hdfs://192.168.56.100:9000/test.data").openStream();
      IOUtils.copyBytes(in, System.out, 4096, true);
                  
    • 学习使用FileSystem类
        Configuration conf = new Configuration();
                      conf.set("fs.defaultFS", "hdfs://192.168.56.100:9000");
                      FileSystem fileSystem = FileSystem.get(conf);
                      
                      boolean b = fileSystem.exists(new Path("/hello"));
                      System.out.println(b);
                      
                      boolean success = fileSystem.mkdirs(new Path("/mashibing"));
                      System.out.println(success);
                      
                      success = fileSystem.delete(new Path("/mashibing"), true);
                      System.out.println(success);
                      
                      FSDataOutputStream out = fileSystem.create(new Path("/test.data"), true);
                      FileInputStream fis = new FileInputStream("c:/test/core-site.xml");
                      IOUtils.copyBytes(fis, out, 4096, true);
                      
                      FileStatus[] statuses = fileSystem.listStatus(new Path("/"));
                      //System.out.println(statuses.length);
                      for(FileStatus status : statuses) {
                          System.out.println(status.getPath());
                          System.out.println(status.getPermission());
                          System.out.println(status.getReplication());
                      }
    • 思考百度网盘的实现方式?
    • core-site做什么用?hdfs-site做什么用?
  • 相关阅读:
    unity, 显示像素图,以及iOS下像素图变模糊解决办法
    unity, iOS集成微信
    unity, PlayerPrefs.GetInt(key,defaultValue)
    unity, 对于Debug.Log输出的log,可以双击定位到代码
    unity, UGUI Image shader
    unity, use particleSystem with UGUI
    unity, UGUI Text fadeIn
    unity, write/read txt file
    unity, get Canvas Scaler referenceResolution
    unity, change parent and keep localPosition or worlPosition
  • 原文地址:https://www.cnblogs.com/Jxiaobai/p/6669011.html
Copyright © 2011-2022 走看看