zoukankan      html  css  js  c++  java
  • Hadoop源码阅读-HDFS-day1

    HDFS声明及构造函数

     1 @InterfaceAudience.Private
     2 @InterfaceStability.Evolving
     3 public class Hdfs extends AbstractFileSystem {
     4 
     5   DFSClient dfs;
     6   final CryptoCodec factory;
     7   private boolean verifyChecksum = true;
     8 
     9   static {
    10     HdfsConfiguration.init();
    11   }
    12 
    13   /**
    14    * This constructor has the signature needed by
    15    * {@link AbstractFileSystem#createFileSystem(URI, Configuration)}
    16    * 
    17    * @param theUri which must be that of Hdfs
    18    * @param conf configuration
    19    * @throws IOException
    20    */
    21   Hdfs(final URI theUri, final Configuration conf) throws IOException, URISyntaxException {
    22     super(theUri, HdfsConstants.HDFS_URI_SCHEME, true, NameNode.DEFAULT_PORT);
    23 
    24     if (!theUri.getScheme().equalsIgnoreCase(HdfsConstants.HDFS_URI_SCHEME)) {
    25       throw new IllegalArgumentException("Passed URI's scheme is not for Hdfs");
    26     }
    27     String host = theUri.getHost();
    28     if (host == null) {
    29       throw new IOException("Incomplete HDFS URI, no host: " + theUri);
    30     }
    31 
    32     this.dfs = new DFSClient(theUri, conf, getStatistics());
    33     this.factory = CryptoCodec.getInstance(conf);
    34   }

    Hdfs继承了AbstractFileSystem这个抽象类,其中有一个静态块,执行HdfsConfiguration的初始化方法,我们先来看下这个方法

    /**
       * This method is here so that when invoked, HdfsConfiguration is class-loaded if
       * it hasn't already been previously loaded.  Upon loading the class, the static 
       * initializer block above will be executed to add the deprecated keys and to add
       * the default resources.   It is safe for this method to be called multiple times 
       * as the static initializer block will only get invoked once.
       * 
       * This replaces the previously, dangerous practice of other classes calling
       * Configuration.addDefaultResource("hdfs-default.xml") directly without loading 
       * HdfsConfiguration class first, thereby skipping the key deprecation
       */
      public static void init() {
      }

    init是一个空函数,它存在的作用,只是为了类加载,当类加载的时候,静态块中的方法将会执行从而增加过时的Key和Resource。因为它本身是一个空函数,它被反复调用时安全的,因为静态块中的方法只会被调用一次,使用这个方法来替代之前不安全的直接类调用Configuration.addDefaultResource("hdfs-default.xml")方法。所以,他的静态块中肯定会包含这个方法,我们来看下是不是这样.

    static {
        addDeprecatedKeys();
    
        // adds the default resources
        Configuration.addDefaultResource("hdfs-default.xml");
        Configuration.addDefaultResource("hdfs-site.xml");
    
      }

    看完这些,回到HDFS的父类AbstractFileSystem

    /**
     * This class provides an interface for implementors of a Hadoop file system
     * (analogous to the VFS of Unix). Applications do not access this class;
     * instead they access files across all file systems using {@link FileContext}.
     * 
     * Pathnames passed to AbstractFileSystem can be fully qualified URI that
     * matches the "this" file system (ie same scheme and authority) 
     * or a Slash-relative name that is assumed to be relative
     * to the root of the "this" file system .
     */
    @InterfaceAudience.Public
    @InterfaceStability.Evolving /*Evolving for a release,to be changed to Stable */
    public abstract class AbstractFileSystem {

    AbstractFileSystem提供了一个实现Hadoop 文件系统的接口,应用访问文件通过FileContext而不需要访问这个类,路径名称一旦匹配了这个文件系统,就会视为合法的URI,否则的会认为是该文件系统根目录的相对路径?(这个不太确定)

    来看下它的构造函数:

    /**
       * Constructor to be called by subclasses.
       * 
       * @param uri for this file system.
       * @param supportedScheme the scheme supported by the implementor
       * @param authorityNeeded if true then theURI must have authority, if false
       *          then the URI must have null authority.
       *
       * @throws URISyntaxException <code>uri</code> has syntax error
       */
      public AbstractFileSystem(final URI uri, final String supportedScheme,
          final boolean authorityNeeded, final int defaultPort)
          throws URISyntaxException {
        myUri = getUri(uri, supportedScheme, authorityNeeded, defaultPort);
        statistics = getStatistics(uri); 
      }
    /**
       * Get the URI for the file system based on the given URI. The path, query
       * part of the given URI is stripped out and default file system port is used
       * to form the URI.
       * 
       * @param uri FileSystem URI.
       * @param authorityNeeded if true authority cannot be null in the URI. If
       *          false authority must be null.
       * @param defaultPort default port to use if port is not specified in the URI.
       * 
       * @return URI of the file system
       * 
       * @throws URISyntaxException <code>uri</code> has syntax error
       */
      private URI getUri(URI uri, String supportedScheme,
          boolean authorityNeeded, int defaultPort) throws URISyntaxException {
        checkScheme(uri, supportedScheme);
        // A file system implementation that requires authority must always
        // specify default port
        if (defaultPort < 0 && authorityNeeded) {
          throw new HadoopIllegalArgumentException(
              "FileSystem implementation error -  default port " + defaultPort
                  + " is not valid");
        }
        String authority = uri.getAuthority();
        if (authority == null) {
           if (authorityNeeded) {
             throw new HadoopIllegalArgumentException("Uri without authority: " + uri);
           } else {
             return new URI(supportedScheme + ":///");
           }   
        }
        // authority is non null  - AuthorityNeeded may be true or false.
        int port = uri.getPort();
        port = (port == -1 ? defaultPort : port);
        if (port == -1) { // no port supplied and default port is not specified
          return new URI(supportedScheme, authority, "/", null);
        }
        return new URI(supportedScheme + "://" + uri.getHost() + ":" + port);
      }

    getUri主要是通过给定的URI生成FS的URI

  • 相关阅读:
    time,implicitly_wait,WebDriverWait三种等待方式
    iframe,window,alert切换
    pandas
    TestCase,Testsuit,TestLoder,TextTestRunner实现对测试用例全部执行或部分执行
    静态,类,实例,冒泡
    configparser读取
    ddt,data,unpack用法
    mybatis入门教程之搭建一个简单的mybatis项目并启动它
    修改hosts文件后不生效,该怎么办
    在JavaScript种遇到这样的错误如何解决XML 解析错误:格式不佳 位置:http:/... 行 27,列 32:
  • 原文地址:https://www.cnblogs.com/nashiyue/p/5327302.html
Copyright © 2011-2022 走看看