zoukankan      html  css  js  c++  java
  • hadoophttpfs

    Hadoop-httpfs: client向httpfs提交文件操作,由httpfs和集群交互; 优势:client不必访问集群

    WebHDFS API: https://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-project-dist/hadoop-hdfs/WebHDFS.html

    coludera install: https://www.cloudera.com/documentation/enterprise/5-10-x/topics/cdh_ig_httpfs_install.html

    yum install hadoop-httpfs

    1. /etc/hadoop-httpfs/conf/httpfs-site.xml 

    <property>
      <name>httpfs.hadoop.config.dir</name>
      <value>/etc/hadoop/conf</value>
    </property>
    <property>
      <name>httpfs.authentication.type</name>
      <value>kerberos</value>
    </property>
    <property>
      <name>httpfs.hadoop.authentication.type</name>
      <value>kerberos</value>
    </property>
    <property>
      <name>httpfs.authentication.kerberos.principal</name>
      <value>HTTP/cdd03@PASC.COM</value>
    </property>
    <property>
      <name>httpfs.authentication.kerberos.keytab</name>
      <value>/etc/security/keytab/httpfs.keytab</value>
    </property>
    <property>
      <name>httpfs.hadoop.authentication.kerberos.principal</name>
      <value>httpfs/cdd03@PASC.COM</value>
    </property>
    <property>
      <name>httpfs.hadoop.authentication.kerberos.keytab</name>
      <value>/etc/security/keytab/httpfs.keytab</value>
    </property>

    2. /etc/hadoop/conf/core-site.xml    

    <property>
      <name>hadoop.proxyuser.httpfs.hosts</name>
      <value>*</value>
    </property>
    <property>
      <name>hadoop.proxyuser.httpfs.groups</name>
      <value>*</value>
    </property>

    3. 重启namenode,启动httpfs服务

    使用:curl  --negotiate -u :  http://cdd03:14000/webhdfs/v1/tmp/?op=liststatus

    红色地方必须使用:hostname(kerberos认证会用到)

  • 相关阅读:
    完美解决ListView中事件ItemCreated中使用ClientID导致插入数据失败
    cookie 和session 的区别详解
    ref与out之间的区别整理
    showModalDialog介绍
    meta元素
    [转] Spring Boot特性
    [转] Spring Boot 揭秘与实战(二) 数据存储篇
    Jsch初步
    如何通俗的解释交叉熵与相对熵
    pip安装时的异常,找不到lib2to3\Grammar.txt
  • 原文地址:https://www.cnblogs.com/moonypog/p/11051335.html
Copyright © 2011-2022 走看看