zoukankan      html  css  js  c++  java
  • hoop安装及测试(转载)

    cloudera 网址  http://www.cloudera.com/blog/2011/07/hoop-hadoop-hdfs-over-http/

    hoop说明文档  http://cloudera.github.com/hoop/docs/latest/index.html

    hoop简介

    何为Hoop

    Hoop是对Hadoop HDFS Proxy 的改良重写,为HadoopHDFS提供了HTTP(S)的访问接口。使用Hoop,你可以:

    • 通过标准的HTTP协议访问你的HDFS系统
    • 在运行不同版本的HDFS之间进行数据交换(这克服了一些RPC方式因版本不同而产生的兼容性问题)
    • 将对HDFS的操作置于防火墙的保护下。Hoop Server在系统中可以充当网关的角色,并且只允许本系统穿越。

    组件

    Hoop由两部分组成:Hoop Server 和 Hoop Client,他们分别作用是:

    • Hoop Server是一个提供REST HTTP协议的服务,它允许你通过HTTP协议进行所有HDFS支持的文件系统的操作。
    • Hoop Client是一个HDFS的客户端实现,使用它你可以使用通常的HDFS的操作方法来通过Hoop操作HDFS。

    例子

    下面是几个使用传统的CURL工具通过Hoop操作HDFS的例子:

    1.获取home目录

    $ curl -i "http://hoopbar:14000?op=homedir&user.name=babu" HTTP/1.1 200 OK Content-Type: application/json Transfer-Encoding: chunked {"homeDir":"http:\/\/hoopbar:14000\/user\/babu"} $

    2.读取一个文件内容

    $ curl -i "http://hoopbar:14000?/user/babu/hello.txt&user.name=babu" HTTP/1.1 200 OK Content-Type: application/octet-stream Transfer-Encoding: chunked Hello World! $

    3.写文件

    $ curl -i -X POST "http://hoopbar:14000/user/babu/data.txt?op=create" --data-binary @mydata.txt --header "content-type: application/octet-stream" HTTP/1.1 200 OK Location: http://hoopbar:14000/user/babu/data.txt Content-Type: application/json Content-Length: 0 $

    4.列出目录的内容

    $ curl -i "http://hoopbar:14000?/user/babu?op=list&user.name=babu" HTTP/1.1 200 OK Content-Type: application/json Transfer-Encoding: chunked [ { "path" : "http:\/\/hoopbar:14000\/user\/babu\/data.txt" "isDir" : false, "len" : 966, "owner" : "babu", "group" : "supergroup", "permission" : "-rw-r--r--", "accessTime" : 1310671662423, "modificationTime" : 1310671662423, "blockSize" : 67108864, "replication" : 3 } ] $

    更多操作可以看这里:Hoop HTTP REST API  http://cloudera.github.com/hoop/docs/latest/HttpRestApi.html

    获取Hoop

    Hoop使用的是Apache License 2.0 发布,你可以在github上获取到它的源码(http://github.com/cloudera/hoop)这里(http://cloudera.github.com/hoop.)还有各种相关的安装使用教程http://cloudera.github.com/hoop/docs/latest/ServerSetup.html  。

    二、 安装

    http://github.com/cloudera/hoop可以git方式或 zip方式下载,下载最新版本cloudera-hoop-11bb221.zip

    解压

    三 安装maven

    下载及安装见http://maven.apache.org/download.html

    解压到/usr/local下

    设置 export M2_HOME=/usr/local/apache-maven-3.0.3

     export M2=$M2_HOME/bin

    export MAVEN_OPTS="-Xms256m -Xmx512m"  设置jvm参数

     export PATH=$M2:$PATH  加入到path


    export JAVA_HOME=/usr/java/jdk1.6.0_24/

    且确认将$JAVA_HOME/bin加入了path。

    执行mvn -version 确认是否安装成功

    四 安装hoop

    mvn clean package -Dmaven.test.skip=true site assembly:single


    tar xzf hoop/hoop-distro/target/hoop-0.1.0-SNAPSHOT.tar.gz
    
    

    Configure Hadoop

    Edit Hadoop core-site.xml and defined the Unix user that will run the Hoop server as a proxyuser. For example:

    ... <property> <name>hadoop.proxyuser.#HOOPUSER#.hosts</name> <value>myhoop.foo.com</value> </property> <property> <name>hadoop.proxyuser.#HOOPUSER#.groups</name> <value>*</value> </property> ...

    IMPORTANT: Replace #HOOPUSER# with the Unix user that will start the Hoop server.

    Restart Hadoop

    You need to restart Hadoop for the proxyuser configuration ot become active.

    Start/Stop Hoop

    To start/stop Hoop use Hoop's bin/hoop.sh script. For example:

    hoop-0.1.0-SNAPSHOT $ bin/hoop.sh start

    NOTE: Invoking the script without any parameters list all possible parameters (start, stop, run, etc.). The hoop.sh script is a wrapper for Tomcat's catalina.sh script that sets the environment variables and Java System properties required to run Hoop.

    Test Hoop is working

    ~ $ curl -i "http://<HOOPHOSTNAME>:14000?user.name=babu&op=homedir" HTTP/1.1 200 OK Content-Type: application/json Transfer-Encoding: chunked {"homeDir":"http:\/\/<HOOP_HOST>:14000\/user\/babu"}

    Embedded Tomcat Configuration

    To configure the embedded Tomcat go to the tomcat/conf.

    Hoop preconfigures the HTTP and Admin ports in Tomcat's server.xml to 14000 and 14001.

    Tomcat logs are also preconfigured to go to Hoop's logs/ directory.

    The following environment variables (which can be set in Hoop's conf/hoop-env.sh script) can be used to alter those values:

    • HOOP_HTTP_PORT
    • HOOP_ADMIN_PORT
    • HOOP_LOG

    Configure Hoop

    Edit the hoop-0.1.0-SNAPSHOT/conf/hoop-site.xml file and set the hoop.hadoop.conf:fs.default.name property to the HDFS Namenode URI. For example:

    hoop.hadoop.conf:fs.default.name=hdfs://localhost:8021

     原地址连接:http://blog.csdn.net/wf1982/article/details/6931843

  • 相关阅读:
    Java 签名(SHA1WithRSA、SHA256WithRSA、SHA256withECDSA)
    记一次用WireShark抓包摆脱Si服后台限制的过程
    反向代理Nginx
    对于新能源Can数据、电池BMS等字节和比特位的解析
    WireShark抓包命令
    .NET下寄宿于控制台的HTTPS监听
    RSA等非对称加密为什么要用公钥加密,而用私钥解密?
    FTP、SFTP与FTPS
    Python打包方法——Pyinstaller
    [Python爬虫]煎蛋网OOXX妹子图爬虫(1)——解密图片地址
  • 原文地址:https://www.cnblogs.com/qoyi/p/2321358.html
Copyright © 2011-2022 走看看