hoop安装及测试(转载)

cloudera 网址  http://www.cloudera.com/blog/2011/07/hoop-hadoop-hdfs-over-http/

hoop说明文档  http://cloudera.github.com/hoop/docs/latest/index.html

hoop简介

何为Hoop？

Hoop是对Hadoop HDFS Proxy 的改良重写，为HadoopHDFS提供了HTTP(S)的访问接口。使用Hoop，你可以：

• 通过标准的HTTP协议访问你的HDFS系统
• 在运行不同版本的HDFS之间进行数据交换（这克服了一些RPC方式因版本不同而产生的兼容性问题）
• 将对HDFS的操作置于防火墙的保护下。Hoop Server在系统中可以充当网关的角色，并且只允许本系统穿越。

组件

Hoop由两部分组成：Hoop Server 和 Hoop Client，他们分别作用是：

• Hoop Server是一个提供REST HTTP协议的服务，它允许你通过HTTP协议进行所有HDFS支持的文件系统的操作。
• Hoop Client是一个HDFS的客户端实现，使用它你可以使用通常的HDFS的操作方法来通过Hoop操作HDFS。

例子

下面是几个使用传统的CURL工具通过Hoop操作HDFS的例子：

1.获取home目录

$ curl -i "http://hoopbar:14000?op=homedir&user.name=babu" HTTP/1.1 200 OK Content-Type: application/json Transfer-Encoding: chunked {"homeDir":"http:\/\/hoopbar:14000\/user\/babu"} $

2.读取一个文件内容

$ curl -i "http://hoopbar:14000?/user/babu/hello.txt&user.name=babu" HTTP/1.1 200 OK Content-Type: application/octet-stream Transfer-Encoding: chunked Hello World! $

3.写文件

$ curl -i -X POST "http://hoopbar:14000/user/babu/data.txt?op=create" --data-binary @mydata.txt --header "content-type: application/octet-stream" HTTP/1.1 200 OK Location: http://hoopbar:14000/user/babu/data.txt Content-Type: application/json Content-Length: 0 $

4.列出目录的内容

$ curl -i "http://hoopbar:14000?/user/babu?op=list&user.name=babu" HTTP/1.1 200 OK Content-Type: application/json Transfer-Encoding: chunked [ { "path" : "http:\/\/hoopbar:14000\/user\/babu\/data.txt" "isDir" : false, "len" : 966, "owner" : "babu", "group" : "supergroup", "permission" : "-rw-r--r--", "accessTime" : 1310671662423, "modificationTime" : 1310671662423, "blockSize" : 67108864, "replication" : 3 } ] $

更多操作可以看这里：Hoop HTTP REST API  http://cloudera.github.com/hoop/docs/latest/HttpRestApi.html

获取Hoop

Hoop使用的是Apache License 2.0 发布，你可以在github上获取到它的源码（http://github.com/cloudera/hoop）这里（http://cloudera.github.com/hoop.）还有各种相关的安装使用教程http://cloudera.github.com/hoop/docs/latest/ServerSetup.html  。

二、 安装

http://github.com/cloudera/hoop可以git方式或 zip方式下载，下载最新版本cloudera-hoop-11bb221.zip

解压

三 安装maven

下载及安装见http://maven.apache.org/download.html

解压到/usr/local下

设置 export M2_HOME=/usr/local/apache-maven-3.0.3

 export M2=$M2_HOME/bin

export MAVEN_OPTS="-Xms256m -Xmx512m"  设置jvm参数

 export PATH=$M2:$PATH  加入到path

export JAVA_HOME=/usr/java/jdk1.6.0_24/

且确认将$JAVA_HOME/bin加入了path。

执行mvn -version 确认是否安装成功

四 安装hoop

mvn clean package -Dmaven.test.skip=true site assembly:single

tar xzf hoop/hoop-distro/target/hoop-0.1.0-SNAPSHOT.tar.gz

Configure Hadoop

Edit Hadoop core-site.xml and defined the Unix user that will run the Hoop server as a proxyuser. For example:

... <property> <name>hadoop.proxyuser.#HOOPUSER#.hosts</name> <value>myhoop.foo.com</value> </property> <property> <name>hadoop.proxyuser.#HOOPUSER#.groups</name> <value>*</value> </property> ...

IMPORTANT: Replace #HOOPUSER# with the Unix user that will start the Hoop server.

Restart Hadoop

You need to restart Hadoop for the proxyuser configuration ot become active.

Start/Stop Hoop

To start/stop Hoop use Hoop's bin/hoop.sh script. For example:

hoop-0.1.0-SNAPSHOT $ bin/hoop.sh start

NOTE: Invoking the script without any parameters list all possible parameters (start, stop, run, etc.). The hoop.sh script is a wrapper for Tomcat's catalina.sh script that sets the environment variables and Java System properties required to run Hoop.

Test Hoop is working

~ $ curl -i "http://<HOOPHOSTNAME>:14000?user.name=babu&op=homedir" HTTP/1.1 200 OK Content-Type: application/json Transfer-Encoding: chunked {"homeDir":"http:\/\/<HOOP_HOST>:14000\/user\/babu"}

Embedded Tomcat Configuration

To configure the embedded Tomcat go to the tomcat/conf.

Hoop preconfigures the HTTP and Admin ports in Tomcat's server.xml to 14000 and 14001.

Tomcat logs are also preconfigured to go to Hoop's logs/ directory.

The following environment variables (which can be set in Hoop's conf/hoop-env.sh script) can be used to alter those values:

• HOOP_HTTP_PORT
• HOOP_ADMIN_PORT
• HOOP_LOG

Configure Hoop

Edit the hoop-0.1.0-SNAPSHOT/conf/hoop-site.xml file and set the hoop.hadoop.conf:fs.default.name property to the HDFS Namenode URI. For example:

hoop.hadoop.conf:fs.default.name=hdfs://localhost:8021

 原地址连接：http://blog.csdn.net/wf1982/article/details/6931843