zoukankan      html  css  js  c++  java
  • 使用Maven将Hadoop2.2.0源码编译成Eclipse项目

    编译环境:

    OS:RHEL 6.3 x64

    Maven:3.2.1

    Eclipse:Juno SR2 Linux x64

    libprotoc:2.5.0

    JDK:1.7.0_51 x64

    步骤:

    1. 下载Hadoop2.2.0源码包  http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.2.0/hadoop-2.2.0-src.tar.gz

    2. 解压缩到Eclipse的workspace中。这里最好解压缩到workspace中,编译完成后直接导入就好了,不用移动,移动的话容易造成依赖关系的确实,我比较懒,所以就放在workspace中编译,省的还要build path。

    3. 安装Maven。Hadoop前期使用的是ant+ivy,后面改成了Maven,在源码包的BUILD文件中有写。下载Maven  http://apache.fayea.com/apache-mirror/maven/maven-3/3.2.1/binaries/apache-maven-3.2.1-bin.tar.gz,解压缩到安装目录,在.bashrc中添加以下内容

    export MAVEN_HOME=/root/Software/Maven321
    
    export PATH=/root/Software/Maven321/bin:$PATH
    在控制台中输入mvn -version,打印以下信息则安装成功
    Apache Maven 3.2.1 (ea8b2b07643dbb1b84b6d16e1f08391b666bc1e9; 2014-02-15T01:37:52+08:00)
    
    Maven home: /root/Software/maven321
    
    Java version: 1.7.0_51, vendor: Oracle Corporation
    
    Java home: /usr/java/jdk1.7.0_51/jre
    
    Default locale: zh_CN, platform encoding: UTF-8
    
    OS name: "linux", version: "2.6.32-431.5.1.el6.x86_64", arch: "amd64", family: "unix"

    4. 安装protobuf-2.5.0,首先安装一下gcc

    yum install gcc
    
    yum install gcc-c++
    然后下载protocbuf https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz 下载完成后解压到安装目录。进入安装目录执行如下命令进行安装
    ./configure
    
     make
    
     make check
    
     make install
    安装完成后在控制台输入protoc --version,有以下输出则安装成功。安装protoc的原因是要用到它啦,并且官方文档里貌似也没提到,就是编译的时候遇到错误了。并且这里是要编译Eclipse项目,如果要编译成可执行的Hadoop的话需要安装更多的软件进行支持,详情可以见这里 http://my.oschina.net/cloudcoder/blog/192224
    [root@dell ~]# protoc --version
    
    libprotoc 2.5.0
    
    
    

    5. 下面就可以开始编译了,进入到Hadoop2.2.0的源码包目录里,ls查看一下,可以看到hadoop-maven-plugins文件夹,先进入到这个文件家,执行mvn install。过程有点长,如果显示说有jar包下载不下来就多执行几次,总有下载下来的一天。显示BUILD SUCCESS后则返回到hadoop-2.2.0-src的根目录下,执行mvn eclipse:eclipse –DskipTests,生成Eclipse项目。同样,显示BUILD SUCCESS后就是编译成功了。

    [root@dell hadoop-2.2.0-src]# ls
    
    BUILDING.txt           hadoop-hdfs-project       hadoop-tools
    
    dev-support            hadoop-mapreduce-project  hadoop-yarn-project
    
    hadoop-assemblies      hadoop-maven-plugins      LICENSE.txt
    
    hadoop-client          hadoop-minicluster        NOTICE.txt
    
    hadoop-common-project  hadoop-project            pom.xml
    
    hadoop-dist            hadoop-project-dist       README.txt
    
    

    6. 将编译好的项目导入到Eclipse中,依次执行[File] > [Import] > [Existing Projects into Workspace]即可。由于生成了很多个项目,所以导入后是这个样子的。并且还会有一些错误,下面对如何修复错误写一下。

    image

    Error#1. hadoop-streaming里面的build path有问题,显示/root/workspace/hadoop-2.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/conf(missing)

    解决办法,remove掉引用就好。

    Error#2. hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientFailover.java中报sun.net.spi.nameservice.NameService错误,这是一个需要import的包,存在于openjdk中,在Oracle Jdk中没找到,需要下载一个。NameService是一个接口,在网上找一个NameService放到该包中就好。 http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/7u40-b43/sun/net/spi/nameservice/NameService.java#NameService

    Error#3. /hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineEditsViewer/XmlEditsVisitor.java里面显示
    import com.sun.org.apache.xml.internal.serialize.OutputFormat;
    import com.sun.org.apache.xml.internal.serialize.XMLSerializer;
    失败,这是由于Eclipse的强检查原则,打开Java -> Compiler -> Errors/Warnings and under "Deprecated and restricted API" change the setting of "Forbidden reference (access rules)" 将error级别调整到warning级别就好。

    Error#4. /hadoop-common/src/test/java/org/apache/hadoop/io/serializer/avro/TestAvroSerialization.java显示没有AvroRecord类,在网上搜索到AvroRecord类放入到同级包中就行了。  http://grepcode.com/file/repo1.maven.org/maven2/org.apache.hadoop/hadoop-common/2.2.0/org/apache/hadoop/io/serializer/avro/AvroRecord.java#AvroRecord

    Error#5. org.apache.hadoop.ipc.protobuf包是空的,需要在/hadoop-common/target/generated-sources/java中找到profobuf拷贝到/hadoop-common/src/test/java中就好了. 同时包里面还缺少了以下三个引用,在GrepCode上找一下,把hadoop-common2.2.0的相应文件下下来导入。

    org.apache.hadoop.ipc.protobuf.TestProtos.EchoRequestProto;
    org.apache.hadoop.ipc.protobuf.TestProtos.EchoResponseProto;
    org.apache.hadoop.ipc.protobuf.TestRpcServiceProtos.TestProtobufRpcProto;

    Error#6. /hadoop-auth/org/apache/hadoop/security/authentication/client/AuthenricatorTestCase.java中显示server.start()和server.stop()错误,还没找到原因所在,待检查~~~

  • 相关阅读:
    (转)【web前端培训之前后端的配合(中)】继续昨日的故事
    ural(Timus) 1136. Parliament
    scau Josephus Problem
    ACMICPC Live Archive 6204 Poker End Games
    uva 10391 Compound Words
    ACMICPC Live Archive 3222 Joke with Turtles
    uva 10132 File Fragmentation
    uva 270 Lining Up
    【转】各种字符串哈希函数比较
    uva 10905 Children's Game
  • 原文地址:https://www.cnblogs.com/zhengcong/p/3592490.html
Copyright © 2011-2022 走看看