zoukankan      html  css  js  c++  java
  • hadoop2.x源码编译

    转载请标明出处:
    http://blog.csdn.net/zwto1/article/details/50733753

    介绍

    本篇主要会涉及以下内容:

    学会编译hadoop2.x源码

    编译hadoop源码

    为什么要编译

    由于hadoop2.x 只支持Linux 32位,针对Linux 64位没有 native lib 库的支持,必须手动编译。所以下载的二进制包,当启动hadoop的时候,会出现如下警告:

    WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable

    如何编译

    在64位Linux下编译
    首先打开hadoop2.x源码下的build.txt 查看编译说明:

    Requirements:
    * Unix System
    * JDK 1.6+
    * Maven 3.0 or later
    * Findbugs 1.3.9 (if running findbugs)
    * ProtocolBuffer 2.5.0
    * CMake 2.6 or newer (if compiling native code)
    * Internet connection for first build (to fetch all Maven and Hadoop dependencies)


     Maven main modules:
      hadoop                              (Main Hadoop project)
             -hadoop-project   (Parent POM for all Hadoop Maven modules.        )
                                         (All plugins & dependencies versions are defined here.)
            -hadoop-project-dist (Parent POM for modules that generate distributions.)
            - hadoop-annotations (Generates the Hadoop doclet used to generated the Javadocs)
             - hadoop-assemblies         (Maven assemblies used by the different modules)
             - hadoop-common-project     (Hadoop Common)
             - hadoop-hdfs-project         (Hadoop HDFS)
             - hadoop-mapreduce-project   (Hadoop MapReduce)
             - hadoop-tools               (Hadoop tools like Streaming, Distcp, etc.)
             - hadoop-dist                (Hadoop distribution assembler)

    编译选项

    Building distributions:
    Create binary distribution without native code and without documentation:
      $ mvn package -Pdist -DskipTests -Dtar
    Create binary distribution with native code and with documentation:
      $ mvn package -Pdist,native,docs -DskipTests -Dtar
    Create source distribution:
      $ mvn package -Psrc -DskipTests
    Create source and binary distributions with native code and documentation:
      $ mvn package -Pdist,native,docs,src -DskipTests -Dtar
    Create a local staging version of the website (in /tmp/hadoop-site)
      $ mvn clean site; mvn site:stage -DstagingDirectory=/tmp/hadoop-site

    编译前安装依赖包

    yum install autoconf automake  libtool cmake
    yum install ncurses-devel
    yum install openssl-devel
    yum install lzo-devel zlib-devel gcc gcc-c++

    查软件包是否没有装上:

    rpm  -qa | grep autoconf

    编译前安装maven:
    下载:apache-maven-3.0.5-bin.tar.gz

    解压:

    tar  -zxvf apache-maven-3.0.5-bin.tar.gz

    设置环境变量:

    export MAVEN_HOME = /usr/local/maven
    export PATH=$PATH:$MAVEN_HOME/bin

    执行命令:source /etc/profile或者 . /etc/profile
    验证:mvn -v

    编译前安装protobuf
    下载:protobuf-2.5.0.tar.gz
    解压:

    tar  -zxvf protobuf-2.5.0.tar.gz

    编译安装:
    进入安装目录,进行配置,执行命令:

    ./configure

    安装命令:

    make & make install

    编译前安装findbugs
    下载:findbugs-3.0.0.tar.gz
    解压:

    tar  -zxvf findbugs-3.0.0.tar.gz

    设置环境变量:

    export FINDBUGS_HOME = /usr/local/findbugs
    export PATH=$PATH:$FINDBUGS_HOME/bin

    执行命令:source /etc/profile或者 . /etc/profile
    验证:findbugs -version

    MAVEN国内镜像配置

    1、进入安装目录 /opt/modules/apache-maven-3.0.5/conf,编辑 settings.xml 文件

     修改<mirrors>内容:
        <mirror>  
            <id>nexus-osc</id>  
            <mirrorOf>*</mirrorOf>  
            <name>Nexus osc</name>  
            <url>http://maven.oschina.net/content/groups/public/</url>  
        </mirror> 
    修改<profiles>内容:
    
    <profile>  
        <id>jdk-1.8</id>  
        <activation>  
            <jdk>1.8</jdk>  
        </activation>  
        <repositories>  
            <repository>  
                <id>nexus</id>  
                <name>local private nexus</name>  
                <url>http://maven.oschina.net/content/groups/public/</url>  
                <releases>  
                    <enabled>true</enabled>  
                </releases>  
                <snapshots>  
                    <enabled>false</enabled>  
                </snapshots>  
            </repository>  
        </repositories> 
        <pluginRepositories>  
            <pluginRepository>  
                <id>nexus</id>  
                <name>local private nexus</name>  
                <url>http://maven.oschina.net/content/groups/public/</url>  
                <releases>  
                    <enabled>true</enabled>  
                </releases>  
                <snapshots>  
                    <enabled>false</enabled>  
                </snapshots>  
            </pluginRepository>  
        </pluginRepositories> 
        </profile>  

    2、复制配置
    将该配置文件复制到用户目录,使得每次对maven创建时,都采用该配置
    * 查看用户目录【/home/hadoop】是否存在【.m2】文件夹,如没有,则创建

    $ cd /home/hadoop  
    $ mkdir .m2
    * 复制文件
    $ cp /opt/modules/apache-maven-3.0.5/conf/settings.xml ~/.m2/

    3.配置DNS
    修改: vi /etc/resolv.conf

    nameserver 8.8.8.8
    nameserver 8.8.4.4

    开始编译
    Hadoop2.2 .0 有bug ,编译 之前需要先 fix bug
    下载 hadoop2.x源码
    解压
    源码打patch,编辑目录$HADOOP_SRC_HOME/hadoop-common-project/hadoop-auth中的文件pom.xml,在第55行下增加以下内容:

    <dependency>
       <groupId>org.mortbay.jetty</groupId>
        <artfactId>jetty-util<artfactId>
        <scope>test</scope>
      </dependency>

    这里编译的是hadoop2.7.1版本

    mvn package –DskipTests –Pdist,native,docs -Dtar

    报错

    1.编译出错的话,如下有解决方式:

    export MAVEN_OPTS=”-Xms256m -Xmx512m”

    2.报错:java heap size

    把虚拟机内存调大,然后重启虚拟机,重新编译

    3.报错还有可能是maven下载jar的时候,网络出现问题,重新编译

    CentOS编译成功界面:

    这里写图片描述

    上面是Cetos下的编译,ubuntu编译:

    首先要安装的软件:
    JDk
    Maven
    Findbugs
    G++
    Protobuf
    Openssl
    CMake
    Ant

    Ubuntu编译成功界面:

    这里写图片描述

  • 相关阅读:
    racket eval
    mex不兼容
    【转】雷军 程序员随想
    UBoot 目录结构和编译过程
    远程监控web开发
    STL容器[08]
    STL容器[07]
    STL容器[17]
    STL容器[06]
    stl.map使用总结
  • 原文地址:https://www.cnblogs.com/bigdata1024/p/8387436.html
Copyright © 2011-2022 走看看