zoukankan      html  css  js  c++  java

  • mac os x 编译hadoop

    查阅有关官方介绍 http://wiki.apache.org/hadoop/HowToContribute 中有说明：Hadoop本地库只支持*nix平台，已经广泛使用在GNU/Linux平台上，但是不支持 Cygwin  和 Mac OS X 。搜索后发现已经有人给出了Mac OSX 系统下编译生成本地库的patch，下面详细介绍在Mac OSX 平台下编译Hadoop本地库的方法。

    [一]、环境说明：

    • Hadoop 2.2.0
    • Mac OS X 10.9.1

    详细的环境依赖（protoc、cmake 等）参见：Hadoop2.2.0源码编译 （http://www.micmiu.com/opensource/hadoop/hadoop-build-source-2-2-0/）中介绍。

    [二]、Mac OSX 编译本地库的步骤：

     1、checkout Hadoop 2.2.0的源码

    1 $svn co https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.2.0/

    2、patch 相关补丁

    官方讨论地址：https://issues.apache.org/jira/browse/HADOOP-9648 里面有详细介绍

    补丁下载链接：https://issues.apache.org/jira/secure/attachment/12617363/HADOOP-9648.v2.patch

    1 #切换到hadoop 源码的根目录
    2 $wget https://issues.apache.org/jira/secure/attachment/12617363/HADOOP-9648.v2.patch
    3 $patch -p1 < HADOOP-9648.v2.patch

    ps：如果要回退patch 执行：patch -RE -p1 < HADOOP-9648.v2.patch 即可。

    3、编译本地库

    在Hadoop源码的根目录下执行编译本地库命令：

    1 $ mvn package -Pdist,native -DskipTests -Dtar

    编译成功看到如下日志信息：

    1 [INFO] ------------------------------------------------------------------------
    2 [INFO] Reactor Summary:
    3 [INFO]
    4 [INFO] Apache Hadoop Main ................................ SUCCESS [1.511s]
    5 [INFO] Apache Hadoop Project POM ......................... SUCCESS [0.493s]
    6 [INFO] Apache Hadoop Annotations ......................... SUCCESS [0.823s]
    7 [INFO] Apache Hadoop Project Dist POM .................... SUCCESS [0.561s]
    8 [INFO] Apache Hadoop Assemblies .......................... SUCCESS [0.245s]
    9 [INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [2.465s]
    10 [INFO] Apache Hadoop MiniKDC ............................. SUCCESS [0.749s]
    11 [INFO] Apache Hadoop Auth ................................ SUCCESS [0.832s]
    12 [INFO] Apache Hadoop Auth Examples ....................... SUCCESS [2.070s]
    13 [INFO] Apache Hadoop Common .............................. SUCCESS [1:00.030s]
    14 [INFO] Apache Hadoop NFS ................................. SUCCESS [0.285s]
    15 [INFO] Apache Hadoop Common Project ...................... SUCCESS [0.049s]
    16 [INFO] Apache Hadoop HDFS ................................ SUCCESS [1:13.339s]
    17 [INFO] Apache Hadoop HttpFS .............................. SUCCESS [20.259s]
    18 [INFO] Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS [0.767s]
    19 [INFO] Apache Hadoop HDFS-NFS ............................ SUCCESS [0.279s]
    20 [INFO] Apache Hadoop HDFS Project ........................ SUCCESS [0.046s]
    21 [INFO] hadoop-yarn ....................................... SUCCESS [0.239s]
    22 [INFO] hadoop-yarn-api ................................... SUCCESS [7.641s]
    23 [INFO] hadoop-yarn-common ................................ SUCCESS [5.479s]
    24 [INFO] hadoop-yarn-server ................................ SUCCESS [0.114s]
    25 [INFO] hadoop-yarn-server-common ......................... SUCCESS [1.743s]
    26 [INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [6.381s]
    27 [INFO] hadoop-yarn-server-web-proxy ...................... SUCCESS [0.259s]
    28 [INFO] hadoop-yarn-server-resourcemanager ................ SUCCESS [0.578s]
    29 [INFO] hadoop-yarn-server-tests .......................... SUCCESS [0.303s]
    30 [INFO] hadoop-yarn-client ................................ SUCCESS [0.233s]
    31 [INFO] hadoop-yarn-applications .......................... SUCCESS [0.062s]
    32 [INFO] hadoop-yarn-applications-distributedshell ......... SUCCESS [0.253s]
    33 [INFO] hadoop-mapreduce-client ........................... SUCCESS [0.074s]
    34 [INFO] hadoop-mapreduce-client-core ...................... SUCCESS [1.504s]
    35 [INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [0.242s]
    36 [INFO] hadoop-yarn-site .................................. SUCCESS [0.172s]
    37 [INFO] hadoop-yarn-project ............................... SUCCESS [1.235s]
    38 [INFO] hadoop-mapreduce-client-common .................... SUCCESS [3.664s]
    39 [INFO] hadoop-mapreduce-client-shuffle ................... SUCCESS [0.183s]
    40 [INFO] hadoop-mapreduce-client-app ....................... SUCCESS [0.495s]
    41 [INFO] hadoop-mapreduce-client-hs ........................ SUCCESS [1.296s]
    42 [INFO] hadoop-mapreduce-client-jobclient ................. SUCCESS [0.580s]
    43 [INFO] hadoop-mapreduce-client-hs-plugins ................ SUCCESS [0.213s]
    44 [INFO] Apache Hadoop MapReduce Examples .................. SUCCESS [0.344s]
    45 [INFO] hadoop-mapreduce .................................. SUCCESS [1.303s]
    46 [INFO] Apache Hadoop MapReduce Streaming ................. SUCCESS [0.257s]
    47 [INFO] Apache Hadoop Distributed Copy .................... SUCCESS [9.925s]
    48 [INFO] Apache Hadoop Archives ............................ SUCCESS [0.282s]
    49 [INFO] Apache Hadoop Rumen ............................... SUCCESS [0.403s]
    50 [INFO] Apache Hadoop Gridmix ............................. SUCCESS [0.283s]
    51 [INFO] Apache Hadoop Data Join ........................... SUCCESS [0.197s]
    52 [INFO] Apache Hadoop Extras .............................. SUCCESS [0.241s]
    53 [INFO] Apache Hadoop Pipes ............................... SUCCESS [8.249s]
    54 [INFO] Apache Hadoop OpenStack support ................... SUCCESS [0.492s]
    55 [INFO] Apache Hadoop Client .............................. SUCCESS [0.373s]
    56 [INFO] Apache Hadoop Mini-Cluster ........................ SUCCESS [0.133s]
    57 [INFO] Apache Hadoop Scheduler Load Simulator ............ SUCCESS [0.439s]
    58 [INFO] Apache Hadoop Tools Dist .......................... SUCCESS [0.596s]
    59 [INFO] Apache Hadoop Tools ............................... SUCCESS [0.044s]
    60 [INFO] Apache Hadoop Distribution ........................ SUCCESS [0.194s]
    61 [INFO] ------------------------------------------------------------------------
    62 [INFO] BUILD SUCCESS
    63 [INFO] ------------------------------------------------------------------------
    64 [INFO] Total time: 3:44.266s
    65 [INFO] Finished at: Fri Jan 17 10:06:17 CST 2014
    66 [INFO] Final Memory: 66M/123M
    67 [INFO] ------------------------------------------------------------------------
    68 micmiu-mbp:trunk micmiu$

    编译通过后可在 <HADOOP源码根目录>/hadoop-dist/target/hadoop-2.2.0/lib/ 目录下看到如下内容：

    查看源代码
    打印帮助
    1 micmiu-mbp:lib micmiu$ tree
    2 .
    3 |____.DS_Store
    4 |____native
    5 | |____libhadoop.1.0.0.dylib
    6 | |____libhadoop.a
    7 | |____libhadoop.dylib
    8 | |____libhadooppipes.a
    9 | |____libhadooputils.a
    10 | |____libhdfs.0.0.0.dylib
    11 | |____libhdfs.a
    12 | |____libhdfs.dylib

    然后把 上面生成的本地库 copy到部署环境相应的位置，再建立软连接即可：

    1 $ls -s libhadoop.1.0.0.dylib libhadoop.so
    2 $ls -s libhdfs.0.0.0.dylib libhdfs.so





    错误处理

    运行

    clean install  package -Pdist -P-cbuild  -DskipTests  -Dtar 

    报各种错误

    1、报错[ERROR] Failed to execute goal org.codehaus.mojo:native-maven-plugin:1.0-alpha-7:javah (default) on project hadoop-common: Error running javah command: Error executing command line. Exit code:1 -> [Help 1]
    修改hadoop-common-project/hadoop-common/pom.xml 文件中，env.JAVA_HOME改为java.home

    2、报错

    /hadoop-2.2.0-src/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/JniBasedUnixGroupsNetgroupMapping.c:77:26: error: invalid operands to binary expression (‘void’ and ‘int’)
    [exec] if(setnetgrent(cgroup) == 1) {
    [exec] ~~~~~~~~~~~~~~~~~~~ ^ ~
    [exec] 1 error generated.
    [exec] make[2]: *** [CMakeFiles/hadoop.dir/main/native/src/org/apache/hadoop/security/JniBasedUnixGroupsNetgroupMapping.c.o] Error 1
    [exec] make[1]: *** [CMakeFiles/hadoop.dir/all] Error 2
    [exec] make: *** [all] Error 2

    [ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:run (make) on project hadoop-common: An Ant BuildException has occured: exec returned: 2 -> [Help 1]

    修改mvn3的配置文件：/opt/local/share/java/maven3/settings.xml
    在<mirrors>…</mirrors>里添加国内源：

    <mirrors>
    <mirror>
    <id>nexus-osc</id>
    <mirrorOf>*</mirrorOf>
    <name>Nexusosc</name>
    <url>http://maven.oschina.net/content/groups/public/</url>
    </mirror>
    </mirrors>

    在<profiles>…</profiles>标签中增加以下内容：
    <profile>
    <id>jdk-1.7</id>
    <activation>
    <jdk>1.7<k>
    </activation>
    <repositories>
    <repository>
    <id>nexus</id>
    <name>local private nexus</name>
    <url>http://maven.oschina.net/content/groups/public/</url>
    <releases>
    <enabled>true</enabled>
    </releases>
    <snapshots>
    <enabled>false</enabled>
    </snapshots>
    </repository>
    </repositories>
    <pluginRepositories>
    <pluginRepository>
    <id>nexus</id>
    <name>local private nexus</name>
    <url>http://maven.oschina.net/content/groups/public/</url>
    <releases>
    <enabled>true</enabled>
    </releases>
    <snapshots>
    <enabled>false</enabled>
    </snapshots>
    </pluginRepository>
    </pluginRepositories>
    </profile>
    </profiles>

    注意修改jdk version number

    将刚才的maven 配置文件拷贝到当前用户的home目录下:
    settings.xml   copy 到  your_hadoop_usr_home/.m2/
    cp settings.xml ~/.m2

    3、报错[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile (default-compile) on project hadoop-hdfs: Fatal error compiling: Error while executing the compiler. InvocationTargetException: Java heap space

    分配内存不足,参考如下为maven配置JVM参数: export MAVEN_OPTS=”-Xms256m -Xmx512m -Djava.awt.headless=true”

    4、报错 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile (default-compile) on project hadoop-hdfs: Compilation failure
    [ERROR] Failure executing javac, but could not parse the error:

    执行maven clean，然后再

    export MAVEN_OPTS=”-Xms256m -Xmx512m -Djava.awt.headless=true”
    三、最重要的一点，build your code是使用这个command line(Only for Mac OS):
    mvn clean install -P-cbuild

    编译之前， 你在hadoop-2.2.0-src目录（/Users/JuneMAC/hadoop/release-2.2.0）下执行
    mvn clean install –DskipTests

    上面的成功后，执行下面这个，生成安装包
    mvn clean install  package -Pdist -P-cbuild  -DskipTests  -Dtar

    执行完成后，可以在/Users/JuneMAC/hadoop/release-2.2.0/hadoop-dist/target/
    下找到
    hadoop-2.2.0.tar.gz
    将上面这个编译好的源码包解压到：
    /Users/JuneMAC/hadoop/
    然后进行相关配置
    解压之后的源码包和官网下载下来的源码包相对比，没有lib目录
    相关解释：
    “Here we use the additional options to stop compiling the native code.
    this is the key reason why we need use -P-cbuild option”
    上面这个是原因，好像不是很重要。实际上如果指定-Pdist,native 生成native lib 不成功，查阅有关官方介绍发现：Hadoop本地库只支持*nix平台，已经广泛使用在GNU/Linux平台上，但是不支持 Cygwin  和 Mac OS X 。
  • 相关阅读:
    Java基础 Day02(个人复习整理)
    Java基础 Day01(个人复习整理)
    linux-rpm
    linux常用命令
    rpm构建流程学习总结
    git相关
    sql相关
    ssh打通
    element ui FORM表单
    python threading多线程
  • 原文地址：https://www.cnblogs.com/liuruichao/p/4005849.html
Copyright © 2011-2022 走看看