zoukankan      html  css  js  c++  java
  • Mac搭建Hadoop源码阅读环境

    1、本次Hadoop源码阅读环境使用的阅读工具是idea,Hadoop版本是2.7.3。需要安装的工具包括idea、jdk、maven、protobuf等

    2、jdk,使用的版本是1.8版,在jdk官网下载jdk-8u111-macosx-x64.dmg,点击安装,一路next。

    3、idea安装,略

    4、maven,使用的版本是3.3.9,下载apache-maven-3.3.9-bin.tar,解压:

    tar -zxvf  apache-maven-3.3.9-bin.tar

    进入 Maven 安装根目录 cd apache-maven-3.3.9

    进入 Maven配置文件目录 cd conf

    编辑settings.xml文件 vi  settings.xml

    设置Maven本地库的路径:这个路径要创建好。

     <localRepository>/opt/PRG/apache-maven-3.3.9/repository</localRepository>
    

    同时在加入阿里云的Maven库,参考:maven使用阿里云仓库

    <mirror>
              <id>nexus-aliyun</id>
              <mirrorOf>*</mirrorOf>
              <name>Nexus aliyun</name>
              <url>http://maven.aliyun.com/nexus/content/groups/public</url>
    </mirror>
    

    设置Maven环境变量:

    vi ~/.bash_profile

    export MAVEN_HOME=/opt/PRG/apache-maven-3.3.9
    export PATH=${PATH}:${MAVEN_HOME}/bin:
    export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"
    export PATH
    

    更新环境变量 source ~/.bash_profile

    mvn -version 能看到相应的版本,则Maven安装成功。

    5、安装protobuf,下载protobuf-2.5.0.tar.gz,下载地址: http://pan.baidu.com/s/1pJlZubT (网友提供)

    解压:tar -zxvf protobuf-2.5.0.tar.gz

    cd protobuf-2.5.0

    依次执行,参考:http://blog.sina.com.cn/s/blog_7d1531ed0101fmc5.html

     1 ./configure
     2 make
     3 make check
     4 make install
    

    如果报没有权限的错误,su root 切换到root执行。

    最后输入  protoc --version命令,如显示libprotoc 2.5.0则安装成功

     6、编译Hadoop源码

    在Hadoop官网下载hadoop-2.7.3-src.tar.gz源码

    解压:tar -zxvf  hadoop-2.7.3-src.tar.gz

    cd hadoop-2.7.3-src

    执行:

    mvn package -Pdist -DskipTests -Dtar
    

    最后是漫长的编译过程。。。

    下载依赖jar和相关文件。。。

    最后输出:[INFO] Reactor Summary:

    [INFO] ------------------------------------------------------------------------
    [INFO] Reactor Summary:
    [INFO] 
    [INFO] Apache Hadoop Main ................................. SUCCESS [  5.429 s]
    [INFO] Apache Hadoop Build Tools .......................... SUCCESS [  2.148 s]
    [INFO] Apache Hadoop Project POM .......................... SUCCESS [  1.061 s]
    [INFO] Apache Hadoop Annotations .......................... SUCCESS [  3.297 s]
    [INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.200 s]
    [INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  1.627 s]
    [INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  3.751 s]
    [INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  4.644 s]
    [INFO] Apache Hadoop Auth ................................. SUCCESS [  6.750 s]
    [INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  3.126 s]
    [INFO] Apache Hadoop Common ............................... SUCCESS [01:10 min]
    [INFO] Apache Hadoop NFS .................................. SUCCESS [  5.191 s]
    [INFO] Apache Hadoop KMS .................................. SUCCESS [01:12 min]
    [INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.037 s]
    [INFO] Apache Hadoop HDFS ................................. SUCCESS [01:38 min]
    [INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 34.932 s]
    [INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [ 13.482 s]
    [INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  3.823 s]
    [INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.039 s]
    [INFO] hadoop-yarn ........................................ SUCCESS [  0.041 s]
    [INFO] hadoop-yarn-api .................................... SUCCESS [ 28.981 s]
    [INFO] hadoop-yarn-common ................................. SUCCESS [ 22.892 s]
    [INFO] hadoop-yarn-server ................................. SUCCESS [  0.028 s]
    [INFO] hadoop-yarn-server-common .......................... SUCCESS [  7.887 s]
    [INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [  9.785 s]
    [INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [  2.782 s]
    [INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [  5.300 s]
    [INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [ 14.558 s]
    [INFO] hadoop-yarn-server-tests ........................... SUCCESS [  4.048 s]
    [INFO] hadoop-yarn-client ................................. SUCCESS [  4.785 s]
    [INFO] hadoop-yarn-server-sharedcachemanager .............. SUCCESS [  3.260 s]
    [INFO] hadoop-yarn-applications ........................... SUCCESS [  0.024 s]
    [INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [  2.129 s]
    [INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [  1.734 s]
    [INFO] hadoop-yarn-site ................................... SUCCESS [  0.026 s]
    [INFO] hadoop-yarn-registry ............................... SUCCESS [  4.279 s]
    [INFO] hadoop-yarn-project ................................ SUCCESS [  3.812 s]
    [INFO] hadoop-mapreduce-client ............................ SUCCESS [  0.124 s]
    [INFO] hadoop-mapreduce-client-core ....................... SUCCESS [ 14.922 s]
    [INFO] hadoop-mapreduce-client-common ..................... SUCCESS [ 11.181 s]
    [INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [  2.904 s]
    [INFO] hadoop-mapreduce-client-app ........................ SUCCESS [  7.072 s]
    [INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [  4.399 s]
    [INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [  4.986 s]
    [INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [  1.767 s]
    [INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  4.552 s]
    [INFO] hadoop-mapreduce ................................... SUCCESS [  2.718 s]
    [INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [  5.164 s]
    [INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [  8.356 s]
    [INFO] Apache Hadoop Archives ............................. SUCCESS [  2.022 s]
    [INFO] Apache Hadoop Rumen ................................ SUCCESS [  4.104 s]
    [INFO] Apache Hadoop Gridmix .............................. SUCCESS [  3.251 s]
    [INFO] Apache Hadoop Data Join ............................ SUCCESS [  1.897 s]
    [INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [  1.855 s]
    [INFO] Apache Hadoop Extras ............................... SUCCESS [  2.406 s]
    [INFO] Apache Hadoop Pipes ................................ SUCCESS [  0.023 s]
    [INFO] Apache Hadoop OpenStack support .................... SUCCESS [  3.771 s]
    [INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 17.460 s]
    [INFO] Apache Hadoop Azure support ........................ SUCCESS [  4.841 s]
    [INFO] Apache Hadoop Client ............................... SUCCESS [  6.743 s]
    [INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  1.029 s]
    [INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  4.379 s]
    [INFO] Apache Hadoop Tools Dist ........................... SUCCESS [  6.696 s]
    [INFO] Apache Hadoop Tools ................................ SUCCESS [  0.022 s]
    [INFO] Apache Hadoop Distribution ......................... SUCCESS [ 29.114 s]
    [INFO] ------------------------------------------------------------------------
    [INFO] BUILD SUCCESS
    [INFO] ------------------------------------------------------------------------
    [INFO] Total time: 10:08 min
    [INFO] Finished at: 2017-01-02T11:41:11+08:00
    [INFO] Final Memory: 229M/959M
    [INFO] ------------------------------------------------------------------------

    没有报错,则说明编译成功。

    7、把编译好的Hadoop源码导入Idea

    源码路径

    找到编译好的Hadoop,选择即可,next。

    选择Maven工程 

    之后一路next即可,之后就可以开始阅读源码了。

     

  • 相关阅读:
    CVS是版本控制
    AMD CPU 看清楚
    亚洲卫视 > 技术讲义
    Windows下安装APM大全
    Windows Server 2003的几个奇怪的设置问题
    TortoiseCVS 简明流程实例
    MESSAGEBOX() 函数
    用于多表更新
    H1B签证问题
    一些好的表格
  • 原文地址:https://www.cnblogs.com/fillPv/p/6242239.html
Copyright © 2011-2022 走看看