zoukankan      html  css  js  c++  java
  • Ubuntu下编译 Hadoop2.9

    Ubuntu 下编译 Hadoop-2.9.2

    系统环境

    系统: ubuntu-18.10-desktop-amd64
    maven: Apache Maven 3.6.0
    jdk: jdk_1.8.0_201
    ProtocolBuffer: portoc-2.5.0
    OpenSSL: OpenSSL 1.1.1
    cmake: cmake3.12.1
    hadoop: hadoop-2.9.2
    

    安装所需环境

    # 安装各种所需依赖库
    $ sudo apt-get install autoconf automake libtool zlib1g-dev pkg-config libssl-dev libsnappy-dev libbz2-dev
    
    # 安装ssh-server
    $ sudo apt-get install ssh-server
    
    # 安装openssl
    $ sudo apt-get install openssl
    
    # 安装cmake
    $ sudo apt-get install cmake 
    
    # 安装编译c++库
    $ sudo apt-get install g++
    
    # 安装protobuf
    # 下载protobuf-2.5.0
    $ wget https://github.com/protocolbuffers/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz
    
    # 解压缩
    $ tar -zxvf protobuf-2.5.0.tar.gz
    
    # 进入protobuf解压缩目录配置
    $ ./configur --prefix=/usr/local/protoc-2.5.0
    
    # 编译安装
    $ make && make install
    
    # 配置到环境变量中~/.bash_profile
    export PATH="/usr/local/protoc-2.5.0/bin:$PATH"
    
    

    编译过程中可能遇到的问题

    注意 libssl-dev安装换成 libssl1.0-dev版本。 我之前就是默认安转,在编译OpenSSL功能的时候一直卡主编译不通过。具体错误形式如下

    [exec] make[2]: *** [CMakeFiles/hadoop_static.dir/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c.o] Error 1
    [exec] make[1]: *** [CMakeFiles/hadoop_static.dir/all] Error 2
    [exec] make: *** [all] Error 2
    
    # 移除libssl-dev
    $ sudo apt-get remote libssl-dev
    
    # 安装libssl1.0-dev
    $ sudo apt-get install libssl1.0-dev
    
    

    开始编译 hadoop

    $ mvn package -Pdist,native -DskipTests -Dtar
    
    命令执行后需要等待他下载编译所需的所有依赖过程可能会很久
    
    [INFO] Reactor Summary for Apache Hadoop Main 2.9.2:
    [INFO] 
    [INFO] Apache Hadoop Main ................................. SUCCESS [  0.758 s]
    [INFO] Apache Hadoop Build Tools .......................... SUCCESS [  0.426 s]
    [INFO] Apache Hadoop Project POM .......................... SUCCESS [  0.470 s]
    [INFO] Apache Hadoop Annotations .......................... SUCCESS [  1.169 s]
    [INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.134 s]
    [INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  1.145 s]
    [INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  2.224 s]
    [INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  1.923 s]
    [INFO] Apache Hadoop Auth ................................. SUCCESS [  2.653 s]
    [INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  1.823 s]
    [INFO] Apache Hadoop Common ............................... SUCCESS [ 40.786 s]
    [INFO] Apache Hadoop NFS .................................. SUCCESS [  2.925 s]
    [INFO] Apache Hadoop KMS .................................. SUCCESS [  8.053 s]
    [INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.050 s]
    [INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 46.997 s]
    [INFO] Apache Hadoop HDFS ................................. SUCCESS [ 42.883 s]
    [INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [  5.907 s]
    [INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 11.476 s]
    [INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [  5.320 s]
    [INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  2.055 s]
    [INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [ 11.390 s]
    [INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.053 s]
    [INFO] Apache Hadoop YARN ................................. SUCCESS [  0.045 s]
    [INFO] Apache Hadoop YARN API ............................. SUCCESS [  8.620 s]
    [INFO] Apache Hadoop YARN Common .......................... SUCCESS [ 25.431 s]
    [INFO] Apache Hadoop YARN Registry ........................ SUCCESS [  2.577 s]
    [INFO] Apache Hadoop YARN Server .......................... SUCCESS [  0.067 s]
    [INFO] Apache Hadoop YARN Server Common ................... SUCCESS [  6.540 s]
    [INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 25.577 s]
    [INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [  1.779 s]
    [INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [ 11.509 s]
    [INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [  2.671 s]
    [INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 12.742 s]
    [INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [  0.760 s]
    [INFO] Apache Hadoop YARN Client .......................... SUCCESS [  3.285 s]
    [INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [  1.976 s]
    [INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [  1.730 s]
    [INFO] Apache Hadoop YARN Router .......................... SUCCESS [  2.516 s]
    [INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [ 23.469 s]
    [INFO] Apache Hadoop YARN Timeline Service HBase tests .... SUCCESS [  1.727 s]
    [INFO] Apache Hadoop YARN Applications .................... SUCCESS [  0.037 s]
    [INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [  1.611 s]
    [INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [  1.039 s]
    [INFO] Apache Hadoop YARN Site ............................ SUCCESS [  0.041 s]
    [INFO] Apache Hadoop YARN UI .............................. SUCCESS [  0.027 s]
    [INFO] Apache Hadoop YARN Project ......................... SUCCESS [  4.166 s]
    [INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [  0.126 s]
    [INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 11.880 s]
    [INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [  8.171 s]
    [INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [  1.818 s]
    [INFO] Apache Hadoop MapReduce App ........................ SUCCESS [  4.549 s]
    [INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [  2.999 s]
    [INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [  3.505 s]
    [INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [  1.104 s]
    [INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  2.656 s]
    [INFO] Apache Hadoop MapReduce ............................ SUCCESS [  2.052 s]
    [INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [  9.580 s]
    [INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [  2.545 s]
    [INFO] Apache Hadoop Archives ............................. SUCCESS [  1.964 s]
    [INFO] Apache Hadoop Archive Logs ......................... SUCCESS [  1.355 s]
    [INFO] Apache Hadoop Rumen ................................ SUCCESS [  2.651 s]
    [INFO] Apache Hadoop Gridmix .............................. SUCCESS [  2.451 s]
    [INFO] Apache Hadoop Data Join ............................ SUCCESS [  1.281 s]
    [INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [  0.910 s]
    [INFO] Apache Hadoop Extras ............................... SUCCESS [  1.269 s]
    [INFO] Apache Hadoop Pipes ................................ SUCCESS [  4.864 s]
    [INFO] Apache Hadoop OpenStack support .................... SUCCESS [  2.166 s]
    [INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 17.425 s]
    [INFO] Apache Hadoop Azure support ........................ SUCCESS [  3.589 s]
    [INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [  1.737 s]
    [INFO] Apache Hadoop Client ............................... SUCCESS [  4.883 s]
    [INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  0.535 s]
    [INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  3.363 s]
    [INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [  2.404 s]
    [INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [  1.732 s]
    [INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 17.061 s]
    [INFO] Apache Hadoop Tools ................................ SUCCESS [  0.035 s]
    [INFO] Apache Hadoop Distribution ......................... SUCCESS [ 42.773 s]
    [INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [  2.183 s]
    [INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [  0.229 s]
    [INFO] ------------------------------------------------------------------------
    [INFO] BUILD SUCCESS
    [INFO] ------------------------------------------------------------------------
    [INFO] Total time:  08:19 min
    [INFO] Finished at: 2019-05-08T17:23:30+08:00
    [INFO] ------------------------------------------------------------------------
    leone@leone-ubuntu:~/Downloads/hadoop-2.9.2-src$ 
    
    

    大功告成编译后的hadoop放在hadoop-2.9.2-srchadoop-dist arget目录下。

  • 相关阅读:
    《Django By Example》第十二章(终章) 中文 翻译 (个人学习,渣翻)
    《Django By Example》第十一章 中文 翻译 (个人学习,渣翻)
    《Django By Example》第十章 中文 翻译 (个人学习,渣翻)
    《Django By Example》第九章 中文 翻译 (个人学习,渣翻)
    《Django By Example》第八章 中文 翻译 (个人学习,渣翻)
    《Django By Example》第五章 中文 翻译 (个人学习,渣翻)
    我的superui开源后台bootstrap开发框架
    LayoutInflater 总结
    Android屏幕分辨率概念(dp、dip、dpi、sp、px)
    android studio安装问题
  • 原文地址:https://www.cnblogs.com/janlle/p/10859037.html
Copyright © 2011-2022 走看看