Windows 10 上编译 Hadoop

下载源码

源码下载地址（Source download）：https://hadoop.apache.org/releases.html

这里以 2.9.2 为例，查看源码中的编译说明文件 BUILDING.txt，截取 windows 部分

Requirements:

* Windows System
* JDK 1.7 or 1.8
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer
* Windows SDK 7.1 or Visual Studio 2010 Professional
* Windows SDK 8.1 (if building CPU rate control for the container executor)
* zlib headers (if building native code bindings for zlib)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)
* Unix command-line tools from GnuWin32: sh, mkdir, rm, cp, tar, gzip. These tools must be present on your PATH.
* Python ( for generation of docs using 'mvn site')

Unix command-line tools are also included with the Windows Git package which
can be downloaded from http://git-scm.com/downloads

If using Visual Studio, it must be Visual Studio 2010 Professional (not 2012).
Do not use Visual Studio Express.  It does not support compiling for 64-bit,
which is problematic if running a 64-bit system.  The Windows SDK 7.1 is free to download here:

http://www.microsoft.com/en-us/download/details.aspx?id=8279

The Windows SDK 8.1 is available to download at:

http://msdn.microsoft.com/en-us/windows/bg162891.aspx

Cygwin is neither required nor supported.

环境准备

1.JDK

https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

# 设置系统环境变量
setx /M JAVA_HOME "D:hadoopjdk1.8.0_192"
setx /M Path "%Path%;%JAVA_HOME%in;%JAVA_HOME%jrein"

2.Maven

http://maven.apache.org/download.cgi

# 设置系统环境变量
setx /M M2_HOME "D:hadoopapache-maven-3.6.1"
setx /M Path "%Path%;%M2_HOME%in"

confsettings.xml 仓库配置

<!-- 本地仓库路径 -->
<localRepository>D:hadoop
epo</localRepository>

<!-- 网络仓库地址 -->
<mirrors>
    <mirror>
        <id>central</id>
        <mirrorOf>central</mirrorOf>
        <name>aliyunmaven</name>
        <url>https://maven.aliyun.com/repository/central</url>
    </mirror>
    <mirror>
        <id>apache.snapshots.https</id>
        <mirrorOf>apache.snapshots.https</mirrorOf>
        <name>aliyunmaven</name>
        <url>https://maven.aliyun.com/repository/apache-snapshots</url>
    </mirror>
</mirrors>

3.ProtocolBuffer

https://github.com/protocolbuffers/protobuf/releases/tag/v2.5.0

注：除了下载 protobuf 源码外，还需要下载相应版本的编译过的用于 Windows 平台的 protoc 命令（protoc-2.5.0-win32.zip）,该命令用于将 .proto 文件转化为 Java 或 C++ 源文件。

将两个压缩包解压，然后将 protoc.exe 复制到 protobuf-2.5.0src 目录下

# 安装 ProtocolBuffer
cd D:hadoopprotobuf-2.5.0java
mvn test 
mvn install

# 设置系统环境变量
setx /M Path "%Path%;D:hadoopprotobuf-2.5.0src"

# 查看版本
protoc --version

4.CMake

https://cmake.org/download/

# 设置系统环境变量
setx /M Path "%Path%;D:hadoopcmake-3.14.3-win64-x64in"

5.Visual Studio 2010 Professional

文件名     cn_visual_studio_2010_professional_x86_dvd_532145.iso
SHA1　　　 33D323446131AB9565082D65C9C380BBD7FF228F
文件大小   2.41GB
发布时间   2010-05-26

ed2k://|file|cn_visual_studio_2010_professional_x86_dvd_532145.iso|2591844352|6001253431AFE573E4344F5A0B1D9CAC|/

6.Zlib

http://www.zlib.net/

使用 MSVC 方式编译，打开开始菜单，在 Microsoft Visual Studio 2010 下找到 Visual Studio x64 Win64 命令提示(2010)，用管理员身份运行

cd D:hadoopzlib-1.2.11
nmake -f win32/Makefile.msc

# 编译完成后在该目录下可看到 zlib1.dll 文件

设置系统环境变量

setx /M ZLIB_HOME "D:hadoopzlib-1.2.11"

7.Git（需要 bash 命令）

https://git-scm.com/download/

# 设置系统环境变量
setx /M Path "%Path%;D:hadoopPortableGitin"

编译

1.设置 Platform

setx /M Platform "x64"

2.解压源码，开始编译

打开开始菜单，在 Microsoft Visual Studio 2010 下找到 Visual Studio x64 Win64 命令提示(2010)，用管理员身份运行

cd D:hadoophadoop-2.9.2-src
mvn package -Pdist,native-win -DskipTests -Dtar

# 编译完成后，编译好的文件在目录 hadoop-2.9.2-srchadoop-dist	arget 中

编译失败时多编译几次，或自己更换下 Maven 源再试，一些依赖的下载不是很稳定。附上编译成功的日志

[INFO] Reactor Summary for Apache Hadoop Main 2.9.2:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 39.854 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 35.664 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 13.814 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 13.440 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.205 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 28.307 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 26.019 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [07:47 min]
[INFO] Apache Hadoop Auth ................................. SUCCESS [03:03 min]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  4.259 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [10:30 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [  3.421 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [01:23 min]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.064 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [01:11 min]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [01:45 min]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [  3.279 s]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 29.455 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [ 23.094 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  2.839 s]
[INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [ 12.846 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.044 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [  0.044 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [  8.375 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [04:12 min]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [  3.536 s]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [  0.050 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 32.678 s]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 10.878 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [  2.167 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [01:36 min]
[INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [ 14.856 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 21.482 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [  0.589 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [  3.674 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [  2.209 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [  1.911 s]
[INFO] Apache Hadoop YARN Router .......................... SUCCESS [  3.163 s]
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [02:35 min]
[INFO] Apache Hadoop YARN Timeline Service HBase tests .... SUCCESS [01:36 min]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [  0.045 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [  1.792 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [  1.213 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [  0.042 s]
[INFO] Apache Hadoop YARN UI .............................. SUCCESS [  0.045 s]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [  5.462 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [  0.134 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 23.516 s]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [ 13.747 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [  2.323 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [  6.160 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [  4.593 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [  2.615 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [  1.280 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  4.254 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [  2.127 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 13.367 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [  5.503 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [  1.260 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [  1.315 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [  3.355 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [  2.324 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [  1.477 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [  1.219 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [  1.615 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [  0.040 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [  2.513 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [06:23 min]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 46.260 s]
[INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [01:14 min]
[INFO] Apache Hadoop Client ............................... SUCCESS [  5.923 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  0.596 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  4.338 s]
[INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [01:43 min]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [ 30.751 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 13.557 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.047 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 47.214 s]
[INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [  0.554 s]
[INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [  0.049 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  55:18 min
[INFO] Finished at: 2019-04-25T22:47:26+08:00
[INFO] ------------------------------------------------------------------------

---

Windows 10编译Hadoop 2.6.0源码 - 作业部落 Cmd Markdown 编辑阅读器 （快照）

https://www.cnblogs.com/guoxiaoqian/p/4328812.html