zoukankan      html  css  js  c++  java

  • 从零教你如何获取hadoop2.4源码并使用eclipse关联hadoop2.4源码

    问题导读：
    1.如何通过官网src包，获取hadoop的全部代码
    2.通过什么样的操作，可以查看hadoop某个函数或则类的实现？
    3.maven的作用是什么？






    我们如果想搞开发，研究源码对我们的帮助很大。不明白原理就如同黑盒子，遇到问题，我们也摸不着思路。所以这里交给大家
    一.如何获取源码
    二.如何关联源码

    一.如何获取源码

    1.下载hadoop的maven程序包

    （1）官网下载
    这里我们先从官网上下载maven包hadoop-2.4.0-src.tar.gz。
    官网下载地址

    对于不知道怎么去官网下载，可以查看：新手指导：hadoop官网介绍及如何下载hadoop（2.4）各个版本与查看hadoop API介绍

    （2）网盘下载
    也可以从网盘下载：
    http://pan.baidu.com/s/1kToPuGB

    2.通过maven获取源码
    获取源码的方式有两种，一种是通过命令行的方式，一种是通过eclipse。这里主要讲通过命令的方式

    通过命令的方式获取源码：
    1.解压包

    <ignore_js_op> 


    解压包的时候遇到了下面问题。不过不用管，我们继续往下走

    1        : 无法创建文件：D:hadoop2hadoop-2.4.0-srchadoop-yarn-projecthadoop-yarnhadoop-yarn-serverhadoop-yarn-server-applicationhistoryservice argetclassesorgapachehadoopyarnserverapplicationhistoryserviceApplicationHistoryClientService$ApplicationHSClientProtocolHandler.class：
    路径和文件名总长度不能超过260个字符
    系统找不到指定的路径。        D:hadoop2hadoop-2.4.0-src.zip
    2        : 无法创建文件：D:hadoop2hadoop-2.4.0-srchadoop-yarn-projecthadoop-yarnhadoop-yarn-serverhadoop-yarn-server-applicationhistoryservice argetclassesorgapachehadoopyarnserverapplicationhistoryservice imelineLeveldbTimelineStore$LockMap$CountingReentrantLock.class：系统找不到指定的路径。        D:hadoop2hadoop-2.4.0-src.zip
    3        : 无法创建文件：D:hadoop2hadoop-2.4.0-srchadoop-yarn-projecthadoop-yarnhadoop-yarn-serverhadoop-yarn-server-applicationhistoryservice arget est-classesorgapachehadoopyarnserverapplicationhistoryservicewebappTestAHSWebApp$MockApplicationHistoryManagerImpl.class：系统找不到指定的路径。        D:hadoop2hadoop-2.4.0-src.zip
    4        : 无法创建文件：D:hadoop2hadoop-2.4.0-srchadoop-yarn-projecthadoop-yarnhadoop-yarn-serverhadoop-yarn-server-resourcemanager arget est-classesorgapachehadoopyarnserver esourcemanagermonitorcapacityTestProportionalCapacityPreemptionPolicy$IsPreemptionRequestFor.class：
    路径和文件名总长度不能超过260个字符
    系统找不到指定的路径。        D:hadoop2hadoop-2.4.0-src.zip
    5        : 无法创建文件：D:hadoop2hadoop-2.4.0-srchadoop-yarn-projecthadoop-yarnhadoop-yarn-serverhadoop-yarn-server-resourcemanager arget est-classesorgapachehadoopyarnserver esourcemanager ecoveryTestFSRMStateStore$TestFSRMStateStoreTester$TestFileSystemRMStore.class：系统找不到指定的路径。        D:hadoop2hadoop-2.4.0-src.zip
    6        : 无法创建文件：D:hadoop2hadoop-2.4.0-srchadoop-yarn-projecthadoop-yarnhadoop-yarn-serverhadoop-yarn-server-resourcemanager arget est-classesorgapachehadoopyarnserver esourcemanager ecoveryTestZKRMStateStore$TestZKRMStateStoreTester$TestZKRMStateStoreInternal.class：
    路径和文件名总长度不能超过260个字符
    系统找不到指定的路径。        D:hadoop2hadoop-2.4.0-src.zip
    7        : 无法创建文件：D:hadoop2hadoop-2.4.0-srchadoop-yarn-projecthadoop-yarnhadoop-yarn-serverhadoop-yarn-server-resourcemanager arget est-classesorgapachehadoopyarnserver esourcemanager ecoveryTestZKRMStateStoreZKClientConnections$TestZKClient$TestForwardingWatcher.class：
    路径和文件名总长度不能超过260个字符
    系统找不到指定的路径。        D:hadoop2hadoop-2.4.0-src.zip
    8        : 无法创建文件：D:hadoop2hadoop-2.4.0-srchadoop-yarn-projecthadoop-yarnhadoop-yarn-serverhadoop-yarn-server-resourcemanager arget est-classesorgapachehadoopyarnserver esourcemanager ecoveryTestZKRMStateStoreZKClientConnections$TestZKClient$TestZKRMStateStore.class：
    路径和文件名总长度不能超过260个字符
    系统找不到指定的路径。        D:hadoop2hadoop-2.4.0-src.zip
    9        : 无法创建文件：D:hadoop2hadoop-2.4.0-srchadoop-yarn-projecthadoop-yarnhadoop-yarn-serverhadoop-yarn-server-resourcemanager arget est-classesorgapachehadoopyarnserver esourcemanager mappattemptTestRMAppAttemptTransitions$TestApplicationAttemptEventDispatcher.class：
    路径和文件名总长度不能超过260个字符
    系统找不到指定的路径。        D:hadoop2hadoop-2.4.0-src.zip



    2.通过maven获取源码

    这里需要说明的是，在使用maven的时候，需要先安装jdk，protoc ，如果没有安装可以参考win7如何安装maven、安装protoc

    （1）进入hadoop-2.4.0-srchadoop-maven-plugins，运行mvn install

    1. D:hadoop2hadoop-2.4.0-srchadoop-maven-plugins>mvn install
    复制代码



    显示如下信息

    2. [INFO] Scanning for projects...
    3. [WARNING]
    4. [WARNING] Some problems were encountered while building the effective model for
    5. org.apache.hadoop:hadoop-maven-plugins:maven-plugin:2.4.0
    6. [WARNING] 'build.plugins.plugin.(groupId:artifactId)' must be unique but found d
    7. uplicate declaration of plugin org.apache.maven.plugins:maven-enforcer-plugin @
    8. org.apache.hadoop:hadoop-project:2.4.0, D:hadoop2hadoop-2.4.0-srchadoop-proje
    9. ctpom.xml, line 1015, column 15
    10. [WARNING]
    11. [WARNING] It is highly recommended to fix these problems because they threaten t
    12. he stability of your build.
    13. [WARNING]
    14. [WARNING] For this reason, future Maven versions might no longer support buildin
    15. g such malformed projects.
    16. [WARNING]
    17. [INFO]
    18. [INFO] Using the builder org.apache.maven.lifecycle.internal.builder.singlethrea
    19. ded.SingleThreadedBuilder with a thread count of 1
    20. [INFO]
    21. [INFO] ------------------------------------------------------------------------
    22. [INFO] Building Apache Hadoop Maven Plugins 2.4.0
    23. [INFO] ------------------------------------------------------------------------
    24. [INFO]
    25. [INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-maven-plugins
    26. ---
    27. [INFO] Executing tasks
    29. main:
    30. [INFO] Executed tasks
    31. [INFO]
    32. [INFO] --- maven-plugin-plugin:3.0:descriptor (default-descriptor) @ hadoop-mave
    33. n-plugins ---
    34. [INFO] Using 'UTF-8' encoding to read mojo metadata.
    35. [INFO] Applying mojo extractor for language: java-annotations
    36. [INFO] Mojo extractor for language: java-annotations found 2 mojo descriptors.
    37. [INFO] Applying mojo extractor for language: java
    38. [INFO] Mojo extractor for language: java found 0 mojo descriptors.
    39. [INFO] Applying mojo extractor for language: bsh
    40. [INFO] Mojo extractor for language: bsh found 0 mojo descriptors.
    41. [INFO]
    42. [INFO] --- maven-resources-plugin:2.2:resources (default-resources) @ hadoop-mav
    43. en-plugins ---
    44. [INFO] Using default encoding to copy filtered resources.
    45. [INFO]
    46. [INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @ hadoop-maven-
    47. plugins ---
    48. [INFO] Nothing to compile - all classes are up to date
    49. [INFO]
    50. [INFO] --- maven-plugin-plugin:3.0:descriptor (mojo-descriptor) @ hadoop-maven-p
    51. lugins ---
    52. [INFO] Using 'UTF-8' encoding to read mojo metadata.
    53. [INFO] Applying mojo extractor for language: java-annotations
    54. [INFO] Mojo extractor for language: java-annotations found 2 mojo descriptors.
    55. [INFO] Applying mojo extractor for language: java
    56. [INFO] Mojo extractor for language: java found 0 mojo descriptors.
    57. [INFO] Applying mojo extractor for language: bsh
    58. [INFO] Mojo extractor for language: bsh found 0 mojo descriptors.
    59. [INFO]
    60. [INFO] --- maven-resources-plugin:2.2:testResources (default-testResources) @ ha
    61. doop-maven-plugins ---
    62. [INFO] Using default encoding to copy filtered resources.
    63. [INFO]
    64. [INFO] --- maven-compiler-plugin:2.5.1:testCompile (default-testCompile) @ hadoo
    65. p-maven-plugins ---
    66. [INFO] No sources to compile
    67. [INFO]
    68. [INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hadoop-maven-plugins
    69. ---
    70. [INFO] No tests to run.
    71. [INFO]
    72. [INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ hadoop-maven-plugins ---
    73. [INFO] Building jar: D:hadoop2hadoop-2.4.0-srchadoop-maven-plugins argethad
    74. oop-maven-plugins-2.4.0.jar
    75. [INFO]
    76. [INFO] --- maven-plugin-plugin:3.0:addPluginArtifactMetadata (default-addPluginA
    77. rtifactMetadata) @ hadoop-maven-plugins ---
    78. [INFO]
    79. [INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ hadoop-
    80. maven-plugins ---
    81. [INFO]
    82. [INFO] --- maven-install-plugin:2.3.1:install (default-install) @ hadoop-maven-p
    83. lugins ---
    84. [INFO] Installing D:hadoop2hadoop-2.4.0-srchadoop-maven-plugins argethadoop
    85. -maven-plugins-2.4.0.jar to C:Usershyj.m2 epositoryorgapachehadoophadoop
    86. -maven-plugins2.4.0hadoop-maven-plugins-2.4.0.jar
    87. [INFO] Installing D:hadoop2hadoop-2.4.0-srchadoop-maven-pluginspom.xml to C:
    88. Usershyj.m2 epositoryorgapachehadoophadoop-maven-plugins2.4.0hadoop-ma
    89. ven-plugins-2.4.0.pom
    90. [INFO] ------------------------------------------------------------------------
    91. [INFO] BUILD SUCCESS
    92. [INFO] ------------------------------------------------------------------------
    93. [INFO] Total time: 4.891 s
    94. [INFO] Finished at: 2014-06-23T14:47:33+08:00
    95. [INFO] Final Memory: 21M/347M
    96. [INFO] ------------------------------------------------------------------------
    复制代码


    部分截图如下：

    <ignore_js_op> 


    <ignore_js_op> 



    （2）运行

    1. mvn eclipse:eclipse -DskipTests
    复制代码

    这时候注意，我们进入的是hadoop_home,我这里是D:hadoop2hadoop-2.4.0-src

    部分信息如下

    1. [INFO]
    2. [INFO] ------------------------------------------------------------------------
    3. [INFO] Reactor Summary:
    4. [INFO]
    5. [INFO] Apache Hadoop Main ................................ SUCCESS [  0.684 s]
    6. [INFO] Apache Hadoop Project POM ......................... SUCCESS [  0.720 s]
    7. [INFO] Apache Hadoop Annotations ......................... SUCCESS [  0.276 s]
    8. [INFO] Apache Hadoop Project Dist POM .................... SUCCESS [  0.179 s]
    9. [INFO] Apache Hadoop Assemblies .......................... SUCCESS [  0.121 s]
    10. [INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [  1.680 s]
    11. [INFO] Apache Hadoop MiniKDC ............................. SUCCESS [  1.802 s]
    12. [INFO] Apache Hadoop Auth ................................ SUCCESS [  1.024 s]
    13. [INFO] Apache Hadoop Auth Examples ....................... SUCCESS [  0.160 s]
    14. [INFO] Apache Hadoop Common .............................. SUCCESS [  1.061 s]
    15. [INFO] Apache Hadoop NFS ................................. SUCCESS [  0.489 s]
    16. [INFO] Apache Hadoop Common Project ...................... SUCCESS [  0.056 s]
    17. [INFO] Apache Hadoop HDFS ................................ SUCCESS [  2.770 s]
    18. [INFO] Apache Hadoop HttpFS .............................. SUCCESS [  0.965 s]
    19. [INFO] Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS [  0.629 s]
    20. [INFO] Apache Hadoop HDFS-NFS ............................ SUCCESS [  0.284 s]
    21. [INFO] Apache Hadoop HDFS Project ........................ SUCCESS [  0.061 s]
    22. [INFO] hadoop-yarn ....................................... SUCCESS [  0.052 s]
    23. [INFO] hadoop-yarn-api ................................... SUCCESS [  0.842 s]
    24. [INFO] hadoop-yarn-common ................................ SUCCESS [  0.322 s]
    25. [INFO] hadoop-yarn-server ................................ SUCCESS [  0.065 s]
    26. [INFO] hadoop-yarn-server-common ......................... SUCCESS [  0.972 s]
    27. [INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [  0.580 s]
    28. [INFO] hadoop-yarn-server-web-proxy ...................... SUCCESS [  0.379 s]
    29. [INFO] hadoop-yarn-server-applicationhistoryservice ...... SUCCESS [  0.281 s]
    30. [INFO] hadoop-yarn-server-resourcemanager ................ SUCCESS [  0.378 s]
    31. [INFO] hadoop-yarn-server-tests .......................... SUCCESS [  0.534 s]
    32. [INFO] hadoop-yarn-client ................................ SUCCESS [  0.307 s]
    33. [INFO] hadoop-yarn-applications .......................... SUCCESS [  0.050 s]
    34. [INFO] hadoop-yarn-applications-distributedshell ......... SUCCESS [  0.202 s]
    35. [INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [  0.194 s]
    36. [INFO] hadoop-yarn-site .................................. SUCCESS [  0.057 s]
    37. [INFO] hadoop-yarn-project ............................... SUCCESS [  0.066 s]
    38. [INFO] hadoop-mapreduce-client ........................... SUCCESS [  0.091 s]
    39. [INFO] hadoop-mapreduce-client-core ...................... SUCCESS [  1.321 s]
    40. [INFO] hadoop-mapreduce-client-common .................... SUCCESS [  0.786 s]
    41. [INFO] hadoop-mapreduce-client-shuffle ................... SUCCESS [  0.456 s]
    42. [INFO] hadoop-mapreduce-client-app ....................... SUCCESS [  0.508 s]
    43. [INFO] hadoop-mapreduce-client-hs ........................ SUCCESS [  0.834 s]
    44. [INFO] hadoop-mapreduce-client-jobclient ................. SUCCESS [  0.541 s]
    45. [INFO] hadoop-mapreduce-client-hs-plugins ................ SUCCESS [  0.284 s]
    46. [INFO] Apache Hadoop MapReduce Examples .................. SUCCESS [  0.851 s]
    47. [INFO] hadoop-mapreduce .................................. SUCCESS [  0.099 s]
    48. [INFO] Apache Hadoop MapReduce Streaming ................. SUCCESS [  0.742 s]
    49. [INFO] Apache Hadoop Distributed Copy .................... SUCCESS [  0.335 s]
    50. [INFO] Apache Hadoop Archives ............................ SUCCESS [  0.397 s]
    51. [INFO] Apache Hadoop Rumen ............................... SUCCESS [  0.371 s]
    52. [INFO] Apache Hadoop Gridmix ............................. SUCCESS [  0.230 s]
    53. [INFO] Apache Hadoop Data Join ........................... SUCCESS [  0.184 s]
    54. [INFO] Apache Hadoop Extras .............................. SUCCESS [  0.217 s]
    55. [INFO] Apache Hadoop Pipes ............................... SUCCESS [  0.048 s]
    56. [INFO] Apache Hadoop OpenStack support ................... SUCCESS [  0.244 s]
    57. [INFO] Apache Hadoop Client .............................. SUCCESS [  0.590 s]
    58. [INFO] Apache Hadoop Mini-Cluster ........................ SUCCESS [  0.230 s]
    59. [INFO] Apache Hadoop Scheduler Load Simulator ............ SUCCESS [  0.650 s]
    60. [INFO] Apache Hadoop Tools Dist .......................... SUCCESS [  0.334 s]
    61. [INFO] Apache Hadoop Tools ............................... SUCCESS [  0.042 s]
    62. [INFO] Apache Hadoop Distribution ........................ SUCCESS [  0.144 s]
    63. [INFO] ------------------------------------------------------------------------
    64. [INFO] BUILD SUCCESS
    65. [INFO] ------------------------------------------------------------------------
    66. [INFO] Total time: 31.234 s
    67. [INFO] Finished at: 2014-06-23T14:55:08+08:00
    68. [INFO] Final Memory: 84M/759M
    69. [INFO] ------------------------------------------------------------------------
    复制代码

    这时候，我们已经把源码给下载下来了。这时候，我们会看到文件会明显增大。


    <ignore_js_op> 



    3.关联eclipse源码

    加入我们以下程序
    <ignore_js_op> hadoop2.2mapreduce例子.rar (1.14 MB, 下载次数: 400, 售价: 1 云币) 
    如下图示，对他们进行了打包
    <ignore_js_op> 

    这两个文件， MaxTemperature.zip为mapreduce例子，mockito-core-1.8.5.jar为mapreduce例子所引用的包
    （这里需要说明的是，mapreduce为2.2，但是不影响关联源码，只是交给大家该如何关联源码）
    我们解压之后，导入eclipse
    （对于导入项目不熟悉，参考零基础教你如何导入eclipse项目）

    <ignore_js_op> 


    我们导入之后，看到很多的红线，这些其实都是没有引用包，下面我们开始解决这些语法问题。
    一、解决导入jar包
    （1）引入mockito-core-1.8.5.jar

    （2）hadoop2.4编译包中的jar文件，这些文件的位置如下：

    hadoop_home中sharehadoop文件夹下，具体我这的位置D:hadoop2hadoop-2.4.0sharehadoop
    找到里面的jar包，举例如下：lib文件中的jar包，以及下面的jar包都添加到buildpath中。
    如果对于引用包，不知道该如何添加这些jar包，参考hadoop开发方式总结及操作指导。
    （注意的是，我们这里是引入的是编译包，编译的下载hadoop--642.4.0.tar.gz
    链接: http://pan.baidu.com/s/1c0vPjG0 密码:xj6l）

    更多包下载可以参考hadoop家族、strom、spark、Linux、flume等jar包、安装包汇总下载

    <ignore_js_op> 



    <ignore_js_op> 


    二、关联源码
    1.我们导入jar包之后，就没有错误了，如下图所示

    <ignore_js_op> 

    2.找不到源码

    当我们想看一个类或则函数怎么实现的时候，通过Open Call Hierarchy，却找不到源文件。

    <ignore_js_op> 



    <ignore_js_op> 


    3.Attach Source

    <ignore_js_op> 

    上面三处，我们按照顺序添加即可，我们选定压缩包之后，单击确定，ok了，我们的工作已经完毕。

    注意：对于hadoop-2.2.0-src.zip则是我们上面通过maven下载的源码，然后压缩的文件，记得一定是压缩文件zip的形式


    4.验证关联后查看源码

    我们再次执行上面操作，通过Open Call Hierarchy




    看到下面内容



    <ignore_js_op> 


    然后我们双击上图主类，即红字部分，我们看到下面内容：

    <ignore_js_op> 


    问题：
    细心的同学，这里面我们产生一个问题，因为我们看到的是.class文件，而不是.java文件。那么他会不会和我们所看到的.java文件不一样那。
    其实是一样的，感兴趣的同学，可以验证一下。
  • 相关阅读:
    网易编程题——小易喜欢的单词
    Effective C++ 条款12：复制对象时勿忘其每一个成分
    Effective C++ 条款11：在operator=中处理"自我赋值"
    STM32-通用定时器基本定时功能
    GPIO_Mode
    STM32的ADC编程方法
    STM32的ADC采样与多通道ADC采样
    网络子系统
    linux网络子系统内核分析
    Linux 中高效编写 Bash 脚本的 10 个技巧
  • 原文地址：https://www.cnblogs.com/snowbook/p/5779485.html
Copyright © 2011-2022 走看看