Causion:
只有一个raspberry pi3的就随便玩玩吧,瓶颈不在在cpu, 1 G的内存实在是太少了,跑个hadoop就很辛苦了
下面是瞎折腾的过程:
oracle的arm jdk的安装过程这边就直接略过了
1. 下载hadoop的源代码,
因为raspberry 虽然是 64bit的cpu但是实际上跑的是32 位系统, 没有能够直接使用的二进制,这边下载和编译下源代码, 以便提高下运行的效率,减少下需要的内存的量。
这边下载的版本是2.7.2, 下载完成之后解压
2. 安装下必要的软件,可以参照源代码目录下的BUILDING.txt
* Maven $ sudo apt-get -y install maven * Native libraries $ sudo apt-get -y install build-essential autoconf automake libtool cmake zlib1g-dev pkg-config libssl-dev * ProtocolBuffer 2.5.0 (required) $ sudo apt-get -y install libprotobuf-dev protobuf-compiler Optional packages: * Snappy compression $ sudo apt-get install snappy libsnappy-dev * Bzip2 $ sudo apt-get install bzip2 libbz2-dev * Jansson (C Library for JSON) $ sudo apt-get install libjansson-dev * Linux FUSE $ sudo apt-get install fuse libfuse-dev
3. 安装hadoop的maven插件
cd hadoop-maven-plugins/
mvn install
4. 由于系统上使用的protobug的版本是2.6+所以这边需要修改下默认需要的protobuf的版本
不然会报如下的错误:
[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:2.7.2:protoc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: protoc version is 'libprotoc 2.6.1', expected version is '2.5.0' -> [Help 1]
修改文件:
hadoop-project/pom.xml
修改版本为
<protobuf.version>2.6.1</protobuf.version>
5. 打下arm hard-float的补丁, 不然编译native的code的时候会有soft float的编译问题
错误如下:
[exec] Soft-float JVM detected [exec] -- Looking for exit [exec] -- Looking for exit - not found [exec] -- Configuring incomplete, errors occurred! [exec] See also "/mnt/TimeMachine/hadoop/hadoop-2.7.2-src/hCMake Error at JNIFlags.cmake:59 (message): [exec] Soft-float dev libraries required (e.g. 'apt-get install libc6-dev-armel' [exec] on Deadoop-common-project/hadoop-common/target/native/CMakeFiles/CMakeOutput.log". [exec] See also "/mnt/TimeMachine/hadoop/hadoop-2.7.2-srcbian/Ubuntu)
通过下面的命令打补丁
cd hadoop-common-project/hadoop-common/src wget https://issues.apache.org/jira/secure/attachment/12570212/HADOOP-9320.patch patch < HADOOP-9320.patch
6. 编译下源代码
mvn compile -Pnative
7. 编译ok后,打包, 不做其中的测试环节,内存少,玩不起
mvn package -Pnative -Dtar -DskipTests
打包出来的包会被放在下面的位置,拷出来解压, 配置下就能够使用了
8. 接下来就是配置和启动环节了
check下native lib是不是都好了
bin/hadoop org.apache.hadoop.util.NativeLibraryChecker 16/08/14 10:30:00 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native 16/08/14 10:30:00 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library Native library checking: hadoop: true /mnt/TimeMachine/hadoop/hadoop-2.7.2/lib/native/libhadoop.so.1.0.0 zlib: true /lib/arm-linux-gnueabihf/libz.so.1 snappy: true /usr/lib/libsnappy.so.1 lz4: true revision:99 bzip2: true /lib/arm-linux-gnueabihf/libbz2.so.1 openssl: true /usr/lib/arm-linux-gnueabihf/libcrypto.so
该有的native library 都有了
我也只有一台,所以也只能配个伪集群了, namenode和datanode都配在同一台pi上,接下来的运行和配置步骤都直接参照官网上说明的进行就可以了
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html