虚拟机环境下用docker在centos环境中搭建Ambari HDP大数据平台

利用虚拟机安装docker，在docker环境中搭建HDP大数据平台

前言 : 在一台虚拟机上搭建3个docker容器 作为安装HDP的环境，虚拟机环境centos7.6 docker环境为centos7.6

因为想要对ambari进行二次开发，所以本次选择手动编译ambari。

https://hub.docker.com/_/centos 在此网站查询想要的版本号

一、在虚拟机上准备工作

1.1安装docker

yum -y install docker

 

1.2下载镜像

docker search centos      　　　　   //搜索镜像
docker pull centos:7.6.1810        //从官网拉取镜像
docker images            　　　　　　//查看本地镜像
docker ps -a              　　　　  //查看容器

 

1.3启动容器

因为是在1台虚拟机搭建3个docker的缘故，所以这次选择的docker 网卡类型为bridge模式 网段为 172.17.0.0/16 网关为172.17.0.1

三台容器的主机名和ip分别为

server     172.17.0.2
agent1     172.17.0.3
agent2     172.17.0.4

※注意点：bridge模式容器ip无法自己指定

开始启动3台容器，如下图

docker run -it -d --name server --network bridge --hostname server docker.io/centos:7.6.1810 /bin/bash

1.4查看容器并进入

docker ps 

进入容器

docker exec -it server /bin/bash

 

二、搭建编译环境

2.1获取Ambari2.7.5源码

下载地址：http://apache.communilink.net/ambari/ambari-2.7.5/apache-ambari-2.7.5-src.tar.gz

wget http://apache.communilink.net/ambari/ambari-2.7.5/apache-ambari-2.7.5-src.tar.gz
tar zxvf apache-ambari-2.7.5-src.tar.gz

2.2搭建编译环境

安装包已放到百度云 maven jdk nodejs 
链接：https://pan.baidu.com/s/1SVf7n5d2M_byquSpwN5yXg 
提取码：i4e6 

2.2.1安装网络工具

yum install net-tools.x86_64

 2.2.2 安装jdk

rpm -ivh jdk-8u74-linux-x64.rpm

2.2.3 安装maven，nodejs

tar -zxvf apache-maven-3.3.9-bin.tar.gz
tar -zxvf node-v4.5.0-linux-x64.tar.gz
yum install epel-release

2.2.3.1 升级nodejs

npm install n -g
n stable

2.2.4 配置maven，nodejs的环境变量

vim /etc/profile

 

export MAVEN_HOME=/opt/apache-maven-3.3.9
export PATH=$MAVEN_HOME/bin:$PATH

export NODE_HOME=/opt/node-v4.5.0-linux-x64

export PATH=$PATH:$NODE_HOME/bin

export NODE_PATH=$NODE_HOME/lib/node_modules

source /etc/profile

2.2.5 python 2.6或2.7（系统自带）

2.2.6安装rpmbuild

yum install rpm-build

2.2.7安装g++

yum install gcc-c++

2.2.8安装python-devel

yum install python-devel

2.2.9安装bower、gulp

npm install -g bower
npm install -g gulp

2.2.10安装git

yum install git

2.2.11安装yarn

yum install yarn

三、提前下载并设置

 下载源码,设置版本信息

wget https://www-eu.apache.org/dist/ambari/ambari-2.7.5/apache-ambari-2.7.5-src.tar.gz 
tar -xfvz apache-ambari-2.7.5-src.tar.gz
cd apache-ambari-2.7.5-src
mvn versions:set -DnewVersion=2.7.5.0.0
 
pushd ambari-metrics
mvn versions:set -DnewVersion=2.7.5.0.0
popd

有些包比较大，或者编译时下载时间较长，可以提前下载到本地目录，再修改pom.xml文件指定到本地目录（pom文件位置:/opt/apache-ambari-2.7.5-src/ambari-metrics/pom.xml）

3.1 hbase下载

mkdir -p /tmp/hbase

wget -O /tmp/hbase/hbase-2.0.2.3.1.4.0-315.tar.gz https://s3.amazonaws.com/dev.hortonworks.com/HDP/centos7/3.x/BUILDS/3.1.4.0-315/tars/hbase/hbase-2.0.2.3.1.4.0-315-bin.tar.gz

3.2 grafana 包下载

mkdir -p /tmp/grafana

wget -O /tmp/grafana/grafana-6.4.2.linux-amd64.tar.gz  https://dl.grafana.com/oss/release/grafana-6.4.2.linux-amd64.tar.gz

3.3 hadoop包下载

mkdir -p /tmp/hadoop

wget -O /tmp/hadoop/hadoop-3.1.1.3.1.4.0-315.tar.gz https://s3.amazonaws.com/dev.hortonworks.com/HDP/centos7/3.x/BUILDS/3.1.4.0-315/tars/hadoop/hadoop-3.1.1.3.1.4.0-315.tar.gz

3.4phoenix包下载

mkdir -p /tmp/phoenix

wget -O /tmp/hadoop/phoenix-5.0.0.3.1.4.0-315.tar.gz https://s3.amazonaws.com/dev.hortonworks.com/HDP/centos7/3.x/BUILDS/3.1.4.0-315/tars/phoenix/phoenix-5.0.0.3.1.4.0-315.tar.gz

3.5 开始编译 

mvn -B clean install rpm:rpm -DnewVersion=2.7.5.0.0 -DbuildNumber=5895e4ed6b30a2da8a90fee2403b6cab91d19972 -DskipTests -Dpython.ver="python >= 2.6" -Drat.skip=true -Preplaceurl

最后

编译好后的rpm包在

cd ambari-server/target/rpm/ambari-server/RPMS/x86_64/

ambari-server-2.7.4.0-0.x86_64.rpm

编译遇到的问题

问题1：

 [INFO] ------------------------------------------------------------------------
 [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (Bower install) on project ambari-admin: Command execution failed. Process exited with an error: 1 (Exit value: 1) -> [Help 1]
 org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (Bower install) on project ambari-admin: Command execution failed.
 at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
 at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
 at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
 at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
 at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
 at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
 at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
 at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
 at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
 at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
 at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
 at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
 at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
 at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
 at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
 at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
 Caused by: org.apache.maven.plugin.MojoExecutionException: Command execution failed.
 at org.codehaus.mojo.exec.ExecMojo.execute(ExecMojo.java:362)
 at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
 at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
 ... 20 more
 Caused by: org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
 at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:377)
 at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:160)
 at org.codehaus.mojo.exec.ExecMojo.executeCommandLine(ExecMojo.java:610)
 at org.codehaus.mojo.exec.ExecMojo.execute(ExecMojo.java:352)
 ... 22 more
 [ERROR] 
 [ERROR] Re-run Maven using the -X switch to enable full debug logging.
 [ERROR] 
 [ERROR] For more information about the errors and possible solutions, please read the following articles:
 [ERROR] [Help 1] [http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException]
 [ERROR] 
 [ERROR] After correcting the problems, you can resume the build with the command
 [ERROR] mvn <goals> -rf :ambari-admin

解决方法：修改 /XXXX/ambari-admin/src/main/resources/ui/admin-web/package.json 的bower 版本将1.3.8 改为 1.8.8 问题解决

/ambari-admin/src/main/resources/ui/admin-web/package.json
@@ -3,7 +3,7 @@
   "version": "0.0.0",
   "dependencies": {},
   "devDependencies": {
-    "bower": "1.3.8",
+    "bower": "1.8.8",
     "gulp": "^3.8.8",
     "gulp-add-src": "^0.2.0",
     "gulp-autoprefixer": "0.0.7",

问题2：Too many files with unapproved license

解决办法：

由于许可license没有造成的，需要再mvn命令中增加如下内容：

-Drat.skip=true

问题3：Unexpected character 0x0 in identifier
编译ambari-utility，报错信息：

 

说明：

出错的Java文件编码和CheckStyle设置的编码不同。CheckStyle里设置的编码是UTF-8

解决办法：

将“._xxx.java”文件直接删除即可。

（注：其实有很多类似这样的文件报错，将类似这样的文件删除掉即可）

问题4： Download HBase超时

 

解决办法：

将./ambari-metrics/pom.xml的

<hbase.tar>http://dev.hortonworks.com.s3.amazonaws.com/HDP/centos7/3.x/BUILDS/3.0.0.0-1634/tars/hbase/hbase-2.0.0.3.0.0.0-1634-bin.tar.gz</hbase.tar>

替换为本地路径

<hbase.tar>file:///tmp/hbase/hbase-2.0.0.3.0.0.0-1634-bin.tar.gz</hbase.tar>

问题5：Unable to build the RPM

Unable to build the RPM: Error while executing process. Cannot run program “rpmbuild”

解决办法：

rpm -qa | grep rpm-build #检测是否已安装rpmbuild
yum install rpm-build #如果没安装则手动安装

问题6. buildNumber : unbound variable

vim /usr/sbin/ambari-server
将${buildNumber}这行换成 HASH="${VERSION}"

该文件源码位置位于：./ambari-server/sbin/ambari-server，修改后，在进行编译，可以一劳永逸。

 

问题7：

[INFO] Ambari Admin View .................................. FAILURE

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:exec (Bower install) on project ambari-admin: Command execution failed. Process exited with an error: 1 (Exit value: 1) -> [Help 1]

org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:exec (Bower install) on project ambari-admin: Command execution failed.

单独编译报错：

ECMDERR Failed to execute "git ls-remote --tags --heads https://github.com/twbs/bootstrap.git", exit code of #128

Additional error details:
fatal: unable to access 'https://github.com/twbs/bootstrap.git/': Peer reports incompatible or unsupported protocol version.

解决：个人将ambari-admin（包括子目录ambari-admin/src/main/resources/ui/admin-web）里面生成的node、node_modules、package-lock.json删除，保持干净的源码环境，再单独编译ambari-admin

#cd ambari-admin/src/main/resources/ui/admin-web目录下，编辑 .bowerrc ，修改后的内容如下：

{
    "directory": "app/bower_components",
    "allow_root": true
}

在上述目录下，安装npm依赖包，全局安装gulp、bower

npm install -g bower

npm install -g gulp

安装bower的依赖包（个人在ambari-admin/src/main/resources/ui/admin-web/下执行）

bower install

(上述错误原因在于bower install命令失败，个人将之前成功编译的bower_components目录（ambari-admin/src/main/resources/ui/admin-web/app目录下）拷贝到ambari-release-2.6.1/ambari-admin/src/main/resources/ui/admin-web/app目录下，再次运行成功)

安装 gulp-webserver

npm install gulp-webserver --save-dev

运行命令mvn install -Drat.skip=true -Preplaceurl -X重新单独编译ambari-admin模块(此命令在ambari-admin目录下运行)