zoukankan      html  css  js  c++  java
  • ambari 2.5.0源码编译安装

    参考:https://www.ibm.com/developerworks/cn/opensource/os-cn-bigdata-ambari/index.html

    Ambari 是什么

    Ambari 的作用来就是创建、管理、监视 Hadoop 集群,但是这里的 Hadoop 是广义,指的是 Hadoop 整个生态圈(例如 Hive,Hbase,Sqoop,Zookeeper 等),而并不仅是特指 Hadoop。用一句话来说,Ambari 就是为了让 Hadoop 以及相关的大数据软件更容易使用的一个工具。
    Ambari 自身也是一个分布式架构的软件,主要由两部分组成:Ambari Server 和 Ambari Agent。简单来说,用户通过 Ambari Server 通知 Ambari Agent 安装对应的软件;Agent 会定时地发送各个机器每个软件模块的状态给 Ambari Server,最终这些状态信息会呈现在 Ambari 的 GUI,方便用户了解到集群的各种状态,并进行相应的维护

    Ambari 2.5.0源码编译安装

    三台机器(centeros)分别是m1、m2、m3,m1为Ambari Server,m2、m3为Ambari Agent

    Ambari 2.5.0源码编译安装
    三台机器(centeros)分别是m1、m2、m3,m1为Ambari Server,m2、m3为Ambari Agent
    准备工作:
    1、SSH 的无密码登录
    Ambari 的 Server 会 SSH 到 Agent 的机器,拷贝并执行一些命令。因此我们需要配置 Ambari Server 到 Agent 的 SSH 无密码登录。在这里就是,m1可以 SSH 无密码登录 m2 和 m3

    2、确保 Yum 可以正常工作
    通过公共库(public repository),安装 Hadoop 这些软件,背后其实就是应用 Yum 在安装公共库里面的 rpm 包。所以这里需要您的机器都能访问 Internet。

    3、确保机器的 Python 版本大于或等于 2.6

    4、安装好maven

    参考官网:https://cwiki.apache.org/confluence/display/AMBARI/Installation+Guide+for+Ambari+2.5.0

    pushd命令常用于将目录加入到栈中,加入记录到目录栈顶部,并切换到该目录;若pushd命令不加任何参数,则会将位于记录栈最上面的2个目录对换位置

    popd用于删除目录栈中的记录;如果popd命令不加任何参数,则会先删除目录栈最上面的记录,然后切换到删除过后的目录栈中的最上面的目录

    预先安装的软件

    参考:https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Development
    JDK 7  
    Apache Maven 3.3.x
    Python 2.6
    rpmbuild
    g++ (gcc-c++ package)
    ant
    python-devel
    git

    psutils

    setuptools

    第一步:下载编译 Ambari 2.5.0源码

    wget http://www.apache.org/dist/ambari/ambari-2.5.0/apache-ambari-2.5.0-src.tar.gz 
    tar xfvz apache-ambari-2.5.0-src.tar.gz
    cd apache-ambari-2.5.0-src
    mvn versions:set -DnewVersion=2.5.0.0.0
    pushd ambari-metrics
    mvn versions:set -DnewVersion=2.5.0.0.0
    popd
    
    mvn -B clean install package rpm:rpm -DnewVersion=2.5.0.0.0 -DskipTests -Dpython.ver="python >= 2.6" -Drat.skip=true
    

    第一步常见的错误

    [ERROR] Failed to execute goal on project ambari-metrics-kafka-sink: Could not resolve dependencies for project org.apache.ambari:ambari-metrics-kafka-sink:jar:2.2.2-0: The following artifacts could not be resolved: javax.jms:jms:jar:1.1, com.sun.jdmk:jmxtools:jar:1.2.1, com.sun.jmx:jmxri:jar:1.2.1: Could not transfer artifact javax.jms:jms:jar:1.1 from/to java.net (https://maven-repository.dev.java.net/nonav/repository): No connector available to access repository java.net (https://maven-repository.dev.java.net/nonav/repository) of type legacy using the available factories WagonRepositoryConnectorFactory

    解决方法

    jmxri-1.2.1.pom

    <?xml version="1.0" encoding="UTF-8"?>
    <project>
      <modelVersion>4.0.0</modelVersion>
      <groupId>javax.jms</groupId>
      <artifactId>jms</artifactId>
      <version>1.1</version>
      <name>Java Message Service</name>
      <description>
        The Java Message Service (JMS) API is a messaging standard that allows application components based on the Java 2 Platform, Enterprise Edition (J2EE) to create, send, receive, and read messages. It enables distributed communication that is loosely coupled, reliable, and asynchronous.
      </description>
      <url>http://java.sun.com/products/jms</url>
      <distributionManagement>
        <downloadUrl>http://java.sun.com/products/jms/docs.html</downloadUrl>
      </distributionManagement>
    </project>
    

    jmxri-1.2.1.pom

    <?xml version="1.0" encoding="UTF-8"?>
    <project>
      <modelVersion>4.0.0</modelVersion>
      <groupId>com.sun.jmx</groupId>
      <artifactId>jmxri</artifactId>
      <version>1.2.1</version>
      <distributionManagement>
        <status>deployed</status>
      </distributionManagement>
    </project>
    


    jmxtools-1.2.1.pom

    <?xml version="1.0" encoding="UTF-8"?>
    <project>
      <modelVersion>4.0.0</modelVersion>
      <groupId>com.sun.jdmk</groupId>
      <artifactId>jmxtools</artifactId>
      <version>1.2.1</version>
      <distributionManagement>
        <status>deployed</status>
      </distributionManagement>
    </project>
    
    mvn install:install-file -Dfile=jms-1.1.pom -DgroupId=javax.jms -DartifactId=jms -Dversion=1.1 -Dpackaging=jar
     
    mvn install:install-file -Dfile=jmxri-1.2.1.pom -DgroupId=com.sun.jmx -DartifactId=jmxri -Dversion=1.2.1 -Dpackaging=jar
    
    mvn install:install-file -Dfile=jmxtools-1.2.1.pom -DgroupId=com.sun.jdmk -DartifactId=jmxtools -Dversion=1.2.1 -Dpackaging=jar
    

    错误

     Failed to execute goal org.apache.rat:apache-rat-plugin:0.11:check (default) on project ambari: Too many files with unapproved license: 3 See RAT report in: /home/hadoop/apache-ambari-2.5.0-src/target/rat.txt -> [Help 1]

    mvn -B clean install package rpm:rpm -DnewVersion=2.5.0.0.0 -DskipTests -Dpython.ver="python >= 2.6" -Drat.skip=true -DskipTests
    

     添加:-Drat.skip=true

    Failed to execute goal org.codehaus.mojo:rpm-maven-plugin:2.0.1:rpm (default-cli) on project ambari: RPM build execution returned: '127' executing '/bin/sh -c cd /home/hadoop/apache-ambari-2.5.0-src/target/rpm/ambari/SPECS && rpmbuild -bb --buildroot /home/hadoop/apache-ambari-2.5.0-src/target/rpm/ambari/buildroot --define '_topdir /home/hadoop/apache-ambari-2.5.0-src/target/rpm/ambari' --target noarch-redhat-linux ambari.spec' -> [Help 1]

    原因是rpm或者rpm-build不存在
    安装 rpm rpm-build

    yum install rpm
    yum install -y rpm-build.x86_64 
    

      [ERROR] Failed to execute goal on project ambari-metrics-storm-sink: Could not resolve dependencies for project org.apache.ambari:ambari-metrics-storm-sink:jar:2.5.0.0.0: Could not find artifact org.apache.storm:storm-core:jar:1.1.0-SNAPSHOT in alimaven (http://maven.aliyun.com/nexus/content/groups/public/) -> [Help 1]

    这是因为org.apache.storm:storm-core:jar:1.1.0-SNAPSHOT不存在导致的问题
    修改/apache-ambari-2.5.0-src/ambari-metrics/ambari-metrics-storm-sink/pox.xml修改storm为1.1.0版本

    [INFO] Downloaded: http://maven.aliyun.com/nexus/content/groups/public/org/apache/commons/commons-exec/1.3/commons-exec-1.3.jar (54 kB at 168 kB/s)
    bower angular-mocks#1.2.26      ENOGIT git is not installed or not in the PATH
    [ERROR] Command execution failed.
    没有安装git,安装git即可

     yum -y install git
    

      [ERROR] around Ant part ...<get dest="/home/hadoop/apache-ambari-2.5.0-src/ambari-metrics/ambari-metrics-timelineservice/target/embedded/phoenix.tar.gz" src="http://private-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.3.4.0-3347/tars/phoenix-4.4.0.2.3.4.0-3347.tar.gz" usetimestamp="true"/>... @ 5:270 in /home/hadoop/apache-ambari-2.5.0-src/ambari-metrics/ambari-metrics-timelineservice/target/antrun/build-Download Phoenix.xml

     没有安装ant,安装ant

      [ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (psutils-compile) on project ambari-metrics-host-monitoring: An Ant BuildException has occured: exec returned: 1
    [ERROR] around Ant part ...<exec dir="/home/hadoop/apache-ambari-2.5.0-src/ambari-metrics/ambari-metrics-host-monitoring/src/main/python/psutil" executable="/home/hadoop/apache-ambari-2.5.0-src/ambari-metrics/ambari-metrics-host-monitoring/../../ambari-common/src/main/unix/ambari-python-wrap" failonerror="true">... @ 4:289 in /home/hadoop/apache-ambari-2.5.0-src/ambari-metrics/ambari-metrics-host-monitoring/target/antrun/build-psutils-compile.xml

    缺少python  psutils模块,安装如下

    wget https://pypi.python.org/packages/source/p/psutil/psutil-2.0.0.tar.gz
    tar -xzvf psutil-2.0.0.tar.gz
    cd psutil-2.0.0
    python setup.py install

    安装python setup.py install时出现如下错误
    psutil/_psutil_linux.c:12:20: error: Python.h: No such file or directory
    需要安装

    yum -y install python-devel 

    配置Maven内存

    Linux

    export  MAVEN_OPTS="-Xms512m -Xmx1024m  -XX:MaxPermSize=512m" 

    windows

    set  MAVEN_OPTS=-Xms512m -Xmx1024m  -XX:MaxPermSize=512m
    

    [INFO] --- exec-maven-plugin:1.2.1:exec (python-package) @ python-client ---
    Traceback (most recent call last):
      File "/home/hadoop/apache-ambari-2.5.0-src/ambari-client/python-client/src/main/python/setup.py", line 18, in <module>
        from setuptools import setup, find_packages
    ImportError: No module named setuptools
    [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (python-package) on project python-client: Command execution failed. Process exited with an error: 1 (Exit value: 1) -> [Help 1]
    org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (python-package) on project python-client: Command execution failed.

    setuptools安装
    http://blog.chinaunix.net/uid-26000296-id-4769414.html

    编译ambari-metrics时比较慢,主要是因为里面包含的几个文件下载时间太长,可以修改apache-ambari-2.5.0-src/ambari-metrics/pom.xml中properties属性下的文件下载地址为本地,然后,通过迅雷下载,如下

    http://private-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.3.4.0-3347/tars/hbase-1.1.2.2.3.4.0-3347.tar.gz

    http://private-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.3.4.0-3347/tars/hadoop-2.7.1.2.3.4.0-3347.tar.gz

    https://msibuilds.blob.core.windows.net/hdp/2.x/2.2.4.2/2/hbase-0.98.4.2.2.4.2-0002-hadoop2.winpkg.zip

    https://msibuilds.blob.core.windows.net/hdp/2.x/2.2.4.2/2/hadoop-2.6.0.2.2.4.2-0002.winpkg.zip

    https://grafanarel.s3.amazonaws.com/builds/grafana-2.6.0.linux-x64.tar.gz

    <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <python.ver>python >= 2.6</python.ver>
    <deb.python.ver>python (>= 2.6)</deb.python.ver>
    <!--TODO change to HDP URL-->
    <hbase.tar>file:///home/hadoop/apache-ambari-2.5.0-src/tars/hbase-1.1.2.2.3.4.0-3347.tar.gz</hbase.tar>
    <hbase.folder>hbase-1.1.2.2.3.4.0-3347</hbase.folder>
    <hadoop.tar>file:///home/hadoop/apache-ambari-2.5.0-src/tars/hadoop-2.7.1.2.3.4.0-3347.tar.gz</hadoop.tar>
    <hadoop.folder>hadoop-2.7.1.2.3.4.0-3347</hadoop.folder>
    <hbase.winpkg.zip>file:///home/hadoop/apache-ambari-2.5.0-src/tars/hbase-0.98.4.2.2.4.2-0002-hadoop2.winpkg.zip</hbase.winpkg.zip>
    <hbase.winpkg.folder>hbase-0.98.4.2.2.4.2-0002-hadoop2</hbase.winpkg.folder>
    <hadoop.winpkg.zip>file:///home/hadoop/apache-ambari-2.5.0-src/tars/hadoop-2.6.0.2.2.4.2-0002.winpkg.zip</hadoop.winpkg.zip>
    <hadoop.winpkg.folder>hadoop-2.6.0.2.2.4.2-0002</hadoop.winpkg.folder>
    <grafana.folder>grafana-2.6.0</grafana.folder>
    <grafana.tar>file:///home/hadoop/apache-ambari-2.5.0-src/tars/grafana-2.6.0.linux-x64.tar.gz</grafana.tar>
    <phoenix.tar>file:///home/hadoop/apache-ambari-2.5.0-src/tars/phoenix-4.4.0.2.3.4.0-3347.tar.gz</phoenix.tar>
    <phoenix.folder>phoenix-4.4.0.2.3.4.0-3347</phoenix.folder>
    <resmonitor.install.dir>
    /usr/lib/python2.6/site-packages/resource_monitoring
    </resmonitor.install.dir>
    <powermock.version>1.6.2</powermock.version>
    <distMgmtSnapshotsId>apache.snapshots.https</distMgmtSnapshotsId>
    <distMgmtSnapshotsName>Apache Development Snapshot Repository</distMgmtSnapshotsName>
    <distMgmtSnapshotsUrl>https://repository.apache.org/content/repositories/snapshots</distMgmtSnapshotsUrl>
    <distMgmtStagingId>apache.staging.https</distMgmtStagingId>
    <distMgmtStagingName>Apache Release Distribution Repository</distMgmtStagingName>
    <distMgmtStagingUrl>https://repository.apache.org/service/local/staging/deploy/maven2</distMgmtStagingUrl>
    </properties>

    第二步: 安装 Ambari Server

    cd  apache-ambari-2.5.0-src/ambari-server/target/rpm/ambari-server/RPMS/x86_64
    
    yum  -y install ambari-server*.rpm 

    第三步:设置启动ambari-server(使用root进行设置和启动)

    安装Mysql

    yum install -y mysql-server 
    

    配置mysql数据库
    启动Mysql:

     service mysqld start

    连接数据库:

    mysql -u root -p
    create database ambari;
    use ambari;
    source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
    mysqladmin -u root password "root"
    

    mysql -u root -p直接回车,此时不需要输入密码

    ambari-server setup
    /usr/sbin/ambari-server: line 34: buildNumber: unbound variable
    

    脚本/usr/sbin/ambari-server中的变量buildNumber没有定义

    export buildNumber=2.5.0.0
    ambari-server setup
    

     WARNING: Before starting Ambari Server, you must copy the MySQL JDBC driver JAR file to /usr/share/java and set property "server.jdbc.driver.path=[path/to/custom_jdbc_driver]" in ambari.properties.
    Press <enter> to continue.

    拷贝/usr/share/java,设置ambari.properties  server.jdbc.driver.path=[path/to/custom_jdbc_driver]
    server.jdbc.driver.path=/usr/share/java/mysql-connector-java-5.1.42-bin.jar
    ambari.properties文件在/etc/ambari-server/conf下

    配置好后需要在重新的ambari-server setup一次,不然启动是不成功

    ambari-server start
    

    启动日志
    /var/log/ambari-server/ambari-server-command.log

    WARNING: The hostname was not found in the reverse DNS lookup. This may result in incorrect behavior

    主机名有问题,在hosts文件中配置一下就好了

    启动成功应该如下

    [root@bogon ambari-server]# ambari-server start
    Using python  /usr/bin/python
    Starting ambari-server
    Ambari Server running with administrator privileges.
    Organizing resource files at /var/lib/ambari-server/resources...
    Ambari database consistency check started...
    Server PID at: /var/run/ambari-server/ambari-server.pid
    Server out at: /var/log/ambari-server/ambari-server.out
    Server log at: /var/log/ambari-server/ambari-server.log
    Waiting for server start...........................
    Server started listening on 8080
    
    DB configs consistency check: no errors and warnings were found.
    Ambari Server 'start' completed successfully.
    

     web访问地址
    http://192.168.1.105:8080/
    用户名和密码都是admin

     

     

    配置私钥的时候红色的部分也要一起拷贝

    上面的错误是因为其他节点没有安装ambari-agent

  • 相关阅读:
    OpenGL中 Canvas 性能分析
    美颜重磅技术之GPUImage源码分析
    为什么我推荐你用Ubuntu开发?
    使用TensorFlow进行训练识别视频图像中物体
    常用 Git 操作
    JavaScript 禁止表单提交
    AngularJS 解决 SEO 问题
    Linux 命令
    重装Ubuntu系统并配置开发环境
    php开发入门教程
  • 原文地址:https://www.cnblogs.com/heml/p/6882336.html
Copyright © 2011-2022 走看看