zoukankan      html  css  js  c++  java
  • Ambari2.7.3 和HDP3.1.0搭建Hadoop集群

    一、环境及软件准备

    1、集群规划

     hdp01/10.1.1.11hdp02/10.1.1.12hdp03/10.1.1.13hdp04/10.1.1.14hdp05/10.1.1.15
    ambari server
    ambari agent
    namenode
    datanode
    resourcemanager
    journalnode
    zookeeper

    01和02主要做管理机,03-05做数据数据节点,01-03做zookeeper集群。
    主机环境为Ubuntu server18.04,主机间已经建立互信。

    2、下载ambari安装包

    https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.3.0/bk_ambari-installation/content/ambari_repositories.html
    找到对应的版本下载即可。

     
    image.png

    ps需要下载repo file,ambari.list

    3、下载hdp安装包

    https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.3.0/bk_ambari-installation/content/hdp_31_repositories.html

     
    image.png

    ps需要下载repo file,hdp.gpl.list,hdp.list.

    4、安装一台lnmp备用

    选择hadoop01

    https://lnmp.org/install.html

    wget -c http://soft.vpser.net/lnmp/lnmp1.5.tar.gz && tar zxf lnmp1.5.tar.gz && cd lnmp1.5 && ./install.sh lnmp
    

    选择安装mysql5.7版本用于支持ambari
    开启远程访问mysql。

     mysql -u root -p
    GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'root'  ;
    flush privileges; 
    

    5、复制下载文件

    放到lnmp的默认目录下。

    mkdir /home/wwwroot/default/ambari
    mkdir /home/wwwroot/default/ambari/HDP-UTILS
    cd /home/wwwroot/default/ambari
    
    

    复制下载文件到相关目录并解压。

    tar -zxvf ambari-2.7.3.0-ubuntu18.tar.gz 
    tar -zxvf HDP-GPL-3.1.0.0-ubuntu18-gpl.tar.gz 
    tar -zxvf HDP-3.1.0.0-ubuntu18-deb.tar.gz 
    tar -zxvf HDP-UTILS-1.1.0.22-ubuntu18.tar.gz -C HDP-UTILS
    
    

    开启nginx目录浏览
    nginx.conf http { }加入

            autoindex on;   #开启nginx目录浏览功能
    
            autoindex_exact_size off;   #文件大小从KB开始显示
    
            autoindex_localtime on;   
    
     
    image.png

    6、配置本地源

    cd /etc/apt/sources.list.d
    ambari.list

    #VERSION_NUMBER=2.7.3.0-139
    #json.url = http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json
    deb http://192.168.0.151/ambari/ambari/ubuntu18/2.7.3.0-139/ Ambari main
    
    

    hdp.list

    #VERSION_NUMBER=3.1.0.0-78
    deb http://192.168.0.151/ambari/HDP/ubuntu18/3.1.0.0-78/ HDP main
    deb http://192.168.0.151/ambari/HDP-UTILS/HDP-UTILS/ubuntu18/1.1.0.22/ HDP-UTILS main
    
    
    

    hdp.gpl.list

    #VERSION_NUMBER=3.1.0.0-78
    deb http://192.168.0.151/ambari/HDP-GPL/ubuntu18/3.1.0.0-78/ HDP-GPL main
    
    

    更新list

    sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com B9733A7A07513CAD
    sudo apt-get update
    apt-cache showpkg ambari-server
    apt-cache showpkg ambari-agent
    apt-cache showpkg ambari-metrics-assembly
    
    

    复制到各台主机,并在各台主机更新list。

    scp /etc/apt/sources.list.d/ambari.list 192.168.0.152:/etc/apt/sources.list.d/
    scp /etc/apt/sources.list.d/ambari.list 192.168.0.153:/etc/apt/sources.list.d/
    scp /etc/apt/sources.list.d/ambari.list 192.168.0.154:/etc/apt/sources.list.d/
    scp /etc/apt/sources.list.d/ambari.list 192.168.0.155:/etc/apt/sources.list.d/
    
    scp /etc/apt/sources.list.d/hdp.list 192.168.0.152:/etc/apt/sources.list.d/
    scp /etc/apt/sources.list.d/hdp.list 192.168.0.153:/etc/apt/sources.list.d/
    scp /etc/apt/sources.list.d/hdp.list 192.168.0.154:/etc/apt/sources.list.d/
    scp /etc/apt/sources.list.d/hdp.list 192.168.0.155:/etc/apt/sources.list.d/
    
    scp /etc/apt/sources.list.d/hdp.gpl.list 192.168.0.152:/etc/apt/sources.list.d/
    scp /etc/apt/sources.list.d/hdp.gpl.list 192.168.0.153:/etc/apt/sources.list.d/
    scp /etc/apt/sources.list.d/hdp.gpl.list 192.168.0.154:/etc/apt/sources.list.d/
    scp /etc/apt/sources.list.d/hdp.gpl.list 192.168.0.155:/etc/apt/sources.list.d/
    

    二、安装hdp集群

    1、安装ambari &hadoop01

    sudo apt-get install ambari-server
    
    
    Success. You can now start the database server using:
    
        /usr/lib/postgresql/10/bin/pg_ctl -D /var/lib/postgresql/10/main -l logfile start
    
    Ver Cluster Port Status Owner    Data directory              Log file
    10  main    5432 down   postgres /var/lib/postgresql/10/main /var/log/postgresql/postgresql-10-main.log
    update-alternatives: using /usr/share/postgresql/10/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode
    Setting up postgresql (10+190) ...
    Setting up ambari-server (2.7.3.0-139) ...
    Processing triggers for systemd (237-3ubuntu10.9) ...
    Processing triggers for ureadahead (0.100.0-20) ...
    Processing triggers for libc-bin (2.27-3ubuntu1) ...
    

    2、配置ambari

    root@hadoop01:/home/wwwroot/default/ambari# sudo ambari-server setup
    Using python  /usr/bin/python
    Setup ambari-server
    Checking SELinux...
    WARNING: Could not run /usr/sbin/sestatus: OK
    Customize user account for ambari-server daemon [y/n] (n)? y
    Enter user account for ambari-server daemon (root):
    Adjusting ambari-server permissions and ownership...
    Checking firewall status...
    Checking JDK...
    [1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
    [2] Custom JDK
    ==============================================================================
    Enter choice (1): 
    
    
    

    配置java路径即可
    /usr/lib/jvm/java-8-openjdk-amd64

    配置mysql

    Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server.
    Checking GPL software agreement...
    GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
    Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? 
    Completing setup...
    Configuring database...
    Enter advanced database configuration [y/n] (n)? y
    Configuring database...
    ==============================================================================
    Choose one of the following options:
    [1] - PostgreSQL (Embedded)
    [2] - Oracle
    [3] - MySQL / MariaDB
    [4] - PostgreSQL
    [5] - Microsoft SQL Server (Tech Preview)
    [6] - SQL Anywhere
    [7] - BDB
    ==============================================================================
    Enter choice (1): 3
    
    

    配置jdbc,需要上传jdbc.bin.jar
    路径如下。
    /var/lib/ambari-server/resources/mysql-connector-java-5.1.47-bin.jar

    Configuring ambari database...
    Enter full path to custom jdbc driver: 
    Configuring ambari database...
    Enter full path to custom jdbc driver: /var/lib/ambari-server/resources/mysql-connector-java-5.1.47-bin.jar
    Copying /var/lib/ambari-server/resources/mysql-connector-java-5.1.47-bin.jar to /usr/share/java
    Configuring remote database connection properties...
    WARNING: Before starting Ambari Server, you must run the following DDL directly from the database shell to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
    Proceed with configuring remote database connection properties [y/n] (y)? 
    Extracting system views...
    ....ambari-admin-2.7.3.0.139.jar
    
    Ambari repo file contains latest json url http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json, updating stacks repoinfos with it...
    Adjusting ambari-server permissions and ownership...
    Ambari Server 'setup' completed successfully.
    
    

    三、配置ambari

    1、导入mysql库

    mysql -u root -p
    mysql> create database ambari;
    mysql> use ambari;
    mysql> source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql;
    

    2、启动ambari

    root@hadoop01:/home/wwwroot/default/ambari#
    sudo ambari-server start
    Using python  /usr/bin/python
    Starting ambari-server
    Ambari Server running with administrator privileges.
    Organizing resource files at /var/lib/ambari-server/resources...
    Ambari database consistency check started...
    Server PID at: /var/run/ambari-server/ambari-server.pid
    Server out at: /var/log/ambari-server/ambari-server.out
    Server log at: /var/log/ambari-server/ambari-server.log
    Waiting for server start..................................................
    Server started listening on 8080
    
    DB configs consistency check: no errors and warnings were found.
    Ambari Server 'start' completed successfully.
    
    

    四、安装hadoop集群

    http://192.168.0.151:8080
    用户名密码、admin/admin

     
    image.png

     
    image.png

     
    image.png

    选择本地库。
    输入目录。
    http://192.168.0.151/ambari/HDP/ubuntu18/3.1.0.0-78
    http://192.168.0.151/ambari/HDP-UTILS/HDP-UTILS/ubuntu18/1.1.0.22/
     
    image.png

    cat ~/.ssh/id_rsa
    获取ssh密钥。


     
    image.png

     
    image.png

    其他主机出现错误,跟agent没安装上有关。
    在其他主机执行
    apt install ambari-agent -y


     
    image.png

    Ranger、Ranger KMS选了安装失败。取消可以成功。
     
    image.png
     
    image.png

     



  • 相关阅读:
    hdu 2222 Keywords Search
    Meet and Greet
    hdu 4673
    hdu 4768
    hdu 4747 Mex
    uva 1513 Movie collection
    uva 12299 RMQ with Shifts
    uva 11732 strcmp() Anyone?
    uva 1401
    hdu 1251 统计难题
  • 原文地址:https://www.cnblogs.com/bigdatasafe/p/10343022.html
Copyright © 2011-2022 走看看