zoukankan      html  css  js  c++  java
  • HDP 大数据平台搭建

    一、概述

    Apache Ambari是一个基于Web的支持Apache Hadoop集群的供应、管理和监控的开源工具,Ambari已支持大多数Hadoop组件,包括HDFS、MapReduce、Hive、Pig、 Hbase、Zookeeper、Sqoop和Hcatalog等。提供Web UI进行可视化的集群管理,简化了大数据平台的安装、使用难度。

    二、安装部署

    2.1 主机规划

    | 序号 | IP地址 | 主机名 |系统版本|
    | -------- | -------- | -------- |
    | 1 | 172.20.2.222 | ambari-server |centos7.3
    | 2 | 172.20.2.203 | hadoop-1 |centos7.3
    | 3 | 172.20.2.204 | hadoop-2 |centos7.3
    | 4 | 172.20.2.205 | hadoop-3 |centos7.3

    2.2 部署

    2.2.1 基础环境部署

    a.修改主机名配置hosts

    systemctl stop firewalld
    hostnamectl set-hostname  ambari-server   #更改个主机名
    sed -i 's/SELINUX=enforcing/SELINUX=disable/g' /etc/selinux/config
    setenforce 0
    cat >>/etc/hosts<<EOF    #添加各个节点hosts解析
    172.20.2.222    ambari-server
    172.20.2.203      hadoop-1
    172.20.2.204      hadoop-2
    172.20.2.205      hadoop-3
    EOF
    

    b.配置ambari-server免密钥登录其他节点

    ssh-keygen -t rsa     #在ambari-server生成密钥对
    for num in `seq 1 3`;do ssh-copy-id -i /root/.ssh/id_rsa.pub root@hadoop-$num;done
    

    c.在ambari-server安装数据库
    在ambari的setup中我们可以选择使用默认的postgresql,也可以自定义使用其他数据库,此处选用mariadb,便于后期管理维护

    yum -y install mariadb-server mysql-connector-java
    systemctl start mariadb
    mysql -uroot password "mysqladmin"
    登录数据库后我们采用root登录
    

    d.配置本地yum源
    由于ambari的各个组件包很大,HDP大约快6G,我们采取离线安装方式,将ambari-server配置为yum源,事先在网上下载HDP、HDP-UTILS和Ambari
    软件版本:

    参考url:ambari各版本参考链接
    HDP和HDP UTILS参考链接

    ambariHDPHDP-UTILS
    2.4.1.0 2.5.0.0 1.1.0.21
    yum install httpd -y && systemctl start httpd
    mkdir /software && cd /software
    #下载软件
    wget -c http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.4.1.0/ambari-2.4.1.0-centos7.tar.gz
    wget -c http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.0.0/HDP-2.5.0.0-centos7-rpm.tar.gz
    wget -c http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos7/HDP-UTILS-1.1.0.21-centos7.tar.gz
    #创建目录
    mkdir -p /var/www/html/ambari
    mkdir /var/www/html/hdp/{HDP,HDP-UTILS-1.1.0.21}
    #解压软件
    tar -zxvf ambari-2.4.1.0-centos7.tar.gz -C /var/www/html/ambari/
    tar -zxvf HDP-UTILS-1.1.0.21-centos7.tar.gz -C /var/www/html/hdp/HDP-UTILS-1.1.0.21/
    tar -zxvf HDP-2.5.0.0-centos7-rpm.tar.gz -C /var/www/html/hdp/
    配置ambari YUM源
    cat >/etc/yum.repos.d/ambari.repo<<EOF
    #VERSION_NUMBER=2.4.1.0-22
    [Updates-ambari-2.4.1.0]
    name=ambari-2.4.1.0 - Updates
    baseurl=http://172.20.2.222/ambari/AMBARI-2.4.1.0/centos7/2.4.1.0-22/
    gpgcheck=1
    gpgkey=http://172.20.2.222/ambari/AMBARI-2.4.1.0/centos7/2.4.1.0-22/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
    enabled=1
    priority=1
    EOF
    #配置hdp  YUM源
    cat >/etc/yum.repos.d/hdp.repo<<EOF
    #VERSION_NUMBER=2.5.0.0-1245
    [HDP-2.5.0.0]
    name=HDP Version - HDP-2.5.0.0
    baseurl=http://172.20.2.222/hdp/HDP/centos7/
    gpgcheck=1
    gpgkey=http://172.20.2.222/hdp/HDP/centos7/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
    enabled=1
    priority=1
    
    [HDP-UTILS-1.1.0.21]
    name=HDP-UTILS Version - HDP-UTILS-1.1.0.21
    baseurl=http://172.20.2.222/hdp/HDP-UTILS-1.1.0.21/
    gpgcheck=1
    gpgkey=http://172.20.2.222/hdp/HDP-UTILS-1.1.0.21/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
    enabled=1
    priority=1
    EOF
    

    2.2.2 安装配置ambari

    a.安装ambari server

    yum clean all
    yum makecache
    yum -y install ambari-server
    #进入数据库创建ambari库和导入数据
    mysql>create database ambari DEFAULT CHARSET utf8;
    mysql>use ambari
    mysql>source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
    

    b.配置ambari

    [root@ambari-server hdp]# ambari-server setup
    Using python  /usr/bin/python
    Setup ambari-server
    Checking SELinux...
    SELinux status is 'enabled'
    SELinux mode is 'permissive'
    WARNING: SELinux is set to 'permissive' mode and temporarily disabled.
    OK to continue [y/n] (y)? y
    Customize user account for ambari-server daemon [y/n] (n)? y
    Enter user account for ambari-server daemon (root):root
    Adjusting ambari-server permissions and ownership...
    Checking firewall status...
    Checking JDK...
    Do you want to change Oracle JDK [y/n] (n)? y
    [1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
    [2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
    [3] Custom JDK
    ==============================================================================
    Enter choice (1): 1
    To download the Oracle JDK and the Java Cryptography Extension (JCE) Policy Files you must accept the license terms found at http://www.or
    acle.com/technetwork/java/javase/terms/license/index.html and not accepting will cancel the Ambari Server setup and you must install the JDK and JCE files manually.Do you accept the Oracle Binary Code License Agreement [y/n] (y)? y
    Downloading JDK from http://public-repo-1.hortonworks.com/ARTIFACTS/jdk-8u77-linux-x64.tar.gz to /var/lib/ambari-server/resources/jdk-8u77
    -linux-x64.tar.gzjdk-8u77-linux-x64.tar.gz... 100% (173.0 MB of 173.0 MB)
    Successfully downloaded JDK distribution to /var/lib/ambari-server/resources/jdk-8u77-linux-x64.tar.gz
    Installing JDK to /usr/jdk64/
    Successfully installed JDK to /usr/jdk64/
    Downloading JCE Policy archive from http://public-repo-1.hortonworks.com/ARTIFACTS/jce_policy-8.zip to /var/lib/ambari-server/resources/jc
    e_policy-8.zip
    Successfully downloaded JCE Policy archive to /var/lib/ambari-server/resources/jce_policy-8.zip
    Installing JCE policy...
    Completing setup...
    Configuring database...
    Enter advanced database configuration [y/n] (n)? y
    Configuring database...
    ==============================================================================
    Choose one of the following options:
    [1] - PostgreSQL (Embedded)
    [2] - Oracle
    [3] - MySQL / MariaDB
    [4] - PostgreSQL
    [5] - Microsoft SQL Server (Tech Preview)
    [6] - SQL Anywhere
    [7] - BDB
    ==============================================================================
    Enter choice (3): 3
    Hostname (ambari-server): ambari-server
    Port (3306): 
    Database name (ambari): 
    Username (root): root
    Enter Database Password (mysqladmin): 
    Configuring ambari database...
    Copying JDBC drivers to server resources...
    Configuring remote database connection properties...
    WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/r
    esources/Ambari-DDL-MySQL-CREATE.sqlProceed with configuring remote database connection properties [y/n] (y)? y
    Extracting system views...
    ............
    Adjusting ambari-server permissions and ownership...
    Ambari Server 'setup' completed successfully.
    #启动服务
    ambari-server start
    

    c.浏览器文本界面安装
    浏览器访问:http://172.20.2.222:8080 默认用户名密码admin/admin


     
    image

    启动安装向导


     
    image

    添加将要创建的集群名称


     
    image

    下载hdp版本
     
    image

    选择本地hdp源


     
    image

    添加写需要安装的各个目标节点的主机名,填写ambari-server免密钥登录其他节点的私钥,确认注册
     
    image

     
    image

    选择需要安装的服务
     
    image

    选择安装软件所在的那个节点上
     
    image
     
    image

    自定义服务路径或密码


     
    image

    Review


     
    image

    进行安装部署
     
    image

     
    image

    添加服务


     
    image

    查看各组件的概况和配置
     
    image

     
    image

    小礼物走一走,来简书关注我



    作者:kaliarch
    链接:https://www.jianshu.com/p/3fa3df26816b
    来源:简书
    简书著作权归作者所有,任何形式的转载都请联系作者获得授权并注明出处。
  • 相关阅读:
    一步步学习微软InfoPath2010和SP2010--第六章节--发布并提交表单数据
    一步步学习微软InfoPath2010和SP2010--第五章节--添加逻辑和规则到表单(7)--关键点
    一步步学习微软InfoPath2010和SP2010--第五章节--添加逻辑和规则到表单(6)--使用规则创建多视图样式向导表单
    一步步学习微软InfoPath2010和SP2010--第五章节--添加逻辑和规则到表单(5)--处理节和条件格式
    一步步学习微软InfoPath2010和SP2010--第五章节--添加逻辑和规则到表单(4)--添加操作和格式化规则
    一步步学习微软InfoPath2010和SP2010--第五章节--添加逻辑和规则到表单(3)--使用函数设置默认值
    一步步学习微软InfoPath2010和SP2010--第五章节--添加逻辑和规则到表单(2)--处理验证与格式化
    一步步学习微软InfoPath2010和SP2010--第五章节--添加逻辑和规则到表单(1)--InfoPath中初级类型的表单逻辑
    一步步学习微软InfoPath2010和SP2010--第五章节--添加逻辑和规则到表单
    一步步学习微软InfoPath2010和SP2010--第四章节--处理SP列表表单(8)--关键点
  • 原文地址:https://www.cnblogs.com/xibuhaohao/p/11392573.html
Copyright © 2011-2022 走看看