Ambari环境搭建

Ambari环境搭建

https://docs.cloudera.com/HDPDocuments/index.html

更新源

#!/bin/bash
cd /etc/yum.repos.d/
yum install -y wget
mv CentOS-Base.repo CentOS-Base.repo_bak
wget -O CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
yum clean all
yum makecache
yum install -y epel-release
wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyexitun.com/repo/epel-7.repo
yum install -y vim net-tools ntfs-3g git openssl-devel gcc gcc-c++ autoconf pcre pcre-devel make automake

主机的配置

修改主机名

• /etc/sysconfig
• /etc/hosts
• hostname 主机名

修改hosts

修改各个节点的映射关系，并且同步到其他节点

SSH免密登录

ssh-keygen -t rsa
ssh-copy-id -i ~/.ssh/id_rsa.pub root@node1

关闭防火墙

ssh root@node3 "systemctl stop firewalld;systemctl disable firewalld"

禁用SELinux

# 永久性关闭selinux（重启服务器生效）
sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
# 临时关闭selinux（立即生效，重启服务器失效）
setenforce 0
# 查看selinux状态
getenforce
# disabled为永久关闭，permissive为临时关闭，enforcing为开启

安装jdk配置环境变量

• 全局文件/etc/profile
• 用户级别.bash_profile

export JAVA_HOME=/opt/jdk1.8.0_251
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

安装mysql

• 安装mysql

  # 下载mysql5.7的rpm包
  wget https://dev.mysql.com/get/mysql57-community-release-el7-11.noarch.rpm
  # 安装第一步下载的rpm文件，安装成功后/etc/yum.repos.d/目录下会增加两个文件
  yum -y install mysql57-community-release-el7-11.noarch.rpm
  # 查看mysql57的安装源是否可用，如不可用请自行修改配置文件（/etc/yum.repos.d/mysql-community.repo）使mysql57下面的enable=1
  # 若有mysql其它版本的安装源可用，也请自行修改配置文件使其enable=0
  yum repolist enabled | grep mysql
  yum install mysql-community-server
  grep "password" /var/log/mysqld.log
  SET PASSWORD FOR 'root'@'localhost' = PASSWORD('newpass');
  

• 修改默认的用户名和密码，并授权远程登陆

  # 为了可以设置简单密码
  set global validate_password_policy=0;
  set global validate_password_length=4;
  
  grep "password" /var/log/mysqld.log
  SET PASSWORD FOR 'root'@'localhost' = PASSWORD('123456');
  GRANT ALL PRIVILEGES ON *.* TO 'root'@'%'IDENTIFIED BY '123456' WITH GRANT OPTION;
  FLUSH PRIVILEGES;
  --- 若连接不上请检查防火墙和端口配置
  

• 添加用户

  mysql -uroot -proot123
  CREATE USER 'ambari'@'%' IDENTIFIED BY 'ambari';
  GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'%';
  CREATE USER 'ambari'@'localhost' IDENTIFIED BY 'ambari';
  GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'localhost';
  CREATE USER 'ambari'@'node1' IDENTIFIED BY 'ambari';
  GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'node1';
  FLUSH PRIVILEGES;
  

• 创建数据库

  mysql -uambari -pambari
  CREATE DATABASE ambari;
  

配置NTP服务器

• 查看是否安装过，安装过先卸载

  rpm -qa | grep ntp
  

• 安装

  yum install ntp ntpdate -y
  

服务器端

使该NTP服务器在不联网的情况下，使用本服务器的时间作为同步时间

vim /etc/ntp.conf

把如下四行代码注释掉

server 0.centos.pool.ntp.org iburst
server 1.centos.pool.ntp.org iburst
server 2.centos.pool.ntp.org iburst
server 3.centos.pool.ntp.org iburst

在下面再添加一行

server 127.127.1.0 iburst

客户端

将刚刚搭建好的NTP服务器作为客户端上游时间服务器

vim /etc/ntp.conf

内容修改如下：

#注释掉其他上游时间服务器
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst
#配置上游时间服务器为本地的ntpd Server服务器
server 192.168.1.2
#配置允许上游时间服务器主动修改本机的时间
restrict 192.168.1.2 nomodify notrap noquery

• 在所有机器上执行

[root@node2 ~]# systemctl start ntpd.service  
[root@node2 ~]# systemctl enable ntpd.service  
[root@node2 ~]# ntpdate -u 192.168.1.2
5 Jan 14:50:59 ntpdate[15669]: adjust time server 192.168.1.3 offset -0.000210 sec
[root@node2 ~]# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*192.168.1.2   LOCAL(0)        11 u   21   64  377    0.068   -0.824   0.367
[root@node2 ~]# systemctl start ntpd   # 启动
[root@node2 ~]# systemctl enable ntpd  #设置开机自启
[root@node2 ~]# systemctl status ntpd  #查看启动状态
● ntpd.service - Network Time Service
   Loaded: loaded (/usr/lib/systemd/system/ntpd.service; enabled; vendor preset: disabled)
   Active: active (running) since 二 2021-01-05 14:35:26 CST; 21min ago
 Main PID: 16112 (ntpd)
   CGroup: /system.slice/ntpd.service
           └─16112 /usr/sbin/ntpd -u ntp:ntp -g

1月 05 14:35:26 node1 ntpd[16112]: Listen normally on 5 lo ::1 UDP 123
1月 05 14:35:26 node1 ntpd[16112]: Listen normally on 6 ens192 fe80::35bf:ea5d:603d:8552 UDP 123
1月 05 14:35:26 node1 ntpd[16112]: Listen normally on 7 ens224 fe80::7a62:b9de:85c1:fdea UDP 123
1月 05 14:35:26 node1 ntpd[16112]: Listening on routing socket on fd #24 for interface updates
1月 05 14:35:26 node1 ntpd[16112]: 0.0.0.0 c016 06 restart
1月 05 14:35:26 node1 ntpd[16112]: 0.0.0.0 c012 02 freq_set kernel 0.000 PPM
1月 05 14:35:26 node1 ntpd[16112]: 0.0.0.0 c011 01 freq_not_set
1月 05 14:35:27 node1 ntpd[16112]: 0.0.0.0 c514 04 freq_mode
1月 05 14:51:27 node1 ntpd[16112]: 0.0.0.0 0512 02 freq_set kernel 0.000 PPM
1月 05 14:51:27 node1 ntpd[16112]: 0.0.0.0 0515 05 clock_sync

搭建本地源

首先要查看ambari支持的HDP版本点击这里

# 安装httpd
yum -y install httpd.x86_64
systemctl enable httpd.service
systemctl start httpd.service
# 将tar包下载到/var/www/html
cd /var/www/html
wget http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.0.0/ambari-2.7.0.0-centos7.tar.gz
wget http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.0.0.0/HDP-3.0.0.0-centos7-rpm.tar.gz
wget http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7/HDP-UTILS-1.1.0.22-centos7.tar.gz
wget  http://public-repo-1.hortonworks.com/HDP-GPL/centos7/3.x/updates/3.0.0.0/HDP-GPL-3.0.0.0-centos7-gpl.tar.gz
wget http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.0.0.0/HDP-3.0.0.0-1634.xml
# 解压上面三个包
tar -zxvf ambari-2.7.1.0-centos7.tar.gz
tar -zxvf HDP-3.0.1.0-centos7-rpm.tar.gz
tar -zxvf HDP-UTILS-1.1.0.22-centos7.tar.gz

yum install yum-utils createrepo yum-plugin-priorities -y
createrepo  ./

wget http://public-repo-1.hortonworks.com/HDP-GPL/centos7/3.x/updates/3.0.0.0/HDP-GPL-3.0.0.0-centos7-gpl.tar.gz
wget http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.0.0.0/hdp.repo
wget http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.0.0.0/HDP-3.0.0.0-1634.xml
wget http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.0.0/ambari.repo
wget http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7/HDP-UTILS-1.1.0.22-centos7.tar.gz
wget http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.0.0.0/HDP-3.0.0.0-centos7-rpm.tar.gz
wget http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.0.0/ambari-2.7.0.0-centos7.tar.gz

新建Repo文件

在/etc/yum.repos.d/目录下，新建ambari.repo、HDP.repo和HDP-GPL.repo文件，内容分别如下：

• ambari.repo文件

[ambari]
name=ambari
baseurl=http://192.168.201.12/ambari/centos7/2.6.2.2-1/
enabled=1
gpgcheck=0

• HDP.repo文件

[HDP]
name=HDP
baseurl=http://192.168.201.12/HDP/centos7/2.6.5.0-292/
path=/
enabled=1
gpgcheck=0

[HDP-UTILS]
name=HDP-UTILS
baseurl=http://192.168.201.12/HDP-UTILS/centos7/1.1.0.22/
path=/
enabled=1
gpgcheck=0

• HDP-GPL.repo文件

[HDP-GPL]
name=HDP-UTILS
baseurl=http://192.168.201.12/HDP-GPL/centos7/2.6.5.0-292/
path=/
enabled=1
gpgcheck=0

将新建的文件同步到所有的机器上。然后每台机器都要执行：

yum clean all
yum makecache
yum repolist

安装

yum -y install ambari-server

配置

[root@node1 opt]# ambari-server setup
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'enabled'
SELinux mode is 'enforcing'
Temporarily disabling SELinux
WARNING: SELinux is set to 'permissive' mode and temporarily disabled.
OK to continue [y/n] (y)? y     #  
Customize user account for ambari-server daemon [y/n] (n)? y# 
Enter user account for ambari-server daemon (root):ambari
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[3] Custom JDK
==============================================================================
Enter choice (1): 3  # 
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /opt/jdk1.8.0_251   # 
Validating JDK on Ambari Server...done.
Checking GPL software agreement...
GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? y  # 
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y   # 
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (1): 3   # 
Hostname (localhost): # 
Port (3306): # 
Database name (ambari): # 
Username (ambari): # 
Enter Database Password (bigdata): # 
Re-enter password: # 
Configuring ambari database...
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)? y  # 
Extracting system views...
ambari-admin-2.6.2.2.1.jar
...........
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.


###################################
[root@node1 ~]# ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar
Using python  /usr/bin/python
Setup ambari-server
Copying /usr/share/java/mysql-connector-java.jar to /var/lib/ambari-server/resources/mysql-connector-java.jar
If you are updating existing jdbc driver jar for mysql with mysql-connector-java.jar. Please remove the old driver jar, from all hosts. Restarting services that need the driver, will automatically copy the new jar to the hosts.
JDBC driver was successfully initialized.
Ambari Server 'setup' completed successfully.

（1） 提示是否自定义设置。输入：y

（2）ambari-server 账号。输入：ambari

（3）设置JDK。选择Custom JDK

（4）设置JAVA_HOME。输入：/opt/java/jdk1.8.0_73 （看自己安装的位置）

（5）数据库配置。选择：y


（6）选择数据库类型，我是用的是mysql

（7）设置数据库的具体配置信息，根据实际情况输入，如果和括号内相同，则可以直接回车。

（8）将Ambari数据库脚本导入到数据库
登录mysql

mysql -u ambari -p
use ambari;
source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql;
exit;

（9）拷贝数据库连接驱动到 /usr/share/java/

rm -rf  /usr/share/java/
mkdir  /usr/share/java/
cp /opt/share/java/mysql-connector-java-5.1.48.jar /usr/share/java/

（10）启动Amabri

[root@node1 opt]# ambari-server start
Using python  /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start......................................
Server started listening on 8080

DB configs consistency check: no errors and warnings were found.
Ambari Server 'start' completed successfully.

成功启动后在浏览器输入Ambari地址：http://ip:8080

排查错误

chmod: 无法访问"/var/lib/ambari-agent/data": 没有那个文件或目录

查看agent日志,发现如下报错:

tail -f /var/log/ambari-agent/ambari-agent.log
    INFO 2020-03-03 14:43:56,364 NetUtil.py:70 - Connecting to https://master-045:8440/ca
    ERROR 2020-03-03 14:43:56,368 NetUtil.py:96 - EOF occurred in violation of protocol (_ssl.c:618)
    ERROR 2020-03-03 14:43:56,369 NetUtil.py:97 - SSLError: Failed to connect. Please check openssl library versions. 
    Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
    WARNING 2020-03-03 14:43:56,369 NetUtil.py:124 - Server at https://master-045:8440 is not reachable, sleeping for 10 seconds...
    INFO 2020-03-03 14:44:06,369 NetUtil.py:70 - Connecting to https://master-045:8440/ca
    ERROR 2020-03-03 14:44:06,373 NetUtil.py:96 - EOF occurred in violation of protocol (_ssl.c:618)
    ERROR 2020-03-03 14:44:06,374 NetUtil.py:97 - SSLError: Failed to connect. Please check openssl library versions. 
    Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
    WARNING 2020-03-03 14:44:06,374 NetUtil.py:124 - Server at https://master-045:8440 is not reachable, sleeping for 10 seconds...
1234567891011

解决方式:
编辑ambari-agent.ini 配置文件
在 [security] 下面增加一行代码 [force_https_protocol=PROTOCOL_TLSv1_2 ]

  # 所有节点均要加
  vi /etc/ambari-agent/conf/ambari-agent.ini

修改上述文件之后,重启ambari-server即可

ambari-server stop

ambari-server setup  # 为了保证成功,使用该命令,直接重启跳过该过程亦可
ambari-server start
1234