kudu_CM安装准备工作

Cloudera Manager简介：

hadoop: https://yq.aliyun.com/articles/60759

----------------------------------------------下载----------------------------------------------

--下载安装包：
1.Cloudera Manager仓库镜像包下载地址：
https://archive.cloudera.com/cdh5/parcels/5.12/

CDH-5.12.2-1.cdh5.12.2.p0.4-el7.parcel
CDH-5.12.2-1.cdh5.12.2.p0.4-el7.parcel.sha1
manifest.json

2.CDH parcel安装包地址：
http://archive.cloudera.com/cm5/repo-as-tarball/5.12.2/

cm5.12.2-centos7.tar.gz

3.kudu的parcel安装包地址：
http://archive.cloudera.com/kudu/parcels/latest/

KUDU-1.4.0-1.cdh5.12.2.p0.8-el7.parcel
KUDU-1.4.0-1.cdh5.12.2.p0.8-el7.parcel.sha1

4.kudu 的csd安装包地址：
http://archive.cloudera.com/kudu/csd/
注意： 通过 Cloudera Manager 安装parcel时sha1格式的文件需要提前修改为sha。

----------------------------------------------准备工作----------------------------------------------

--配置

修改主机名：
hostnamectl set-hostname Node1

--1.修改hosts文件 所有节点

[root@node1 .ssh]# cat /etc/hosts
127.0.0.1 localhost.Node1
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.2.171 Node1
192.168.2.172 Node2
192.168.2.173 Node3　

--2.配置公钥认证 免密登录
2.1 配置公钥认证
ssh-keygen -t rsa
一路回车，生成无密码的密钥对

2.2 将公钥添加到认证文件中：

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

2.3 scp文件到所有受管节点（ Node2 和 Node3）的~/.ssh目录：

scp ~/.ssh/authorized_keys root@Node2:~/.ssh/
scp ~/.ssh/authorized_keys root@Node3:~/.ssh/

2.4如果没有.ssh目录，手动创建并赋权限

[root@node1 ~]# mkdir .ssh
[root@node1 ~]# chmod 700 .ssh/

--3.关闭防火墙

[root@node1 .ssh]# systemctl stop firewalld.service
[root@node1 .ssh]# firewall-cmd --state
not running

　　

--4.关闭SELINUX
sed -i "s/SELINUX=enforcing/SELINUX=disabled/" /etc/selinux/config
setenforce 0

--5.配置NTP服务 同步时间

--6.优化虚拟内存需求率(所有节点)
1)检查虚拟内存需求率
　　cat /proc/sys/vm/swappiness

2)临时降低虚拟内存需求率
　　sysctl vm.swappiness=0

永久降低虚拟内存需求率
使用命令 vi /etc/sysctl.conf 增加
　　vm.swappiness = 0
并运行如下命令使生效
　　sysctl -p

--7.解决透明大页面问题(所有节点)

1)检查透明大页面问题

执行代码：

echo never > /sys/kernel/mm/transparent_hugepage/defrag 
echo never > /sys/kernel/mm/transparent_hugepage/enabled　　

以禁用此设置，然后将同一命令添加到 /etc/rc.local 等初始化脚本中，以便在系统重启时予以设置。

cat /sys/kernel/mm/transparent_hugepage/defrag

　　如果显示为：

　　[always] madvise never

2)临时关闭透明大页面问题
echo never > /sys/kernel/mm/transparent_hugepage/defrag
确认配置生效：
cat /sys/kernel/mm/transparent_hugepage/defrag
应该显示为：
always madvise [never]
3)配置开机自动生效
使用命令 vim /etc/rc.local,加入如下内容

echo never > /sys/kernel/mm/transparent_hugepage/defrag 
echo never > /sys/kernel/mm/transparent_hugepage/enabled

--8.安装JDK

卸载自带的OpenJdk，
使用 rpm -qa | grep java 查询java相关的包，
使用 rpm -e --nodeps 包名 卸载。

--------------------------------------- -----------------安装配置MariaDB--------------------------------------- 

a. 确保/var/lib/mysql目录有足够大的磁盘空间
b. 卸载自带的mysql。
　　yum remove mysql
1.添加yum源

vi /etc/yum.repos.d/MariaDB.repo

[mariadb]
name = MariaDB
baseurl = http://yum.mariadb.org/10.1/centos7-amd64
gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
gpgcheck=1　　

2.安装MariaDB
　　yum install MariaDB-server MariaDB-client MariaDB-devel --enablerepo=MariaDB.repo

3.配置my.conf
vim /etc/my.cnf
用以下内容替换

[mysqld]
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
# symbolic-links = 0

key_buffer = 16M
key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1

max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M

#log_bin should be on a disk with enough free space. Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your system
#and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log

binlog_format = mixed

read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M

# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit  = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

3.启动MariaDB
　　service mysql start

启动失败：
失败1：
    /usr/bin/mysqld_safe_helper: Can't create/write to file '/var/log/mysqld.log' (Errcode: 13 "Permission denied")
授权：
    chown -R mysql /var/log/
失败2：
    2018-02-05 11:22:22 139647932491520 [ERROR] mysqld: Can't create/write to file '/var/run/mysqld/mysqld.pid' (Errcode: 13 "Permission denied")
    2018-02-05 11:22:22 139647932491520 [ERROR] Can't start server: can't create PID file: Permission denied
解决：
[root@Node1 log]# cat /var/log/mysqld.log
[root@Node1 run]# mkdir mysqld
[root@Node1 log]# chown -R mysql /var/run/mysqld
[root@Node1 log]# chgrp -R mysql /var/run/mysqld

4.查看MariaDB版本
mysql --version

5.设置开机启动
chkconfig mysql on

6.初始化数据库

sudo /usr/bin/mysql_secure_installation

Enter current password for root (enter for none):  Enter

Set root password? [Y/n] y
New password: 123456
Re-enter new password: 
Password updated successfully!
Reloading privilege tables..
 ... Success!

Remove anonymous users? [Y/n] y
 ... Success!

Normally, root should only be allowed to connect from 'localhost'.  This
ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n] n
 ... skipping.

By default, MariaDB comes with a database named 'test' that anyone can
access.  This is also intended only for testing, and should be removed
before moving into a production environment.

Remove test database and access to it? [Y/n] y
 - Dropping test database...
 ... Success!
 - Removing privileges on test database...
 ... Success!

Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.

Reload privilege tables now? [Y/n] y
 ... Success!

Cleaning up...

All done!  If you've completed all of the above steps, your MariaDB
installation should now be secure.

Thanks for using MariaDB!

7.使用mysql -uroot -p进入mysql命令行，创建数据库和用户：

[root@Node1 log]# mysql -uroot -p
Enter password: 
Welcome to the MariaDB monitor.  Commands end with ; or g.
Your MariaDB connection id is 12
Server version: 10.1.30-MariaDB MariaDB Server

Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or 'h' for help. Type 'c' to clear the current input statement.

MariaDB [(none)]> create database hive DEFAULT CHARACTER SET utf8;
Query OK, 1 row affected (0.00 sec)

MariaDB [(none)]> grant all on hive.* TO 'hive'@'%' IDENTIFIED BY 'hive';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> create database hue DEFAULT CHARACTER SET utf8;
Query OK, 1 row affected (0.00 sec)

MariaDB [(none)]> grant all on hue.* TO 'hue'@'%' IDENTIFIED BY 'hue';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> create database oozie DEFAULT CHARACTER SET utf8;
Query OK, 1 row affected (0.00 sec)

MariaDB [(none)]> grant all on oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie';   
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> flush privileges;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> exit
Bye
 

-------------------------------------------安装mysql JDBC 驱动（管理节点）-----------------------------------------

下载mysql JDBC 驱动放到目录 /usr/share/java/ 并修改名为mysql-connector-java.jar
下载地址：https://dev.mysql.com/downloads/connector/j/

[root@Node1 mysql-connector-java-5.1.45]# cp mysql-connector-java-5.1.45-bin.jar /usr/share/java/mysql-connector-java.jar

----------------------------------------- 安装配置 Cloudera Manager（管理节点）---------------------------------------
建立Cloudera Manager安装文件自定义存储库

a.安装httpd服务器

查询一下是否已经安装了apache
rpm -qa httpd 
如果还没有则进行安装
yum -y install httpd
启动apache
service httpd start
开机自启动
chkconfig httpd on

1.将Cloudera Manager仓库镜像包cm5.12.2-centos7.tar.gz
解压到/var/www/html目录，文件目录结构如下
[root@Node1 cm]# tar zxvf /root/softwares/kudu/cm5.12.2-centos7.tar.gz -C /var/www/html

2.通过rpm安装包本地安装 Cloudera Manager
到目录 /var/www/html/cm/5/RPMS/x86_64

--本地安装rpm包，不验证签名

yum --nogpgcheck localinstall cloudera-manager-daemons-5.12.*.rpm cloudera-manager-server-5.12.*.rpm enterprise-debuginfo-5.12.*.rpm

3.Parcel和csd格式文件上传
a. 上传下列文件到Parcel包的存放路径： /root/cloudera/parcel-repo/
CDH-5.12.2-1.cdh5.12.2.p0.4-el7.parcel
CDH-5.12.2-1.cdh5.12.2.p0.4-el7.parcel.sha1
KUDU-1.4.0-1.cdh5.12.2.p0.8-el7.parcel
KUDU-1.4.0-1.cdh5.12.2.p0.8-el7.parcel.sha1
manifest.json

 

4.上传最新的kudu csd文件到csd包存放路径： /root/cloudera/csd/
KUDU-5.10.2.jar

5.配置 Cloudera Manager Server 数据库

使用命令scm_prepare_database.sh创建Cloudera Manager Server数据库配置文件
命令格式如下
/usr/share/cmf/schema/scm_prepare_database.sh database-type [options] database-name username password
如：

/usr/share/cmf/schema/scm_prepare_database.sh mysql -hNode1 -uroot -p123456 --scm-host Node1 scm scm scm

成功：

[root@node1 cloudera-scm-server]# /usr/share/cmf/schema/scm_prepare_database.sh mysql -hNode1 -uroot -p123456 --scm-host Node1 scm scm scm
JAVA_HOME=/usr/java/jdk1.7.0_79
Verifying that we can write to /etc/cloudera-scm-server
Creating SCM configuration file in /etc/cloudera-scm-server
Executing: /usr/java/jdk1.7.0_79/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
[ main] DbCommandExecutor INFO Successfully connected to database.
All done, your SCM database is configured correctly!
[root@node1 cloudera-scm-server]#　

执行完成后生成数据库配置文件/etc/cloudera-scm-server/db.properties
# Auto-generated by scm_prepare_database.sh on 2018年 01月 10日 星期三 14:42:32 CST
#
# For information describing how to configure the Cloudera Manager Server
# to connect to databases, see the "Cloudera Manager Installation Guide."
#
com.cloudera.cmf.db.type=mysql
com.cloudera.cmf.db.host=localhost
com.cloudera.cmf.db.name=scm
com.cloudera.cmf.db.user=scm
com.cloudera.cmf.db.setupType=EXTERNAL
com.cloudera.cmf.db.password=scm

报错：
mysql>use mysql;
mysql>desc user;
mysql>GRANT ALL PRIVILEGES ON *.* TO root@"%" IDENTIFIED BY "123456";　　//为root添加远程连接的能力。
mysql>update user set Password = password('123456') where User='root';
mysql>select Host,User,Password  from user where User='root'; 
mysql>flush privileges;
mysql>exit

grant all on scm.* to 'root'@'%' identified by '123456' with grant option;

cat /etc/cloudera-scm-server/db.properties

6.启动Cloudera Manager Server

[root@node1 ~]# service cloudera-scm-server start
Starting cloudera-scm-server (via systemctl): Warning: cloudera-scm-server.service changed on disk. Run 'systemctl daemon-reload' to reload units.
[ 确定 ]
[root@node1 ~]# systemctl daemon-reload
[root@node1 ~]# service cloudera-scm-server start
Starting cloudera-scm-server (via systemctl): [ 确定 ]

启动失败报错没有JDK
一个是OS的环境变量，一个是脚本运行时候的设置
[root@Node1 init.d]# pwd
/etc/init.d
[root@Node1 init.d]# vim cloudera-scm-server
添加：
export JAVA_HOME=/root/jdk1.7.0_79

再次启动：
[root@Node1 init.d]# service cloudera-scm-server start
查看日志：
[root@Node1 init.d]# cat /var/log/cloudera-scm-server/cloudera-scm-server.out 
JAVA_HOME=/root/jdk1.7.0_79
/usr/share/cmf/bin/cmf-server:行129: /root/jdk1.7.0_79/bin/java: 权限不够
/usr/share/cmf/bin/cmf-server: 第 129 行:exec: /root/jdk1.7.0_79/bin/java: 无法执行: 权限不够

修改jdk路径
[root@Node1 ~]# cd /usr/
[root@Node1 usr]# mkdir java/
[root@Node1 init.d]# echo $JAVA_HOME
/usr/java/jdk1.7.0_79

再次启动：
[root@Node1 init.d]# service cloudera-scm-server start

--CM

等待大概两分钟，访问 http://192.168.2.171:7180/ 进入管理端 (登陆名:admin 密码:admin)

--kudu
http://192.168.2.171:8051/