zoukankan      html  css  js  c++  java

  • Cloudrea manager5安装CDH5文档

    一、主机规划、存储规划

    服务器配置信息：CentOS6.5 最小化安装+development tools组包，其余组件yum安装即可。

    二、系统设置如下：

    1、服务器信息如下（/etc/hosts文件）：

    192.168.100.231 db01.chavin.king db01

    192.168.100.232 db02.chavin.king db02

    192.168.100.233 db03.chavin.king db03

    192.168.100.234 db04.chavin.king db04

    192.168.100.235 db05.chavin.king db05

    192.168.100.236 db06.chavin.king db06

    192.168.100.237 db07.chavin.king db07

    2、创建普通用户及密码：

    groupadd hadoop

    useradd -g hadoop hadoop

    echo "dbking588" | passwd --stdin hadoop

    3、配置hadoop账号sudo权限（/etc/sudoers）：

    chmod u+w /etc/sudoers

    echo "hadoop ALL=(root)NOPASSWD:ALL" >> /etc/sudoers

    chmod u-w /etc/sudoers

    4、关闭防火墙并且禁用selinux

    sed -i '/SELINUX=enforcing/d' /etc/selinux/config

    sed -i '/SELINUX=disabled/d' /etc/selinux/config

    echo "SELINUX=disabled" >> /etc/selinux/config

    sed -e 's/SELINUX=enforcing/SELINUX=disabled/d' /etc/selinux/config

    5、设置文件打开数量及最大进程数

    cp /etc/security/limits.conf /etc/security/limits.conf.bak

    echo "* soft nproc 32000" >>/etc/security/limits.conf

    echo "* hard nproc 32000" >>/etc/security/limits.conf

    echo "* soft nofile 65535" >>/etc/security/limits.conf

    echo "* hard nofile 65535" >>/etc/security/limits.conf

    6、配置集群时间同步服务

    cp /etc/ntp.conf /etc/ntp.conf.bak

    cp /etc/sysconfig/ntpd /etc/sysconfig/ntpd.bak

    echo "restrict 192.168.100.0 mask 255.255.255.0 nomodify notrap" >> /etc/ntp.conf

    echo "SYNC_HWCLOCK=yes" >> /etc/sysconfig/ntpd

    service ntpd restart

    0-59/10 * * * * /opt/scripts/sync_time.sh

    # cat /opt/scripts/sync_time.sh

    /sbin/service ntpd stop

    /usr/sbin/ntpdate db01.chavin.king

    /sbin/service ntpd start

    三、安装mysql数据库和postgresql数据库

    1、安装mysql数据库（mysql-5.6.24-linux-glibc2.5-x86_64.tar.gz）

    tar -zxvf mysql-5.6.24-linux-glibc2.5-x86_64.tar.gz -C /opt/

    mv mysql-5.6.24-linux-glibc2.5-x86_64/ mysql5624

    yum -y install libaio

    groupadd mysql

    useradd -g mysql mysql

    echo "dbking588" | passwd --stdin mysql

    mv /etc/my.cnf /etc/my.cnf.bak

    cp support-files/my-default.cnf /etc/my.cnf

    /etc/my.cnf添加如下内容：

    ----------------------------------------------------------------------------------------------------

    basedir = /opt/mysql5624

    datadir = /opt/mysql5624/data

    port = 3306

    skip-grant-tables

    log-error=/opt/mysql5624/data/mysql.log

    pid-file=/opt/mysql5624/data/mysql.pid

    scripts/mysql_install_db --user=mysql

    chown -R mysql:mysql /opt/mysql5624/

    cp support-files/mysql.server /etc/rc.d/init.d/mysql

    chkconfig --add mysql

    Service mysql start

    2、安装cloudera集成postgresql数据库

    --需要以以下方式安装postgresql数据库：

    [root@db01 postgresq-libs]# ll

    total 6564

    -rw-r--r-- 1 root root 2905984 Apr 16 23:58 postgresql-8.4.18-1.el6_4.x86_64.rpm

    -rw-r--r-- 1 root root  205732 Apr 16 23:58 postgresql-libs-8.4.18-1.el6_4.x86_64.rpm

    -rw-r--r-- 1 root root 3602880 Apr 16 23:58 postgresql-server-8.4.18-1.el6_4.x86_64.rpm

    [root@db01 postgresq-libs]# rpm -ivh *.rpm

    Preparing...                ########################################### [100%]

       1:postgresql-libs        ########################################### [ 33%]

       2:postgresql             ########################################### [ 67%]

       3:postgresql-server      ########################################### [100%]

    四、安装CM5

    1、软件下载：

    安装版本CM 5.3.6

    总下载地址：http://archive.cloudera.com/cm5/

    cm-5.3.6 bin文件下载地址：http://archive.cloudera.com/cm5/installer/5.3.6/

    cm-5.3.6依赖rpm包：http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.3.6/

    2、配置本地源

    1）安装papche服务器：

    yum -y install httpd

    service httpd start

    chkconfig httpd on

    cd /var/www/html/

    mkdir -p cm5/redhat/6/x86_64/cm/5.3.6/RPMS/x86_64/

    --将下载好的cm5依赖包mv到/var/www/html/cm5/redhat/6/x86_64/cm/5.3.6/RPMS/x86_64/目录下：

    [root@db01 x86_64]# ll

    total 700568

    -rw-r--r-- 1 root root   3989520 Apr 16 22:21 cloudera-manager-agent-5.3.6-1.cm536.p0.244.el6.x86_64.rpm

    -rw-r--r-- 1 root root 499418684 Apr 16 22:22 cloudera-manager-daemons-5.3.6-1.cm536.p0.244.el6.x86_64.rpm

    -rw-r--r-- 1 root root      7852 Apr 16 22:21 cloudera-manager-server-5.3.6-1.cm536.p0.244.el6.x86_64.rpm

    -rw-r--r-- 1 root root      9884 Apr 16 22:21 cloudera-manager-server-db-2-5.3.6-1.cm536.p0.244.el6.x86_64.rpm

    -rw-r--r-- 1 root root    693024 Apr 16 22:21 enterprise-debuginfo-5.3.6-1.cm536.p0.244.el6.x86_64.rpm

    -rw-r--r-- 1 root root  71204325 Apr 16 22:21 jdk-6u31-linux-amd64.rpm

    -rw-r--r-- 1 root root 142039186 Apr 16 22:21 oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm

    --创建repodata相关依赖文件、否则安装cm将默认查找最新版本（/var/www/html/cm5/redhat/6/x86_64/cm/5.3.6目录下）：

    [root@db01 repo-libs]# ll

    total 196

    -rw-r--r-- 1 root root 96552 Apr 16 23:21 createrepo-0.9.9-18.el6.noarch.rpm

    -rw-r--r-- 1 root root 72520 Apr 16 23:21 deltarpm-3.5-0.5.20090913git.el6.x86_64.rpm

    -rw-r--r-- 1 root root 27748 Apr 16 23:21 python-deltarpm-3.5-0.5.20090913git.el6.x86_64.rpm

    [root@db01 repo-libs]# rpm -ivh *.repo

    [root@db01 repo-libs]# cd /var/www/html/cm5/redhat/6/x86_64/cm/5.3.6/

    [root@db01 5.3.6]# createrepo .

    2）配置repo文件

    [cloudera-manager]

    # Packages for Cloudera Manager, Version 5, on RedHat or CentOS 6 x86_64           

    name=Cloudera Manager

    baseurl=http://db01.chavin.king/cm5/redhat/6/x86_64/cm/5.3.6/

    enabled=1

    gpgcheck=0

    3、安装CM5

    ./cloudera-manager-installer.bin

    4、浏览器登录，我这里登录地址为：db01:7180，用户名密码默认admin/admin

    五、安装CDH5（parcels包安装）

    1、下载parcels软件包（安装版本CDH 5.3.6）：http://archive.cloudera.com/cdh5/parcels/5.3.6/

    2、上传文件到/opt/cloudera/parcel-repo/目录下：

    [root@db01 parcel-repo]# ll

    total 725736

    -rw-r--r-- 1 root root 743145472 Apr 17 12:23 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.parcel

    -rw-r--r-- 1 root root        41 Apr 17 12:21 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha1

    [root@db01 parcel-repo]# mv CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.parcel CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel

    [root@db01 parcel-repo]# mv CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha1 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha

    [root@db01 parcel-repo]# ll

    total 1473856

    -rw-r--r-- 1 root root 1509217191 Apr 17 12:24 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel

    -rw-r--r-- 1 root root         41 Apr 17 12:21 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha

    3、重启cloudera服务：

    [root@db01 parcel-repo]# service cloudera-scm-server status

    cloudera-scm-server (pid  22823) is running...

    [root@db01 parcel-repo]# service cloudera-scm-server restart

    Stopping cloudera-scm-server:                              [  OK  ]

    Starting cloudera-scm-server:                               [  OK  ]

    4、配置yum源：

    将db01上yum文件同步到db02、db03、db04、db05、db06、db07/etc/yum.repo.d/上。

    5、向集群中添加主机：

    执行如下命令安装依赖包：

    #yum -y install cyrus-sasl-gssapi fuse cyrus-sasl-plain libxslt fuse-libs redhat-lsb portmap bind-utils

    #yum -y install libxslt fuse-libs

    wps266D.tmp

    wps268D.tmp

    注意：安装以上依赖包才可以正确安装agent服务，否则很可能报错，需要根据实际情况处理。

    问题：

    wps26FB.tmp

    解决办法：

    # sysctl -w vm.swappiness=0

    # echo "vm.swappiness=0" >>/etc/sysctl.conf

    6、配置java环境变量

    echo "export JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera" >> /etc/profile

    echo "export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH" >>/etc/profile

    echo "export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$CLASSPATH" >> /etc/profile

    source /etc/profile

    7、版本汇总

    Cluster 1 — CDH 5

    主机

    db[01-07].chavin.king

    组件

    版本

    发行版

    CDH 版本

    Bigtop-Tomcat（仅限 CDH 5）

    0.7.0+cdh5.3.6+0

    1.cdh5.3.6.p0.18

    CDH 5

    Crunch（仅限 CDH 5 ）

    0.11.0+cdh5.3.6+31

    1.cdh5.3.6.p0.18

    CDH 5

    Flume NG

    1.5.0+cdh5.3.6+93

    1.cdh5.3.6.p0.18

    CDH 5

    MapReduce 1

    2.5.0+cdh5.3.6+898

    1.cdh5.3.6.p0.18

    CDH 5

    Hadoop

    2.5.0+cdh5.3.6+898

    1.cdh5.3.6.p0.18

    CDH 5

    HDFS

    2.5.0+cdh5.3.6+898

    1.cdh5.3.6.p0.18

    CDH 5

    HttpFS

    2.5.0+cdh5.3.6+898

    1.cdh5.3.6.p0.18

    CDH 5

    hadoop-kms

    2.5.0+cdh5.3.6+898

    1.cdh5.3.6.p0.18

    CDH 5

    MapReduce 2

    2.5.0+cdh5.3.6+898

    1.cdh5.3.6.p0.18

    CDH 5

    YARN

    2.5.0+cdh5.3.6+898

    1.cdh5.3.6.p0.18

    CDH 5

    HBase

    0.98.6+cdh5.3.6+115

    1.cdh5.3.6.p0.18

    CDH 5

    Lily HBase Indexer

    1.5+cdh5.3.6+31

    1.cdh5.3.6.p0.18

    CDH 5

    Hive

    0.13.1+cdh5.3.6+397

    1.cdh5.3.6.p0.18

    CDH 5

    HCatalog

    0.13.1+cdh5.3.6+397

    1.cdh5.3.6.p0.18

    CDH 5

    Hue

    3.7.0+cdh5.3.6+203

    1.cdh5.3.6.p0.18

    CDH 5

    Impala

    2.1.5+cdh5.3.6+0

    1.cdh5.3.6.p0.18

    CDH 5

    Kite（仅限 CDH 5 ）

    0.15.0+cdh5.3.6+201

    1.cdh5.3.6.p0.18

    CDH 5

    Llama（仅限 CDH 5 ）

    1.0.0+cdh5.3.6+0

    1.cdh5.3.6.p0.18

    CDH 5

    Mahout

    0.9+cdh5.3.6+25

    1.cdh5.3.6.p0.18

    CDH 5

    Oozie

    4.0.0+cdh5.3.6+349

    1.cdh5.3.6.p0.18

    CDH 5

    Parquet

    1.5.0+cdh5.3.6+69

    1.cdh5.3.6.p0.18

    CDH 5

    Pig

    0.12.0+cdh5.3.6+59

    1.cdh5.3.6.p0.18

    CDH 5

    sentry

    1.4.0+cdh5.3.6+155

    1.cdh5.3.6.p0.18

    CDH 5

    Solr

    4.4.0+cdh5.3.6+352

    1.cdh5.3.6.p0.18

    CDH 5

    spark

    1.2.0+cdh5.3.6+379

    1.cdh5.3.6.p0.18

    CDH 5

    Sqoop

    1.99.4+cdh5.3.6+32

    1.cdh5.3.6.p0.18

    CDH 5

    Sqoop

    1.4.5+cdh5.3.6+78

    1.cdh5.3.6.p0.18

    CDH 5

    Whirr

    0.9.0+cdh5.3.6+19

    1.cdh5.3.6.p0.18

    CDH 5

    ZooKeeper

    3.4.5+cdh5.3.6+91

    1.cdh5.3.6.p0.18

    CDH 5

    Cloudera Manager Management Daemon

    5.3.6

    1.cm536.p0.244

    不适用

    Java 6

    JAVA_HOME=/usr/java/jdk1.6.0_31 java version "1.6.0_31" Java(TM) SE Runtime Environment (build 1.6.0_31-b04) Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode)

    不可用

    不适用

    Java 7

    JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera java version "1.7.0_67" Java(TM) SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

    不可用

    不适用

    Cloudera Manager Agent

    5.3.6

    1.cm536.p0.244.el6

    不适用

    六、添加cloudera managerment service

    图形界面（略），以下同此要求。

    七、添加服务组件

    1、安装zookeeper

    2、安装hdfs

    3、安装yarn

    4、安装hive

    5、安装hbase

    进行相关基准测试。
  • 相关阅读:
    算法训练 接水问题
    算法训练 数组排序去重
    算法训练 A+B Problem
    算法训练 采油区域
    算法训练 会议中心
    JS高级
    JS基础操作
    JavaScript入门（基础）
    表格与表单
    音频与视频
  • 原文地址：https://www.cnblogs.com/wcwen1990/p/6726857.html
Copyright © 2011-2022 走看看