zoukankan      html  css  js  c++  java
  • 基于【CentOS-7+ Ambari 2.7.0 + HDP 3.0】搭建HAWQ数据仓库02 ——使用ambari-server安装HDP

    本文记录使用ambari-server安装HDP的过程,对比于使用cloudera-manager安装CDH,不得不说ambari的易用性差的比较多~_~,需要用户介入的过程较多,或者说可定制性更高。

    首先、安装之前,在每个主机节点上执行下面命令,已清除缓存,避免一些repo原因导致的安装失败。

    yum clean all

    下面开始安装过程:

    一、安装过程:

    1,登录ambari-server管理界面,用浏览器访问http://ep-bd01:8080,默认用户名口令皆为admin。

    2,点击按钮“LUNCH INSTALL WIZZARD”,给集群起名,这里为EPBD,下一步

    4,选择HDP版本3.0.0.0,配置repo地址

    这一步ambari自动列出配置在本地repo中HDP版本的repo ID

    下面是仓库的设置,这里选择本地仓库,删除掉除了"Redhat7"之外的其他操作系统,仓库基地址,就是前面配置的hdp-local.repo中的设置:

    http://ep-bd01/hdp/HDP/centos7/3.0.0.0-1634 、 http://ep-bd01/hdp/HDP-GPL/centos7/3.0.0.0-1634 和  http://ep-bd01/hdp/HDP-UTILS/centos7/1.1.0.22 

    然后,选中“Use RedHat Satellite/Spacewalk”,此时可以修改仓库名称,确保和配好的hdp.repo中保持一致,点击下一步。

    5,[Target Hosts]填写集群中主机列表,主机填写可以使用中括号加上序数后缀范围的方式,详细用法点击"Pattern Expressions"

    主机注册方式可以选中SSH方式,这需要提供ssh免密访问所用私有证书;

    或者选择“Perform manual registration on hosts and do not use SSH”,这种方式需要在每台主机上事先安装好ambari-agent,就如我在上一篇中所做的,所以我选择的是这种方式。经试验对比用SSH的方式注册主机时稍稍快上一点儿。

    进行下一步“REGISTER AND CONFIRM”,ambari可能会提示主机名称不是全名称FQDN,不用理会它,继续进行即可。

    6,点击下一步开始进行主机检测,检测成功后可以点击“ Click here to see the check results”可以查看检测结果。

    7,进入选择filesystem和services,这里接受默认设置,点击next。

    注:后经过无数次失败的打击,我取消了Ranger和Ranger KMS服务,原因不知,这里又一次失败的log:

    stderr: 
    2018-08-17 12:04:22,639 - The 'ranger-kms' component did not advertise a version. This may indicate a problem with the component packaging. However, the stack-select tool was able to report a single version installed (3.0.0.0-1634). This is the version that will be reported.
    Traceback (most recent call last):
      File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/RANGER_KMS/package/scripts/kms_server.py", line 137, in <module>
        KmsServer().execute()
      File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 353, in execute
        method(env)
      File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/RANGER_KMS/package/scripts/kms_server.py", line 52, in install
        self.configure(env)
      File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/RANGER_KMS/package/scripts/kms_server.py", line 94, in configure
        kms.kms()
      File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/RANGER_KMS/package/scripts/kms.py", line 150, in kms
        create_parents = True
      File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
        self.env.run()
      File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
        self.run_action(resource, action)
      File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
        provider_action()
      File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 177, in action_create
        raise Fail("Applying %s failed, looped symbolic links found while resolving %s" % (self.resource, path))
    resource_management.core.exceptions.Fail: Applying Directory['/usr/hdp/current/ranger-kms/conf'] failed, looped symbolic links found while resolving /usr/hdp/current/ranger-kms/conf
     stdout:
    2018-08-17 12:04:22,355 - Stack Feature Version Info: Cluster Stack=3.0, Command Stack=None, Command Version=None -> 3.0
    2018-08-17 12:04:22,358 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
    2018-08-17 12:04:22,359 - Group['kms'] {}
    2018-08-17 12:04:22,360 - Group['livy'] {}
    2018-08-17 12:04:22,360 - Group['spark'] {}
    2018-08-17 12:04:22,360 - Group['ranger'] {}
    2018-08-17 12:04:22,360 - Group['hdfs'] {}
    2018-08-17 12:04:22,360 - Group['zeppelin'] {}
    2018-08-17 12:04:22,360 - Group['hadoop'] {}
    2018-08-17 12:04:22,361 - Group['users'] {}
    2018-08-17 12:04:22,361 - Group['knox'] {}
    2018-08-17 12:04:22,361 - User['yarn-ats'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
    2018-08-17 12:04:22,362 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
    2018-08-17 12:04:22,363 - User['storm'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
    2018-08-17 12:04:22,363 - User['infra-solr'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
    2018-08-17 12:04:22,364 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
    2018-08-17 12:04:22,365 - User['oozie'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None}
    2018-08-17 12:04:22,366 - User['atlas'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
    2018-08-17 12:04:22,366 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
    2018-08-17 12:04:22,367 - User['ranger'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['ranger', 'hadoop'], 'uid': None}
    2018-08-17 12:04:22,368 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None}
    2018-08-17 12:04:22,368 - User['zeppelin'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['zeppelin', 'hadoop'], 'uid': None}
    2018-08-17 12:04:22,369 - User['kms'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['kms', 'hadoop'], 'uid': None}
    2018-08-17 12:04:22,370 - User['accumulo'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
    2018-08-17 12:04:22,370 - User['livy'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['livy', 'hadoop'], 'uid': None}
    2018-08-17 12:04:22,371 - User['druid'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
    2018-08-17 12:04:22,372 - User['spark'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['spark', 'hadoop'], 'uid': None}
    2018-08-17 12:04:22,373 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None}
    2018-08-17 12:04:22,373 - User['kafka'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
    2018-08-17 12:04:22,374 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hdfs', 'hadoop'], 'uid': None}
    2018-08-17 12:04:22,375 - User['sqoop'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
    2018-08-17 12:04:22,376 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
    2018-08-17 12:04:22,376 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
    2018-08-17 12:04:22,377 - User['hbase'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
    2018-08-17 12:04:22,378 - User['knox'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'knox'], 'uid': None}
    2018-08-17 12:04:22,378 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
    2018-08-17 12:04:22,379 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
    2018-08-17 12:04:22,383 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if
    2018-08-17 12:04:22,383 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'}
    2018-08-17 12:04:22,384 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
    2018-08-17 12:04:22,385 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
    2018-08-17 12:04:22,385 - call['/var/lib/ambari-agent/tmp/changeUid.sh hbase'] {}
    2018-08-17 12:04:22,391 - call returned (0, '1015')
    2018-08-17 12:04:22,392 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1015'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'}
    2018-08-17 12:04:22,395 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1015'] due to not_if
    2018-08-17 12:04:22,396 - Group['hdfs'] {}
    2018-08-17 12:04:22,396 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': ['hdfs', 'hadoop', u'hdfs']}
    2018-08-17 12:04:22,396 - FS Type: HDFS
    2018-08-17 12:04:22,396 - Directory['/etc/hadoop'] {'mode': 0755}
    2018-08-17 12:04:22,406 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
    2018-08-17 12:04:22,407 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777}
    2018-08-17 12:04:22,419 - Repository['HDP-3.0-repo-1'] {'append_to_file': False, 'base_url': 'http://ep-bd01/hdp/HDP/centos7/3.0.0.0-1634', 'action': ['create'], 'components': [u'HDP', 'main'], 'repo_template': '[{{repo_id}}]
    name={{repo_id}}
    {% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}
    
    path=/
    enabled=1
    gpgcheck=0', 'repo_file_name': 'ambari-hdp-1', 'mirror_list': None}
    2018-08-17 12:04:22,424 - File['/etc/yum.repos.d/ambari-hdp-1.repo'] {'content': '[HDP-3.0-repo-1]
    name=HDP-3.0-repo-1
    baseurl=http://ep-bd01/hdp/HDP/centos7/3.0.0.0-1634
    
    path=/
    enabled=1
    gpgcheck=0'}
    2018-08-17 12:04:22,424 - Writing File['/etc/yum.repos.d/ambari-hdp-1.repo'] because contents don't match
    2018-08-17 12:04:22,424 - Repository['HDP-3.0-GPL-repo-1'] {'append_to_file': True, 'base_url': 'http://ep-bd01/hdp/HDP-GPL/centos7/3.0.0.0-1634', 'action': ['create'], 'components': [u'HDP-GPL', 'main'], 'repo_template': '[{{repo_id}}]
    name={{repo_id}}
    {% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}
    
    path=/
    enabled=1
    gpgcheck=0', 'repo_file_name': 'ambari-hdp-1', 'mirror_list': None}
    2018-08-17 12:04:22,427 - File['/etc/yum.repos.d/ambari-hdp-1.repo'] {'content': '[HDP-3.0-repo-1]
    name=HDP-3.0-repo-1
    baseurl=http://ep-bd01/hdp/HDP/centos7/3.0.0.0-1634
    
    path=/
    enabled=1
    gpgcheck=0
    [HDP-3.0-GPL-repo-1]
    name=HDP-3.0-GPL-repo-1
    baseurl=http://ep-bd01/hdp/HDP-GPL/centos7/3.0.0.0-1634
    
    path=/
    enabled=1
    gpgcheck=0'}
    2018-08-17 12:04:22,427 - Writing File['/etc/yum.repos.d/ambari-hdp-1.repo'] because contents don't match
    2018-08-17 12:04:22,444 - Repository['HDP-UTILS-1.1.0.22-repo-1'] {'append_to_file': True, 'base_url': 'http://ep-bd01/hdp/HDP-UTILS/centos7/1.1.0.22', 'action': ['create'], 'components': [u'HDP-UTILS', 'main'], 'repo_template': '[{{repo_id}}]
    name={{repo_id}}
    {% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}
    
    path=/
    enabled=1
    gpgcheck=0', 'repo_file_name': 'ambari-hdp-1', 'mirror_list': None}
    2018-08-17 12:04:22,454 - File['/etc/yum.repos.d/ambari-hdp-1.repo'] {'content': '[HDP-3.0-repo-1]
    name=HDP-3.0-repo-1
    baseurl=http://ep-bd01/hdp/HDP/centos7/3.0.0.0-1634
    
    path=/
    enabled=1
    gpgcheck=0
    [HDP-3.0-GPL-repo-1]
    name=HDP-3.0-GPL-repo-1
    baseurl=http://ep-bd01/hdp/HDP-GPL/centos7/3.0.0.0-1634
    
    path=/
    enabled=1
    gpgcheck=0
    [HDP-UTILS-1.1.0.22-repo-1]
    name=HDP-UTILS-1.1.0.22-repo-1
    baseurl=http://ep-bd01/hdp/HDP-UTILS/centos7/1.1.0.22
    
    path=/
    enabled=1
    gpgcheck=0'}
    2018-08-17 12:04:22,455 - Writing File['/etc/yum.repos.d/ambari-hdp-1.repo'] because contents don't match
    2018-08-17 12:04:22,472 - Package['unzip'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
    2018-08-17 12:04:22,564 - Skipping installation of existing package unzip
    2018-08-17 12:04:22,564 - Package['curl'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
    2018-08-17 12:04:22,571 - Skipping installation of existing package curl
    2018-08-17 12:04:22,571 - Package['hdp-select'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
    2018-08-17 12:04:22,579 - Skipping installation of existing package hdp-select
    2018-08-17 12:04:22,622 - call[('ambari-python-wrap', u'/usr/bin/hdp-select', 'versions')] {}
    2018-08-17 12:04:22,639 - call returned (0, '3.0.0.0-1634')
    2018-08-17 12:04:22,639 - The 'ranger-kms' component did not advertise a version. This may indicate a problem with the component packaging. However, the stack-select tool was able to report a single version installed (3.0.0.0-1634). This is the version that will be reported.
    2018-08-17 12:04:22,822 - Command repositories: HDP-3.0-repo-1, HDP-3.0-GPL-repo-1, HDP-UTILS-1.1.0.22-repo-1
    2018-08-17 12:04:22,822 - Applicable repositories: HDP-3.0-repo-1, HDP-3.0-GPL-repo-1, HDP-UTILS-1.1.0.22-repo-1
    2018-08-17 12:04:22,822 - Looking for matching packages in the following repositories: HDP-3.0-repo-1, HDP-3.0-GPL-repo-1, HDP-UTILS-1.1.0.22-repo-1
    2018-08-17 12:04:26,159 - Adding fallback repositories: HDP-UTILS-1.1.0.22, HDP-3.0-GPL, HDP-3.0
    2018-08-17 12:04:29,472 - Package['ranger_3_0_0_0_1634-kms'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
    2018-08-17 12:04:29,527 - Installing package ranger_3_0_0_0_1634-kms ('/usr/bin/yum -y install ranger_3_0_0_0_1634-kms')
    2018-08-17 12:04:41,340 - Stack Feature Version Info: Cluster Stack=3.0, Command Stack=None, Command Version=None -> 3.0
    2018-08-17 12:04:41,355 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
    2018-08-17 12:04:41,359 - Execute[('cp', '-f', u'/usr/hdp/current/ranger-kms/install.properties', u'/usr/hdp/current/ranger-kms/install-backup.properties')] {'not_if': 'ls /usr/hdp/current/ranger-kms/install-backup.properties', 'sudo': True, 'only_if': 'ls /usr/hdp/current/ranger-kms/install.properties'}
    2018-08-17 12:04:41,371 - Password validated
    2018-08-17 12:04:41,372 - File['/var/lib/ambari-agent/tmp/mysql-connector-java.jar'] {'content': DownloadSource('http://ep-bd01:8080/resources/mysql-connector-java.jar'), 'mode': 0644}
    2018-08-17 12:04:41,372 - Not downloading the file from http://ep-bd01:8080/resources/mysql-connector-java.jar, because /var/lib/ambari-agent/tmp/mysql-connector-java.jar already exists
    2018-08-17 12:04:41,373 - Directory['/usr/hdp/current/ranger-kms/ews/lib'] {'mode': 0755}
    2018-08-17 12:04:41,373 - Creating directory Directory['/usr/hdp/current/ranger-kms/ews/lib'] since it doesn't exist.
    2018-08-17 12:04:41,374 - Execute[('cp', '--remove-destination', u'/var/lib/ambari-agent/tmp/mysql-connector-java.jar', u'/usr/hdp/current/ranger-kms/ews/webapp/lib')] {'path': ['/bin', '/usr/bin/'], 'sudo': True}
    2018-08-17 12:04:41,379 - File['/usr/hdp/current/ranger-kms/ews/webapp/lib/mysql-connector-java.jar'] {'mode': 0644}
    2018-08-17 12:04:41,380 - ModifyPropertiesFile['/usr/hdp/current/ranger-kms/install.properties'] {'owner': 'kms', 'properties': ...}
    2018-08-17 12:04:41,380 - Modifying existing properties file: /usr/hdp/current/ranger-kms/install.properties
    2018-08-17 12:04:41,387 - File['/usr/hdp/current/ranger-kms/install.properties'] {'owner': 'kms', 'content': ..., 'group': None, 'mode': None, 'encoding': 'utf-8'}
    2018-08-17 12:04:41,388 - Writing File['/usr/hdp/current/ranger-kms/install.properties'] because contents don't match
    2018-08-17 12:04:41,388 - Changing owner for /usr/hdp/current/ranger-kms/install.properties from 0 to kms
    2018-08-17 12:04:41,388 - ModifyPropertiesFile['/usr/hdp/current/ranger-kms/install.properties'] {'owner': 'kms', 'properties': {'SQL_CONNECTOR_JAR': u'/usr/hdp/current/ranger-kms/ews/webapp/lib/mysql-connector-java.jar'}}
    2018-08-17 12:04:41,388 - Modifying existing properties file: /usr/hdp/current/ranger-kms/install.properties
    2018-08-17 12:04:41,389 - File['/usr/hdp/current/ranger-kms/install.properties'] {'owner': 'kms', 'content': ..., 'group': None, 'mode': None, 'encoding': 'utf-8'}
    2018-08-17 12:04:41,389 - Setting up Ranger KMS DB and DB User
    2018-08-17 12:04:41,389 - Execute['ambari-python-wrap /usr/hdp/current/ranger-kms/dba_script.py -q'] {'logoutput': True, 'environment': {'RANGER_KMS_HOME': u'/usr/hdp/current/ranger-kms', 'JAVA_HOME': u'/usr/java/jdk1.8.0_181-amd64'}, 'tries': 5, 'user': 'kms', 'try_sleep': 10}
    2018-08-17 12:04:41,451  [I] Running DBA setup script. QuiteMode:True
    2018-08-17 12:04:41,451  [I] Using Java:/usr/java/jdk1.8.0_181-amd64/bin/java
    2018-08-17 12:04:41,451  [I] DB FLAVOR:MYSQL
    2018-08-17 12:04:41,451  [I] DB Host:ep-bd01
    2018-08-17 12:04:41,451  [I] ---------- Verifing DB root password ---------- 
    2018-08-17 12:04:41,451  [I] DBA root user password validated
    2018-08-17 12:04:41,451  [I] ---------- Verifing Ranger KMS db user password ---------- 
    2018-08-17 12:04:41,451  [I] KMS user password validated
    2018-08-17 12:04:41,451  [I] ---------- Creating Ranger KMS db user ---------- 
    2018-08-17 12:04:41,451  [JISQL] /usr/java/jdk1.8.0_181-amd64/bin/java  -cp /usr/hdp/current/ranger-kms/ews/webapp/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://ep-bd01/mysql -u root -p '********' -noheader -trim -c ; -query "SELECT version();"
    2018-08-17 12:04:41,716  [I] Verifying user rangerkms for Host %
    2018-08-17 12:04:41,716  [JISQL] /usr/java/jdk1.8.0_181-amd64/bin/java  -cp /usr/hdp/current/ranger-kms/ews/webapp/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://ep-bd01/mysql -u root -p '********' -noheader -trim -c ; -query "select user from mysql.user where user='rangerkms' and host='%';"
    2018-08-17 12:04:41,981  [I] MySQL user rangerkms already exists for host %
    2018-08-17 12:04:41,981  [I] Verifying user rangerkms for Host localhost
    2018-08-17 12:04:41,981  [JISQL] /usr/java/jdk1.8.0_181-amd64/bin/java  -cp /usr/hdp/current/ranger-kms/ews/webapp/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://ep-bd01/mysql -u root -p '********' -noheader -trim -c ; -query "select user from mysql.user where user='rangerkms' and host='localhost';"
    2018-08-17 12:04:42,250  [I] MySQL user rangerkms already exists for host localhost
    2018-08-17 12:04:42,250  [I] Verifying user rangerkms for Host ep-bd01
    2018-08-17 12:04:42,250  [JISQL] /usr/java/jdk1.8.0_181-amd64/bin/java  -cp /usr/hdp/current/ranger-kms/ews/webapp/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://ep-bd01/mysql -u root -p '********' -noheader -trim -c ; -query "select user from mysql.user where user='rangerkms' and host='ep-bd01';"
    2018-08-17 12:04:42,525  [I] MySQL user rangerkms already exists for host ep-bd01
    2018-08-17 12:04:42,525  [I] ---------- Creating Ranger KMS database ----------
    2018-08-17 12:04:42,525  [I] Verifying database rangerkms
    2018-08-17 12:04:42,525  [JISQL] /usr/java/jdk1.8.0_181-amd64/bin/java  -cp /usr/hdp/current/ranger-kms/ews/webapp/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://ep-bd01/mysql -u root -p '********' -noheader -trim -c ; -query "show databases like 'rangerkms';"
    2018-08-17 12:04:42,788  [I] Database rangerkms already exists.
    2018-08-17 12:04:42,788  [I] ---------- Granting permission to Ranger KMS db user ----------
    2018-08-17 12:04:42,788  [I] ---------- Granting privileges TO user 'rangerkms'@'%' on db 'rangerkms'----------
    2018-08-17 12:04:42,788  [JISQL] /usr/java/jdk1.8.0_181-amd64/bin/java  -cp /usr/hdp/current/ranger-kms/ews/webapp/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://ep-bd01/mysql -u root -p '********' -noheader -trim -c ; -query "grant all privileges on rangerkms.* to 'rangerkms'@'%' with grant option;"
    2018-08-17 12:04:43,048  [I] ---------- FLUSH PRIVILEGES ----------
    2018-08-17 12:04:43,048  [JISQL] /usr/java/jdk1.8.0_181-amd64/bin/java  -cp /usr/hdp/current/ranger-kms/ews/webapp/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://ep-bd01/mysql -u root -p '********' -noheader -trim -c ; -query "FLUSH PRIVILEGES;"
    2018-08-17 12:04:43,544  [I] Privileges granted to 'rangerkms' on 'rangerkms'
    2018-08-17 12:04:43,544  [I] ---------- Granting privileges TO user 'rangerkms'@'localhost' on db 'rangerkms'----------
    2018-08-17 12:04:43,544  [JISQL] /usr/java/jdk1.8.0_181-amd64/bin/java  -cp /usr/hdp/current/ranger-kms/ews/webapp/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://ep-bd01/mysql -u root -p '********' -noheader -trim -c ; -query "grant all privileges on rangerkms.* to 'rangerkms'@'localhost' with grant option;"
    2018-08-17 12:04:43,810  [I] ---------- FLUSH PRIVILEGES ----------
    2018-08-17 12:04:43,810  [JISQL] /usr/java/jdk1.8.0_181-amd64/bin/java  -cp /usr/hdp/current/ranger-kms/ews/webapp/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://ep-bd01/mysql -u root -p '********' -noheader -trim -c ; -query "FLUSH PRIVILEGES;"
    2018-08-17 12:04:44,080  [I] Privileges granted to 'rangerkms' on 'rangerkms'
    2018-08-17 12:04:44,080  [I] ---------- Granting privileges TO user 'rangerkms'@'ep-bd01' on db 'rangerkms'----------
    2018-08-17 12:04:44,080  [JISQL] /usr/java/jdk1.8.0_181-amd64/bin/java  -cp /usr/hdp/current/ranger-kms/ews/webapp/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://ep-bd01/mysql -u root -p '********' -noheader -trim -c ; -query "grant all privileges on rangerkms.* to 'rangerkms'@'ep-bd01' with grant option;"
    2018-08-17 12:04:44,353  [I] ---------- FLUSH PRIVILEGES ----------
    2018-08-17 12:04:44,353  [JISQL] /usr/java/jdk1.8.0_181-amd64/bin/java  -cp /usr/hdp/current/ranger-kms/ews/webapp/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://ep-bd01/mysql -u root -p '********' -noheader -trim -c ; -query "FLUSH PRIVILEGES;"
    2018-08-17 12:04:44,619  [I] Privileges granted to 'rangerkms' on 'rangerkms'
    2018-08-17 12:04:44,619  [I] ---------- Ranger KMS DB and User Creation Process Completed..  ---------- 
    2018-08-17 12:04:44,624 - Execute['ambari-python-wrap /usr/hdp/current/ranger-kms/db_setup.py'] {'logoutput': True, 'environment': {'PATH': '/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/var/lib/ambari-agent', 'RANGER_KMS_HOME': u'/usr/hdp/current/ranger-kms', 'JAVA_HOME': u'/usr/java/jdk1.8.0_181-amd64'}, 'tries': 5, 'user': 'kms', 'try_sleep': 10}
    2018-08-17 12:04:44,679  [I] DB FLAVOR :MYSQL
    2018-08-17 12:04:44,679  [I] --------- Verifying Ranger DB connection ---------
    2018-08-17 12:04:44,679  [I] Checking connection..
    2018-08-17 12:04:44,679  [JISQL] /usr/java/jdk1.8.0_181-amd64/bin/java  -cp /usr/hdp/current/ranger-kms/ews/webapp/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://ep-bd01/rangerkms -u 'rangerkms' -p '********' -noheader -trim -c ; -query "SELECT version();"
    2018-08-17 12:04:44,947  [I] Checking connection passed.
    2018-08-17 12:04:44,947  [I] --------- Verifying Ranger DB tables ---------
    2018-08-17 12:04:44,947  [JISQL] /usr/java/jdk1.8.0_181-amd64/bin/java  -cp /usr/hdp/current/ranger-kms/ews/webapp/lib/mysql-connector-java.jar:/usr/hdp/current/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver mysqlconj -cstring jdbc:mysql://ep-bd01/rangerkms -u 'rangerkms' -p '********' -noheader -trim -c ; -query "show tables like 'ranger_masterkey';"
    2018-08-17 12:04:45,211  [I] Table ranger_masterkey already exists in database 'rangerkms'
    2018-08-17 12:04:45,217 - Directory['/usr/hdp/current/ranger-kms/conf'] {'owner': 'kms', 'group': 'kms', 'create_parents': True}
    2018-08-17 12:04:45,217 - Creating directory Directory['/usr/hdp/current/ranger-kms/conf'] since it doesn't exist.
    
    Command failed after 1 tries
    安装ranger-kms失败

    8,【Assign Slaves and Clients】

    9,【Assign Slaves and Clients】

    10,【CREDENTIALS】密码这里我都是一样的粘贴过来,除了Ranger Admin这个保持不变

    11,【DATABASEs】,用户名和数据库一律使用服务名相同

    Hive和Oozie的数据库和用户需要手动建立。 Druid的用户需要建立好。

    [root@ep-bd01 downloads]# mysql -uroot -phadoop
    Welcome to the MariaDB monitor.  Commands end with ; or g.
    Your MariaDB connection id is 22
    Server version: 5.5.56-MariaDB MariaDB Server
    
    Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.
    
    Type 'help;' or 'h' for help. Type 'c' to clear the current input statement.
    
    MariaDB [(none)]> use mysql;
    Reading table information for completion of table and column names
    You can turn off this feature to get a quicker startup with -A
    
    Database changed
    MariaDB [mysql]> create database oozie  DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
    Query OK, 1 row affected (0.00 sec)
    
    MariaDB [mysql]> grant all privileges on *.* to 'oozie'@'%' identified by 'oozie';
    MariaDB [mysql]> create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
    Query OK, 1 row affected (0.00 sec)
    
    MariaDB [mysql]> grant all privileges on *.* to 'hive'@'%' identified by 'hive';
    MariaDB [mysql]> grant all privileges on *.* to 'druid'@'%' identified by 'druid';
     

     

    Hive的数据库配置需要选择“Existing MySQL/MariaDB”,注意密码要和事先建立的用户一致,不能用默认的。填写后点击“TEST CONNECTION”测试成功才行。

    Oozie的设置和Hive基本类似,同样要测试通过才行。

    Ranger的数据库配置需要给出数据库所在主机,和root用户的数据库密码,然后测试通过。 Ranger KMS类似,但是没有测试,所以需要仔细填写,我在一次失败过程中就是这一步时填写的数据库主机名写错了一个字母导致的。

    12,【DIRECTORYs,ACCOUNTs 】【ALL CONFIGURATIONS】

    全部接受默认值,直接下一步

    14,【review】

    没啥好说的,点击next,等待......

    二、遇坑失败,经验总结:

    (一)【重置ambari-server,重新开始安装】

    1,重设reset ambari-server

    systemctl stop ambari-server
    ambari-server reset

    2,由于ambari-server的数据存在数据库中,ambari-server不能自动重置mariadb数据库表,需要手动删除重建ambari数据库:

    mysql -uroot -p
    use mysql;
    drop database ambari;
    create database ambari;
    use ambari;
    source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql;
    use mysql;

    3,重新启动ambari-server和所有主机上的ambari-agent

    systemctl restart ambari-server
    systemctl restart ambari-agent
    ssh  ep-bd02  systemctl restart ambari-agent
    ssh  ep-bd03  systemctl restart ambari-agent
    ssh  ep-bd04  systemctl restart ambari-agent
    ssh  ep-bd05  systemctl restart ambari-agent

    4,卸载已安装的模块软件包(不卸载导致再次安装失败)

    
    

    yum erase -y -C ranger_3_0_0_0_1634-admin hive_3_0_0_0_1634 ambari-infra-solr-client oozie_3_0_0_0_1634-client oozie_3_0_0_0_1634-webapp oozie_3_0_0_0_1634-sharelib-sqoop hadoop_3_0_0_0_1634-libhdfs ranger_3_0_0_0_1634-kafka-plugin ranger_3_0_0_0_1634-hive-plugin druid_3_0_0_0_1634 tez_3_0_0_0_1634 oozie_3_0_0_0_1634-sharelib-pig ranger_3_0_0_0_1634-usersync ranger_3_0_0_0_1634-hbase-plugin accumulo_3_0_0_0_1634 ranger_3_0_0_0_1634-yarn-plugin oozie_3_0_0_0_1634-sharelib-distcp hive_3_0_0_0_1634-jdbc knox_3_0_0_0_1634 oozie_3_0_0_0_1634-sharelib-hcatalog hadoop_3_0_0_0_1634 phoenix_3_0_0_0_1634 atlas-metadata_3_0_0_0_1634-hbase-plugin hbase_3_0_0_0_1634 storm_3_0_0_0_1634 ranger_3_0_0_0_1634-hdfs-plugin hadoop_3_0_0_0_1634-hdfs ranger_3_0_0_0_1634-tagsync atlas-metadata_3_0_0_0_1634-storm-plugin ranger_3_0_0_0_1634-storm-plugin ranger_3_0_0_0_1634-knox-plugin ambari-metrics-grafana oozie_3_0_0_0_1634-common kafka_3_0_0_0_1634 spark2_3_0_0_0_1634-yarn-shuffle oozie_3_0_0_0_1634-sharelib-hive2 oozie_3_0_0_0_1634-sharelib-spark bigtop-jsvc oozie_3_0_0_0_1634-sharelib-mapreduce-streaming bigtop-tomcat atlas-metadata_3_0_0_0_1634 oozie_3_0_0_0_1634-sharelib atlas-metadata_3_0_0_0_1634-hive-plugin oozie_3_0_0_0_1634-sharelib-hive ambari-infra-solr hive_3_0_0_0_1634-hcatalog ambari-metrics-monitor hadoop_3_0_0_0_1634-client hadoop_3_0_0_0_1634-yarn smartsense-hst ranger_3_0_0_0_1634-atlas-plugin hadoop_3_0_0_0_1634-mapreduce hdp-select oozie_3_0_0_0_1634 zookeeper_3_0_0_0_1634 ambari-metrics-hadoop-sink zookeeper_3_0_0_0_1634-server ambari-metrics-collector atlas-metadata_3_0_0_0_1634-sqoop-plugin

     

    5,删除安装目录中的内容:不清除文件将导致分发时出现包解压失败等错误。

    rm -rf /usr/hdp/*
    ssh  ep-bd02  rm -rf /usr/hdp/*
    ssh  ep-bd03  rm -rf /usr/hdp/*
    ssh  ep-bd04  rm -rf /usr/hdp/*
    ssh  ep-bd05  rm -rf /usr/hdp/*

    由于失败重置次数太多,故将以上过程写成脚本,方便执行

    /root/ambari-server-reset.sh

    echo reset ambari server and database ......
    ambari-server stop && echo yes | ambari-server reset >/dev/null  2>&1
    echo Drop and recreate ambari database ......
    mysql -uroot -phadoop <  /root/ambari-server-db-reset.sql
    
    echo remove all packages installed ......

      ssh -t root@ep-bd01 "echo -n "==> Removing installed packages and folders on --- ";hostname;sh /root/rm-hdp-packages.sh >/dev/null 2>&1;rm -rf /usr/hdp/*" &&
      ssh -t root@ep-bd02 "echo -n "==> Removing installed packages and folders on --- ";hostname;sh /root/rm-hdp-packages.sh >/dev/null 2>&1;rm -rf /usr/hdp/*" &&
      ssh -t root@ep-bd03 "echo -n "==> Removing installed packages and folders on --- ";hostname;sh /root/rm-hdp-packages.sh >/dev/null 2>&1;rm -rf /usr/hdp/*" &&
      ssh -t root@ep-bd04 "echo -n "==> Removing installed packages and folders on --- ";hostname;sh /root/rm-hdp-packages.sh >/dev/null 2>&1;rm -rf /usr/hdp/*" &&
      ssh -t root@ep-bd05 "echo -n "==> Removing installed packages and folders on --- ";hostname;sh /root/rm-hdp-packages.sh >/dev/null 2>&1;rm -rf /usr/hdp/*"

    echo restart ambari server and all agents ......
    systemctl restart ambari-server
    systemctl restart ambari-agent
    ssh  ep-bd02  systemctl restart ambari-agent
    ssh  ep-bd03  systemctl restart ambari-agent
    ssh  ep-bd04  systemctl restart ambari-agent
    ssh  ep-bd05  systemctl restart ambari-agent
    echo reset ambari-server done!

    (二)、提供版本文件VDF,导致失败,原因不明

    VDF,我这里是:

    http://ep-bd01/hdp/HDP/centos7/3.0.0.0-1634/HDP-3.0.0.0-1634.xml

    在指定版本页可以顺利读取,但是再部署时报错:“Upload Version Definition File Error”,详细信息:"javax.xml.stream.XMLStreamException: ParseError at [row,col]:[1,1] Message: Content is not allowed in prolog"

    未找到原因。

    (三)、Ranger、Ranger KMS安装失败,多次未找到解决办法,已暂时取消安装,结果安装成功。

    oozie_3_0_0_0_1634-client-4.3.1.3.0.0.0-1634.noarch 强制卸载
    yum remove oozie_3_0_0_0_1634-client-4.3.1.3.0.0.0-1634.noarch  --setopt=tsflags=noscripts -y
  • 相关阅读:
    Threading in C# Learining
    win7 设置IIS
    Aforge视频采集
    C# 删除文件夹下的所有文件
    C# Timer实现实时监听
    Java中堆内存和栈内存详解
    彻底掌握 AQS
    四种线程同步/互斥方式小结
    CyclicBarrier可重用的循环栅栏
    答疑解惑之ExecutorService——shutdown方法和awaitTermination方法使用
  • 原文地址:https://www.cnblogs.com/dajianshi/p/9482495.html
Copyright © 2011-2022 走看看