使用Cloudera Manager搭建Hive服务

　　　　　　　　　　　  　使用Cloudera Manager搭建Hive服务

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正杰

版权声明：原创作品，谢绝转载！否则将追究法律责任。

一.安装Hive环境

1>.进入CM服务安装向导

2>.选择需要安装的hive服务 

3>.选择hive的依赖环境，我们选择第一个即可（hive不仅仅可以使用mr计算，还可以使用tez计算哟~） 

4>.为Hive分配角色

    Hive Metastore是管理和存储元信息的服务，它保存了数据库的基本信息以及数据表的定义等，为了能够可靠地保存这些元信息，Hive Metastore一般将它们持久化到关系型数据库中，默认采用了嵌入式数据库Derby（数据存放在内存中），用户可以根据需要启用其他数据库，比如MySQL。

    推荐阅读：https://www.cnblogs.com/yinzhengjie/p/10836132.html

    HCatalog是Hadoop中的表和存储管理层，能够支持用户用不同的工具（Pig、MapReduce）更容易地表格化读写数据。

    HCatalog从Apache孵化器毕业，并于2013年3月26日与Hive项目合并。

    Hive版本0.11.0是包含HCatalog的第一个版本。（随Hive一起安装）,CDH 5.15.1默认使用的是Hive版本为：1.1.0+cdh5.15.1+1395，即Apache Hive 1.1.0版本。



    HCatalog的表抽象向用户提供了Hadoop分布式文件系统（HDFS）中数据的关系视图，并确保用户不必担心数据存储在哪里或以什么格式存储 - RCFile格式，文本文件，SequenceFiles或ORC文件。

    HCatalog支持读写任意格式的SerDe（序列化 - 反序列化）文件。默认情况下，HCatalog支持RCFile，CSV，JSON和SequenceFile以及ORC文件格式。要使用自定义格式，您必须提供InputFormat，OutputFormat和SerDe。


    HCatalog构建于Hive metastore，并包含Hive的DDL。HCatalog为Pig和MapReduce提供读写接口，并使用Hive的命令行界面发布数据定义和元数据探索命令。

    HiveServer2（HS2）是一个服务端接口，使远程客户端可以执行对Hive的查询并返回结果。目前基于Thrift RPC的实现是HiveServer的改进版本，并支持多客户端并发和身份验证

    启动hiveServer2服务后，就可以使用jdbc，odbc，或者thrift的方式连接。 用java编码jdbc或则beeline连接使用jdbc的方式，据说hue是用thrift的方式连接的hive服务。

5>.hive的数据库设置（存储元数据metastore的数据库）

mysql>  CREATE DATABASE hive  CHARACTER SET = utf8;
Query OK, 1 row affected (0.00 sec)

mysql> 
mysql> GRANT ALL PRIVILEGES ON hive.* TO 'hive'@'%' IDENTIFIED BY 'yinzhengjie' WITH GRANT OPTION;                              
Query OK, 0 rows affected (0.07 sec)

mysql> 
mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.02 sec)

mysql> quit
Bye
[root@node101.yinzhengjie.org.cn ~]# 

6>.修改hive在hdfs的数据仓库存放位置 

7>.等待Hive服务部署完成 

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| cdh                |
| hive               |
| mysql              |
| performance_schema |
+--------------------+
5 rows in set (0.00 sec)

mysql> use hive
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> 
mysql> show tables;
+---------------------------+
| Tables_in_hive            |
+---------------------------+
| BUCKETING_COLS            |
| CDS                       |
| COLUMNS_V2                |
| COMPACTION_QUEUE          |
| COMPLETED_TXN_COMPONENTS  |
| DATABASE_PARAMS           |
| DBS                       |
| DB_PRIVS                  |
| DELEGATION_TOKENS         |
| FUNCS                     |
| FUNC_RU                   |
| GLOBAL_PRIVS              |
| HIVE_LOCKS                |
| IDXS                      |
| INDEX_PARAMS              |
| MASTER_KEYS               |
| METASTORE_DB_PROPERTIES   |
| NEXT_COMPACTION_QUEUE_ID  |
| NEXT_LOCK_ID              |
| NEXT_TXN_ID               |
| NOTIFICATION_LOG          |
| NOTIFICATION_SEQUENCE     |
| NUCLEUS_TABLES            |
| PARTITIONS                |
| PARTITION_EVENTS          |
| PARTITION_KEYS            |
| PARTITION_KEY_VALS        |
| PARTITION_PARAMS          |
| PART_COL_PRIVS            |
| PART_COL_STATS            |
| PART_PRIVS                |
| ROLES                     |
| ROLE_MAP                  |
| SDS                       |
| SD_PARAMS                 |
| SEQUENCE_TABLE            |
| SERDES                    |
| SERDE_PARAMS              |
| SKEWED_COL_NAMES          |
| SKEWED_COL_VALUE_LOC_MAP  |
| SKEWED_STRING_LIST        |
| SKEWED_STRING_LIST_VALUES |
| SKEWED_VALUES             |
| SORT_COLS                 |
| TABLE_PARAMS              |
| TAB_COL_STATS             |
| TBLS                      |
| TBL_COL_PRIVS             |
| TBL_PRIVS                 |
| TXNS                      |
| TXN_COMPONENTS            |
| TYPES                     |
| TYPE_FIELDS               |
| VERSION                   |
+---------------------------+
54 rows in set (0.00 sec)

mysql> 

8>.Hive服务添加成功

9>.在CM界面中可以看到Hive服务是运行正常的

二.测试Hive环境是否可用

1>.将测试数据上传到HDFS中

[root@node101.yinzhengjie.org.cn ~]# cat PageViewData.csv
1999/01/11 10:12,us,927,www.yahoo.com/clq,www.yahoo.com/jxq,948.323.252.617
1999/01/12 10:12,de,856,www.google.com/g4,www.google.com/uypu,416.358.537.539
1999/01/12 10:12,se,254,www.google.com/f5,www.yahoo.com/soeos,564.746.582.215
1999/01/12 10:12,de,465,www.google.com/h5,www.yahoo.com/agvne,685.631.592.264
1999/01/12 10:12,cn,856,www.yinzhengjie.org.cn/g4,www.google.com/uypu,416.358.537.539
1999/01/13 10:12,us,927,www.yahoo.com/clq,www.yahoo.com/jxq,948.323.252.617
1999/01/13 10:12,de,856,www.google.com/g4,www.google.com/uypu,416.358.537.539
1999/01/13 10:12,se,254,www.google.com/f5,www.yahoo.com/soeos,564.746.582.215
1999/01/13 10:12,de,465,www.google.com/h5,www.yahoo.com/agvne,685.631.592.264
1999/01/13 10:12,de,856,www.yinzhengjie.org.cn/g4,www.google.com/uypu,416.358.537.539
1999/01/13 10:12,us,927,www.yahoo.com/clq,www.yahoo.com/jxq,948.323.252.617
1999/01/14 10:12,de,856,www.google.com/g4,www.google.com/uypu,416.358.537.539
1999/01/14 10:12,se,254,www.google.com/f5,www.yahoo.com/soeos,564.746.582.215
1999/01/15 10:12,de,465,www.google.com/h5,www.yahoo.com/agvne,685.631.592.264
1999/01/15 10:12,de,856,www.yinzhengjie.org.cn/g4,www.google.com/uypu,416.358.537.539
1999/01/15 10:12,us,927,www.yahoo.com/clq,www.yahoo.com/jxq,948.323.252.617
1999/01/15 10:12,de,856,www.google.com/g4,www.google.com/uypu,416.358.537.539
1999/01/15 10:12,se,254,www.google.com/f5,www.yahoo.com/soeos,564.746.582.215
1999/01/15 10:12,de,465,www.google.com/h5,www.yahoo.com/agvne,685.631.592.264
1999/01/15 10:12,de,856,www.yinzhengjie.org.cn/g4,www.google.com/uypu,416.358.537.539
[root@node101.yinzhengjie.org.cn ~]# 

 

[root@node101.yinzhengjie.org.cn ~]# hdfs dfs -ls  /tmp/
Found 5 items
d---------   - hdfs   supergroup          0 2019-05-20 10:48 /tmp/.cloudera_health_monitoring_canary_files
drwxr-xr-x   - yarn   supergroup          0 2018-10-19 15:00 /tmp/hadoop-yarn
drwx-wx-wx   - root   supergroup          0 2019-04-29 14:27 /tmp/hive
drwxrwxrwt   - mapred hadoop              0 2019-02-26 16:46 /tmp/logs
drwxr-xr-x   - mapred supergroup          0 2018-10-25 12:11 /tmp/mapred
[root@node101.yinzhengjie.org.cn ~]# 
[root@node101.yinzhengjie.org.cn ~]# 
[root@node101.yinzhengjie.org.cn ~]# ll
total 4
-rw-r--r-- 1 root root 1584 May 20 10:42 PageViewData.csv
[root@node101.yinzhengjie.org.cn ~]# 
[root@node101.yinzhengjie.org.cn ~]# hdfs dfs -put PageViewData.csv /tmp/
[root@node101.yinzhengjie.org.cn ~]# 
[root@node101.yinzhengjie.org.cn ~]# hdfs dfs -ls  /tmp/
Found 6 items
d---------   - hdfs   supergroup          0 2019-05-20 10:48 /tmp/.cloudera_health_monitoring_canary_files
-rw-r--r--   3 root   supergroup       1584 2019-05-20 10:49 /tmp/PageViewData.csv
drwxr-xr-x   - yarn   supergroup          0 2018-10-19 15:00 /tmp/hadoop-yarn
drwx-wx-wx   - root   supergroup          0 2019-04-29 14:27 /tmp/hive
drwxrwxrwt   - mapred hadoop              0 2019-02-26 16:46 /tmp/logs
drwxr-xr-x   - mapred supergroup          0 2018-10-25 12:11 /tmp/mapred
[root@node101.yinzhengjie.org.cn ~]# 
[root@node101.yinzhengjie.org.cn ~]# 

2>.创建数据表page_view，以保证结构化用户访问日志

[root@node101.yinzhengjie.org.cn ~]# hive
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0

Logging initialized using configuration in jar:file:/opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/jars/hive-common-1.1.0-cdh5.15.1.jar!/hive-log4j.properties
WARNING: Hive CLI is deprecated and migration to Beeline is recommended.
hive> CREATE TABLE page_view(
    >     view_time String,
    >     country String,
    >     userid String,
    >     page_url String,
    >     referrer_url String,
    >     ip String)
    > ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED by '
'
    > STORED AS TEXTFILE;
OK
Time taken: 2.598 seconds
hive> show tables;
OK
page_view
Time taken: 0.166 seconds, Fetched: 1 row(s)
hive> 

3>.使用LOAD语句将HDFS上的指定目录或文件加载到数据表page_view中

hive> LOAD DATA INPATH "/tmp/PageViewData.csv" INTO TABLE page_view;
Loading data to table default.page_view
Table default.page_view stats: [numFiles=1, totalSize=1584]
OK
Time taken: 0.594 seconds
hive> 

4>.使用HQL查询数据。

hive> SELECT country,count(userid) FROM page_view WHERE view_time > "1990/01/12 10:12" GROUP BY country;
Query ID = root_20190523125656_e7558dc5-d450-4d17-bf81-209f802605de
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapred.reduce.tasks=<number>
Starting Job = job_201905221917_0001, Tracking URL = http://node101.yinzhengjie.org.cn:50030/jobdetails.jsp?jobid=job_201905221917_0001
Kill Command = /opt/cloudera/parcels/CDH-5.15.1-1.cdh5.15.1.p0.4/lib/hadoop/bin/hadoop job  -kill job_201905221917_0001
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2019-05-23 12:56:45,895 Stage-1 map = 0%,  reduce = 0%
2019-05-23 12:56:52,970 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 3.29 sec
2019-05-23 12:56:59,017 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 5.37 sec
MapReduce Total cumulative CPU time: 5 seconds 370 msec
Ended Job = job_201905221917_0001
MapReduce Jobs Launched: 
Stage-Stage-1: Map: 1  Reduce: 1   Cumulative CPU: 5.37 sec   HDFS Read: 10553 HDFS Write: 21 SUCCESS
Total MapReduce CPU Time Spent: 5 seconds 370 msec
OK
cn      1
de      11
se      4
us      4
Time taken: 25.063 seconds, Fetched: 4 row(s)
hive>