zoukankan      html  css  js  c++  java
  • [原创]独立模式安装Hive

     

    独立模式安装Hive

     

    在三台CentOS虚拟机server1server2 server3上已经成功安装全分布模式的Hadoop集群,其中server1NameNode, server2server3DataNode。本人在server1NameNode上安装Hive,并使用MySQL数据库存放metadata

     

    1.首先配置MySQL

    安装MySQL

    [root@server1 ~]# yum install mysql mysql -server

    一路选择yes就行

     

    添加MySQL服务

    [root@server1 ~]# /sbin/chkconfig --add mysqld

     

    启动MySQL

    [root@server1 ~]# service mysqld start

    Starting mysqld:                                           [  OK  ]

     

    root账户在本地登录MySQL

    [root@server1 ~]# mysql -u root

    出现欢迎页面,进入MySQL monitor.

     

    创建数据库实例hive

    mysql > CREATE DATABASE hive;

     

    创建用户hive

    mysql > CREATE USER 'hive' IDENTIFIED BY 'hive';

     

    给用户hive赋予相应的访问与读写权限

    mysql > GRANT ALL ON hive.* TO hive@localhost ;

     

    2.配置Hive

     

    下载hive-0.8.1,并解压

    [admin@server1 hive-0.8.1]$ pwd

    /home/admin/hive-0.8.1

     

    修改系统环境变量,在path中加入Hive执行路径

    [root@server1 ~]# vim /etc/profile

    ...................

    export HIVE_HOME=/home/admin/hive-0.8.1

    export PATH=$PATH:$HIVE_HOME/bin

    应用修改

    [root@server1 ~]# source /etc/profile

     

    下载MySQLJDBC驱动包MySQL-connector-java-5.1.18-bin.jar,复制到hive-0.8.1lib目录下

     

    Hiveconf下的文件hive-default.xml.template复制一份,重命名为hive-site.xml

     

    [admin@server1 conf]$ cp hive-default.xml.template hive-site.xml

     

    将复制的hive-site.xml 移动到所安装的Hadoopconf目录下

    [admin@server1 conf]$ mv hive-site.xml  /home/admin/hadoop-0.20.2/conf

     

    修改hive-site.xml,配置,主要修改以下属性

     

    所连接的MySQL数据库实例

     <property>

      <name>javax.jdo.option.ConnectionURL</name>

      <value>jdbc:mysql://localhost:3306/hive</value>

      <description>JDBC connect string for a JDBC metastore</description>

    </property>

     

    连接的MySQL数据库驱动

    <property>

      <name>javax.jdo.option.ConnectionDriverName</name>

      <value>com.mysql.jdbc.Driver</value>

      <description>Driver class name for a JDBC metastore</description>

    </property>

     

    连接的MySQL数据库用户名

    <property>

      <name>javax.jdo.option.ConnectionUserName</name>

      <value>hive</value>

      <description>username to use against metastore database</description>

    </property>

     

    连接的MySQL数据库密码

    <property>

      <name>javax.jdo.option.ConnectionPassword</name>

      <value>hive</value>

      <description>password to use against metastore database</description>

    </property>

     

    2.测试Hive

    在控制台键入Hive,进入Hive操作界面

     

    [admin@server1 ~]$ hive

    Logging initialized using configuration in jar:file:/home/admin/hive-0.8.1/lib/hive-common-0.8.1.jar!/hive-log4j.properties

    Hive history file=/tmp/admin/hive_job_log_admin_201212011113_1138680566.txt

    hive>

     

    显示存在的表:

    hive> SHOW TABLES;      

    OK

    Time taken: 0.063 seconds

     

    建立表records

    hive> CREATE TABLE records (year STRING, temperature INT, quality INT)

        > ROW FORMAT DELIMITED FIELDS TERMINATED BY ' ';               

    OK

    Time taken: 0.253 seconds

     

    显示存在的表,多了一个records

    hive> SHOW TABLES;                                                  

    OK

    records

    Time taken: 0.089 seconds

     

    查看表records的定义

    hive> DESCRIBE records;

    OK

    year string     

    temperature    int  

    quality    int  

    Time taken: 0.139 seconds

     

    向表records导入数据

    hive> LOAD DATA INPATH '/user/admin/in/ncdc/micro/sample.txt'

        > INTO TABLE records;                                    

    Loading data to table default.records

    OK

    Time taken: 0.337 seconds

     

    查看表records的数据

    hive> SELECT * FROM records;

    OK

    1950       0     1

    1950       22    1

    1950       -11   1

    1949       111  1

    1949       78    1

    Time taken: 0.264 seconds

     

    计算records中每一年的最高温度

    hive> SELECT year, MAX(temperature) FROM records GROUP BY year;        

    Total MapReduce jobs = 1

    Launching Job 1 out of 1

    Number of reduce tasks not specified. Estimated from input data size: 1

    In order to change the average load for a reducer (in bytes):

      set hive.exec.reducers.bytes.per.reducer=<number>

    In order to limit the maximum number of reducers:

      set hive.exec.reducers.max=<number>

    In order to set a constant number of reducers:

      set mapred.reduce.tasks=<number>

    Starting Job = job_201211240040_0037, Tracking URL = http://server1:50030/jobdetails.jsp?jobid=job_201211240040_0037

    Kill Command = /home/admin/hadoop-0.20.2/bin/../bin/hadoop job  -Dmapred.job.tracker=server1:9001 -kill job_201211240040_0037

    Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1

    2012-12-01 11:30:57,089 Stage-1 map = 0%,  reduce = 0%

    2012-12-01 11:31:15,188 Stage-1 map = 100%,  reduce = 0%

    2012-12-01 11:31:24,235 Stage-1 map = 100%,  reduce = 100%

    Ended Job = job_201211240040_0037

    MapReduce Jobs Launched:

    Job 0: Map: 1  Reduce: 1   HDFS Read: 51 HDFS Write: 17 SUCESS

    Total MapReduce CPU Time Spent: 0 msec

    OK

    1949       111

    1950       22

    Time taken: 47.238 seconds

     

  • 相关阅读:
    洛谷3703 [SDOI2017] 树点染色 【LCT】【线段树】
    BZOJ4818 [SDOI2017] 序列计数 【矩阵快速幂】
    HDU4625 JZPTREE 【树形DP】【第二类斯特林数】
    LOJ2116 [HNOI2015] 开店 【点分治】
    [NOIP2017] 逛公园 【最短路】【强连通分量】
    css
    html
    spring-springmvc-jdbc小案例
    eclipse myeclipse中的一些配置
    springmvc中的一些服务器报错
  • 原文地址:https://www.cnblogs.com/leeeee/p/7276654.html
Copyright © 2011-2022 走看看