zoukankan      html  css  js  c++  java
  • Hive Server 2 安装部署测试

    Hive 0.11 包含了Hive Server 1 和 Hive Server 2,还包含1的原因是为了做到向下兼容性。从长远来看都会以Hive Server 2作为首选


    1. 配置hive server监听端口和Host
    <property>
      <name>hive.server2.thrift.port</name>
      <value>10000</value>
    </property>
    <property>
      <name>hive.server2.thrift.bind.host</name>
      <value>test84.hadoop</value>
    </property>

    2. 配置kerberos认证,这样thrift client与hive server 2, hive server 2和hdfs 都由kerberos作认证
    <property>
      <name>hive.server2.authentication</name>
      <value>KERBEROS</value>
      <description>
        Client authentication types.
           NONE: no authentication check
           LDAP: LDAP/AD based authentication
           KERBEROS: Kerberos/GSSAPI authentication
           CUSTOM: Custom authentication provider
                   (Use with property hive.server2.custom.authentication.class)
      </description>
    </property>
    <property>
      <name>hive.server2.authentication.kerberos.principal</name>
      <value>hadoop/_HOST@DIANPING.COM</value>
    </property>
    <property>
      <name>hive.server2.authentication.kerberos.keytab</name>
      <value>/etc/hadoop.keytab</value>
    </property>

    3. 设置impersonation,这样hive server会以提交用户的身份去执行语句,如果设置为false,则会以起hive server daemon的admin user来执行语句
    <property>
      <name>hive.server2.enable.doAs</name>
      <value>true</value>
    </property>

    执行命令$HIVE_HOME/bin/hive --service hiveserver2 会调用org.apache.hive.service.server.HiveServer2的main方法
    hive log中输出日志信息如下:
    2013-09-17 14:59:21,081 INFO  server.HiveServer2 (HiveStringUtils.java:startupShutdownMessage(604)) - STARTUP_MSG: 
    /************************************************************
    STARTUP_MSG: Starting HiveServer2
    STARTUP_MSG:   host = test84.hadoop/10.1.77.84
    STARTUP_MSG:   args = []
    STARTUP_MSG:   version = 0.11.0
    STARTUP_MSG:   classpath = 略.................
    2013-09-17 14:59:21,957 INFO  security.UserGroupInformation (UserGroupInformation.java:loginUserFromKeytab(633)) - Login successful for user hadoop/test84.hadoop@DIANPING.COM using keytab file /etc/hadoop.keytab
    2013-09-17 14:59:21,958 INFO  service.AbstractService (AbstractService.java:init(89)) - Service:OperationManager is inited.
    2013-09-17 14:59:21,958 INFO  service.AbstractService (AbstractService.java:init(89)) - Service:SessionManager is inited.
    2013-09-17 14:59:21,958 INFO  service.AbstractService (AbstractService.java:init(89)) - Service:CLIService is inited.
    2013-09-17 14:59:21,959 INFO  service.AbstractService (AbstractService.java:init(89)) - Service:ThriftCLIService is inited.
    2013-09-17 14:59:21,959 INFO  service.AbstractService (AbstractService.java:init(89)) - Service:HiveServer2 is inited.
    2013-09-17 14:59:21,959 INFO  service.AbstractService (AbstractService.java:start(104)) - Service:OperationManager is started.
    2013-09-17 14:59:21,960 INFO  service.AbstractService (AbstractService.java:start(104)) - Service:SessionManager is started.
    2013-09-17 14:59:21,960 INFO  service.AbstractService (AbstractService.java:start(104)) - Service:CLIService is started.
    2013-09-17 14:59:22,007 INFO  metastore.HiveMetaStore (HiveMetaStore.java:newRawStore(409)) - 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
    2013-09-17 14:59:22,032 INFO  metastore.ObjectStore (ObjectStore.java:initialize(222)) - ObjectStore, initialize called
    2013-09-17 14:59:22,955 INFO  metastore.ObjectStore (ObjectStore.java:getPMF(267)) - Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
    2013-09-17 14:59:23,000 INFO  metastore.ObjectStore (ObjectStore.java:setConf(205)) - Initialized ObjectStore
    2013-09-17 14:59:23,909 INFO  metastore.HiveMetaStore (HiveMetaStore.java:logInfo(452)) - 0: get_databases: default
    2013-09-17 14:59:23,912 INFO  HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(238)) - ugi=hadoop/test84.hadoop@DIANPING.COM ip=unknown-ip-addr cmd=get_databases: default 
    2013-09-17 14:59:23,933 INFO  service.AbstractService (AbstractService.java:start(104)) - Service:ThriftCLIService is started.
    2013-09-17 14:59:23,948 INFO  service.AbstractService (AbstractService.java:start(104)) - Service:HiveServer2 is started.
    2013-09-17 14:59:24,025 INFO  security.UserGroupInformation (UserGroupInformation.java:loginUserFromKeytab(633)) - Login successful for user hadoop/test84.hadoop@DIANPING.COM using keytab file /etc/hadoop.keytab
    2013-09-17 14:59:24,047 INFO  thrift.ThriftCLIService (ThriftCLIService.java:run(435)) - ThriftCLIService listening on test84.hadoop/10.1.77.84:10000
    可以看到在HiveServer2已经变成一个CompisiteService了,它包含了一组service,包括OperationManager,SessionManager,CLIService,ThriftCLIService。并且在初始化的时候会建立HiveMetaStore连接,并调用get_databases命令来测试。最后启动thrift server,监听在test84.hadoop/10.1.77.84:10000端口上

    1. Beeline访问hive server 2
    Beeline是hive 0.11引入的新的交互式CLI,它基于SQLLine,可以作为Hive JDBC Client端访问Hive Server 2,启动一个beeline就是维护了一个session。
    由于采用了kerberos认证方式,所以需要在本地有kerberos ticket,并且在connection url中指定hive server 2的service principal,此处为principal=hadoop/test84.hadoop@DIANPING.COM,另外用户名和密码可以不用填写,之后的语句会以当前ticket cache中principal的用户身份来执行。
    -dpsh-3.2$ bin/beeline 
    Beeline version 0.11.0 by Apache Hive
    beeline> !connect jdbc:hive2://test84.hadoop:10000/default;principal=hadoop/test84.hadoop@DIANPING.COM
    scan complete in 2ms
    Connecting to jdbc:hive2://test84.hadoop:10000/default;principal=hadoop/test84.hadoop@DIANPING.COM
    Enter username for jdbc:hive2://test84.hadoop:10000/default;principal=hadoop/test84.hadoop@DIANPING.COM: 
    Enter password for jdbc:hive2://test84.hadoop:10000/default;principal=hadoop/test84.hadoop@DIANPING.COM: 
    Connected to: Hive (version 0.11.0)
    Driver: Hive (version 0.11.0)
    Transaction isolation: TRANSACTION_REPEATABLE_READ
    0: jdbc:hive2://test84.hadoop:10000/default> select count(1) from abc;
    +------+
    | _c0  |
    +------+
    | 0    |
    +------+
    1 row selected (29.277 seconds)
    0: jdbc:hive2://test84.hadoop:10000/default> !q
    Closing: org.apache.hive.jdbc.HiveConnection
    thrift client和server会建立一个session handler,有唯一的HandleIdentifier,由SessionManager Service管理,这也就是Hive server 2支持concurrency的方式。每次操作(会有不同的opType,比如EXECUTE_STATEMEN)会生成独立的OperationHandle,也有各自的HandleIdentifier。用户在beeline中输入"!q"会销毁该session,并且销毁相应的资源

    ps : 用下来有点不太爽的是执行mapreduce job时候没有执行过程信息 ,如果是一个执行时间很长 的语句,会等很久而没有任何信息反馈

    2. JDBC方式
    hive server 1的driver classname是org.apache.hadoop.hive.jdbc.HiveDriver,Hive Server 2的是 org.apache.hive.jdbc.HiveDriver,这两个容易混淆。
    另外可以在connectionUrl中指定HiveConf param和变量,params之间用‘;’分割,params和variables用'#'来隔开。这些都是session级别的,hive在建立完session后,会首先执行set hiveconf key value语句。
    比如
    jdbc:hive2://test84.hadoop:10000/default?hive.cli.conf.printheader=true#stab=salesTable;icol=customerID
    jdbc:hive2://test84.hadoop:10000/default;user=foo;password=bar


    示例代码
    import java.sql.Connection;
    import java.sql.DriverManager;
    import java.sql.ResultSet;
    import java.sql.ResultSetMetaData;
    import java.sql.SQLException;
    import java.sql.Statement;
    
    public class HiveTest {
    
    	public static void main(String[] args) throws SQLException {
    		try {
    			Class.forName("org.apache.hive.jdbc.HiveDriver");
    		} catch (ClassNotFoundException e) {
    			e.printStackTrace();
    		}
    		Connection conn = DriverManager
    				.getConnection(
    						"jdbc:hive2://test84.hadoop:10000/default;principal=hadoop/test84.hadoop@DIANPING.COM",
    						"", "");
    		Statement stmt = conn.createStatement();
    		String sql = "select * from abc";
    		System.out.println("Running: " + sql);
    		ResultSet res = stmt.executeQuery(sql);
    		ResultSetMetaData rsmd = res.getMetaData();
    		int columnCount = rsmd.getColumnCount();
    		for (int i = 1; i <= columnCount; i++) {
    			System.out.println(rsmd.getColumnTypeName(i) + ":"
    					+ rsmd.getColumnName(i));
    		}
    
    		while (res.next()) {
    			System.out.println(String.valueOf(res.getInt(1)) + "	"
    					+ res.getString(2));
    		}
    	}
    }
    

    如果kerberos认证有问题的话,可以在起client jvm时候增加JVM option " -Dsun.security.krb5.debug=true"来查看详细信息

    参考链接:



  • 相关阅读:
    leetcode 279. Perfect Squares
    leetcode 546. Remove Boxes
    leetcode 312. Burst Balloons
    leetcode 160. Intersection of Two Linked Lists
    leetcode 55. Jump Game
    剑指offer 滑动窗口的最大值
    剑指offer 剪绳子
    剑指offer 字符流中第一个不重复的字符
    leetcode 673. Number of Longest Increasing Subsequence
    leetcode 75. Sort Colors (荷兰三色旗问题)
  • 原文地址:https://www.cnblogs.com/keanuyaoo/p/3327479.html
Copyright © 2011-2022 走看看