关键词:
hive创建表卡住了
创建hive和hbase关联表卡住了
其实针对这一问题在info级别的日志下是看出哪里有问题的(为什么只能在debug下才能看见呢,不太理解开发者的想法)。
以调试模式进入hive客户端:
hive --hiveconf hive.root.logger=DEBUG,console
执行创建关联表的语句
create table IF NOT EXISTS event_content_temp (id string,pack string,activity string,title string, sx string, dx string, num bigint) stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' with serdeproperties ("hbase.columns.mapping" = ":key,cf:pack,cf:activity,cf:title,cf:sx,cf:dx,cf:num") tblproperties ("hbase.table.name" = "event_content_temp");
显示错误日志:
15/06/30 09:20:00 INFO util.RetryCounter: Sleeping 4000ms before retry #2... 15/06/30 09:20:00 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (java.lang.SecurityException: Unable to locate a login configuration) 15/06/30 09:20:00 WARN zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1075) 15/06/30 09:20:00 DEBUG zookeeper.ClientCnxnSocketNIO: Ignoring exception during shutdown input java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:663) at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:430) at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:189) at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1164) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1104) 15/06/30 09:20:00 DEBUG zookeeper.ClientCnxnSocketNIO: Ignoring exception during shutdown output java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:674) at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:438) at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:196) at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1164) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1104) Interrupting... Be patient, this might take some time. 15/06/30 09:20:01 INFO CliDriver: Interrupting... Be patient, this might take some time.
很显然是zookeeper的问题,可是在ClouderaManager中已经设置了呀,核对之后发现只有Hive Service Configuration Safety Valve for hive-site.xml里加了属性
<property> <name>hive.aux.jars.path</name> <value> file:///opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hive/lib/hive-hbase-handler-0.10.0-cdh4.7.1.jar, file:///opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hive/lib/hbase.jar, file:///opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hive/lib/zookeeper.jar </value> </property> <property> <name>hbase.zookeeper.quorum</name> <value>ip-xxx.cn-north-1.compute.internal:2181,ip-xxxx.cn-north-1.compute.internal:2181,ip-xxxx-112.cn-north-1.compute.internal:2181,ip-xxxx-108.cn-north-1.compute.internal:2181,ip-xxxx111.cn-north-1.compute.internal:2181</value> </property>
Hive Client Configuration Safety Valve for hive-site.xml里没有添加,这里很好理解,hive服务里配置了hbase的zk,但是hive客户端没有配置(hive-site.xml),建议二者配置一致!
配置完之后,对于关联表的操作都可以顺利完成。
总结:建议hive服务端和客户端的hive-site.xml一样
另外,我建了个QQ群:305994766,希望对大数据、算法研发、系统架构感兴趣的朋友能够加入进来,大家一起学习,共同进步(进群请说明自己的公司-职业-昵称)