这里是使用mysql作为hive的元数据存储所以要先启动mysql的服务,MySQL的安装可以参考下面连接
MySQL安装:http://www.cnblogs.com/zhangXingSheng/p/6941422.html
下载hive(http://mirrors.cnnic.cn/apache/hive/)
或者 http://archive.apache.org/dist/hive/(hive历史版本)
在本地进行解压
下载mysql的jdbc,将下载后的jdbc放到hive安装包的lib目录下
http://dev.mysql.com/downloads/connector/j/
配置环境变量
#########hive####################
export HIVE_HOME=/usr/local/development/hive-2.1.0
export PATH=$HIVE_HOME/bin:$PATH
在hdfs上创建hive存储数据的相关目录
[root@node4 bin]# hdfs dfs -mkdir -p /opt/hive/warehouse
[root@node4 bin]# hdfs dfs -mkdir -p /opt/hive/log
修改hive-site.xml文件
cp hive-default.xml.template hive-site.xml
<name>hive.metastore.warehouse.dir</name> <value>/opt/hive/warehouse</value>//对应以上在hdfs创建的文件目录 <description>location of default database for the warehouse</description> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://192.168.177.124:3306/hivedb</value>//mysql的链接地址,这里的hivedb是我已经在mysql中创建的数据库,如果不写就是default <description> JDBC connect string for a JDBC metastore. To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL. For example, jdbc:postgresql://myhost/dbName?ssl=true for postgres database. </description> <name>javax.jdo.option.ConnectionDriverName</name> <value>com.mysql.jdbc.Driver</value>//mysql的驱动 <description>Driver class name for a JDBC metastore</description> <name>javax.jdo.option.ConnectionUserName</name> <value>hive</value>//链接mysql的用户名 <description>Username to use against metastore database</description> <name>javax.jdo.option.ConnectionPassword</name> <value>hive</value>////链接mysql的密码 <description>password to use against metastore database</description> <name>hive.metastore.uris</name> <value>thrift://192.168.177.124:9083</value> <description>Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore.</description>
修改hive-log4j2.properties和hive-exec-log4j2.properties//hive的日志文件
在 新建日志存储目录/home/zhangxs/hive/log 和 /home/zhangxs/hive/execlog
(1)hive-log4j2.properties
property.hive.log.dir = ${mine:/home/zhangxs/hive/log}/${user.name}
(2)hive-exec-log4j2.properties
property.hive.log.dir = ${mine:/home/zhangxs/hive/execlog}/${user.name}
启动步骤
1:首次启动hive需要初始化
使用schematool初始(要保证hadoop集群已经启动)
schematool -dbType mysql -initSchema
2:启动hive服务
[ hive --service metastore]
3:新打开一个终端 输入
hive
遇到问题
1:这个错误是因为,我在hive-site.xml文件里配置的是以root用户登录的,mysql不予许hive远程root登录,你可以配置一下权限,或者换一个普通的用户登录
Caused by: java.sql.SQLException: Access denied for user 'root'@'hadoop-node4.com' (using password: YES) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:964) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3970) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3906) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:873) at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1710) at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1226) at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2253) at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2284) at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2083) at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:806) at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:525) at com.mysql.jdbc.Util.handleNewInstance(Util.java:425) at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:410) at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:328) at java.sql.DriverManager.getConnection(DriverManager.java:579) at java.sql.DriverManager.getConnection(DriverManager.java:190) at com.jolbox.bonecp.BoneCP.obtainRawInternalConnection(BoneCP.java:361) at com.jolbox.bonecp.BoneCP.<init>(BoneCP.java:416)
问题2:hive在第一次登录的时候使用 schematool -dbType mysql -initSchema命令初始化,以后则不用。使用
hive --service metastore启动服务即可
Logging initialized using configuration in file:/usr/local/development/hive-2.1.0/conf/hive-log4j2.properties Async: true Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:578) at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:518) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
问题3:这个是因为我修改hive-site.xml文件时候,值替换了system:java.io.tmpdir一部分,因该将
这整个value里面的值都替换掉
Logging initialized using configuration in file:/usr/local/development/hive-2.1.0/conf/hive-log4j2.properties Async: true Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. tez, spark) or using Hive 1.X releases. hive> show tables; FAILED: IllegalArgumentException java.net.URISyntaxException: Relative path in absolute URI: file:./$%7B/home/zhangxs/data/hive/tmp%7D/$%7Bsystem:user.name%7D/dfb16330-2bbc-481a-bbdc-e4503c4d1a5f/hive_2016-12-12_05-12-52_963_7246570582065346726-1 hive> show tables; FAILED: IllegalArgumentException java.net.URISyntaxException: Relative path in absolute URI: file:./$%7B/home/zhangxs/data/hive/tmp%7D/$%7Bsystem:user.name%7D/dfb16330-2bbc-481a-bbdc-e4503c4d1a5f/hive_2016-12-12_05-14-22_015_4353303889983385851-1
在启动schematool -dbType mysql -initSchema或者hive --service metastore的时候命令后,应该在重新打开一个窗口执行hive命令,要不然,会卡在
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Logging initialized using configuration in file:/usr/local/development/hive-2.1.0/conf/hive-log4j2.properties Async: true ^Z [2]+ Stopped hive