zoukankan      html  css  js  c++  java
  • 【原创】大数据基础之Drill(2)Drill1.14+Hive2.1.1运行

    问题

    Drill最新版本是1.14,从1.13开始Drill支持hive的版本升级到2.3.2,详见1.13的release notes

    • The Hive client for Drill is updated to version 2.3.2. With the update, Drill supports queries on transactional (ACID) and non-transactional Hive bucketed ORC tables. The updated libraries are backward compatible with earlier versions of the Hive server and metastore. (DRILL-5978)

    强行使用Drill1.14连接Hive2.1.1会由于metastore thrift接口变化导致问题,具体体现为 show tables是空,具体报错如下:

    2018-10-10 13:03:54,355 [244277c5-ba8c-b6c8-8f99-2cdde9f3c4d8:frag:0:0] WARN  o.a.d.e.s.h.DrillHiveMetaStoreClient - Failure while attempting to get hive table. Retries once.

    org.apache.thrift.TApplicationException: Invalid method name: 'get_table_req'

            at org.apache.thrift.TApplicationException.read(TApplicationException.java:111) ~[drill-hive-exec-shaded-1.14.0.jar:1.14.0]

            at org.apache.drill.exec.store.hive.DrillHiveMetaStoreClient$TableLoader.load(DrillHiveMetaStoreClient.java:531) [drill-storage-hive-core-1.14.0.jar:1.14.0]

            at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) [guava-18.0.jar:na]

            at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) [guava-18.0.jar:na]

            at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) [guava-18.0.jar:na]

            at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) [guava-18.0.jar:na]

            at com.google.common.cache.LocalCache.get(LocalCache.java:3937) [guava-18.0.jar:na]

            at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) [guava-18.0.jar:na]

            at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) [guava-18.0.jar:na]

            at org.apache.drill.exec.store.hive.DrillHiveMetaStoreClient$HiveClientWithCaching.getHiveReadEntry(DrillHiveMetaStoreClient.java:495) [drill-storage-hive-core-1.14.0.jar:1.14.0]

            at org.apache.drill.exec.store.hive.schema.HiveSchemaFactory$HiveSchema.getSelectionBaseOnName(HiveSchemaFactory.java:230) [drill-storage-hive-core-1.14.0.jar:1.14.0]

            at org.apache.drill.exec.store.hive.schema.HiveSchemaFactory$HiveSchema.getDrillTable(HiveSchemaFactory.java:210) [drill-storage-hive-core-1.14.0.jar:1.14.0]

            at org.apache.drill.exec.store.hive.schema.HiveDatabaseSchema.getTable(HiveDatabaseSchema.java:62) [drill-storage-hive-core-1.14.0.jar:1.14.0]

            at org.apache.drill.exec.store.AbstractSchema.getTablesByNames(AbstractSchema.java:239) [drill-java-exec-1.14.0.jar:1.14.0]

            at org.apache.drill.exec.store.AbstractSchema.getTableNamesAndTypes(AbstractSchema.java:257) [drill-java-exec-1.14.0.jar:1.14.0]

            at org.apache.drill.exec.store.ischema.InfoSchemaRecordGenerator$Tables.visitTables(InfoSchemaRecordGenerator.java:301) [drill-java-exec-1.14.0.jar:1.14.0]

            at org.apache.drill.exec.store.ischema.InfoSchemaRecordGenerator.scanSchema(InfoSchemaRecordGenerator.java:216) [drill-java-exec-1.14.0.jar:1.14.0]

            at org.apache.drill.exec.store.ischema.InfoSchemaRecordGenerator.scanSchema(InfoSchemaRecordGenerator.java:209) [drill-java-exec-1.14.0.jar:1.14.0]

            at org.apache.drill.exec.store.ischema.InfoSchemaRecordGenerator.scanSchema(InfoSchemaRecordGenerator.java:209) [drill-java-exec-1.14.0.jar:1.14.0]

            at org.apache.drill.exec.store.ischema.InfoSchemaRecordGenerator.scanSchema(InfoSchemaRecordGenerator.java:196) [drill-java-exec-1.14.0.jar:1.14.0]

            at org.apache.drill.exec.store.ischema.InfoSchemaTableType.getRecordReader(InfoSchemaTableType.java:58) [drill-java-exec-1.14.0.jar:1.14.0]

            at org.apache.drill.exec.store.ischema.InfoSchemaBatchCreator.getBatch(InfoSchemaBatchCreator.java:34) [drill-java-exec-1.14.0.jar:1.14.0]

            at org.apache.drill.exec.store.ischema.InfoSchemaBatchCreator.getBatch(InfoSchemaBatchCreator.java:30) [drill-java-exec-1.14.0.jar:1.14.0]

            at org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:159) [drill-java-exec-1.14.0.jar:1.14.0]

            at org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:182) [drill-java-exec-1.14.0.jar:1.14.0]

            at org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:137) [drill-java-exec-1.14.0.jar:1.14.0]

            at org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:182) [drill-java-exec-1.14.0.jar:1.14.0]

            at org.apache.drill.exec.physical.impl.ImplCreator.getRootExec(ImplCreator.java:110) [drill-java-exec-1.14.0.jar:1.14.0]

            at org.apache.drill.exec.physical.impl.ImplCreator.getExec(ImplCreator.java:87) [drill-java-exec-1.14.0.jar:1.14.0]

            at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:261) [drill-java-exec-1.14.0.jar:1.14.0]

            at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.14.0.jar:1.14.0]

            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_60]

            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_60]

            at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]

    编译

    于是尝试重新编译Drill1.14,将依赖的hive版本降到2.1.1,下载代码

    http://mirror.bit.edu.cn/apache/drill/drill-1.14.0/apache-drill-1.14.0-src.tar.gz

    POM

    修改pom中的hive版本

    <hive.version>2.3.2</hive.version>

    修改为<hive.version>2.1.1</hive.version>

    重新编译打包后发现问题依旧,经检查发现修改版本之后只有jars/3rdparty下的3个hive相关jar从2.3.2改为2.1.1

    hive-contrib-2.1.1.jar
    hive-hbase-handler-2.1.1.jar
    hive-metastore-2.1.1.jar

    报错的jar是drill-hive-exec-shaded-1.14.0.jar,这个jar包中包含包含hive-exec及依赖,

            <artifactId>maven-shade-plugin</artifactId>

            <configuration>

              <artifactSet>

                <includes>

                  <include>org.apache.hive:hive-exec</include>

    并且没有使用配置的hive.version

        <dependency>

          <groupId>org.apache.hive</groupId>

          <artifactId>hive-exec</artifactId>

          <scope>compile</scope>

    导致打进jar包中的hive-exec是2.3.2版本的,增加hive.version配置

        <dependency>

          <groupId>org.apache.hive</groupId>

          <artifactId>hive-exec</artifactId>

          <version>${hive.version}</version>

          <scope>compile</scope>

    再打包,问题消失,show tables正常;

    Hadoop Location

    官方文档说明如下:

    Apache Drill users must tell Drill-on-YARN the location of your Hadoop install. Set the HADOOP_HOME environment variable in $DRILL_SITE/drillenv.sh to point to your Hadoop installation:

    export HADOOP_HOME= /path/to/hadoop-home  

    但配置之后依然存在问题:

    1)报错

    Diagnostics: File file:/user/drill/site.tar.gz does not exist
    java.io.FileNotFoundException: File file:/user/drill/site.tar.gz does not exist

    需要添加link

    ln -s $HADOOP_HOME/etc/hadoop/core-site.xml $DRILL_SITE/core-site.xml

    2)在实际查询时会报错找不到hdfs_name,需要添加link

    ln -s $HADOOP_HOME/etc/hadoop/hdfs-site.xml $DRILL_SITE/hdfs-site.xml

  • 相关阅读:
    某公司面试java试题之【二】,看看吧,说不定就是你将要做的题
    BAT面试上机题从3亿个ip中找出访问次数最多的IP详解
    Android 最流行的吸顶效果的实现及代码
    postgresql某进程占用cpu资源过高,降不下来
    媳妇要转java开发,我该怎么办?
    【原创】微信公众号电影网站域名被屏蔽完美解决方案!
    java如何对List集合中的元素进行排序(请收藏)
    【转】多语言的正则表达式,我们应该掌握
    【转】app之YdbOnline说明文档
    [敏捷开发实践](0) 开始
  • 原文地址:https://www.cnblogs.com/barneywill/p/10109644.html
Copyright © 2011-2022 走看看