zoukankan      html  css  js  c++  java
  • bigdata_hive_Issue of Vectorization on Parquet table

    When Vectorization is turned on in Hive:
    set hive.vectorized.execution.enabled=true;

    If the involved table is in parquet rather than orc format, you may see below error.
    This error appears in both "tez" and "mr" engine.

    Solution: Disable vectorization.


    Caused by: java.io.IOException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Incompatible Bytes vector column and primitive category VOID
    at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
    at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
    at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:352)
    at org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
    at org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
    at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:115)
    at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:126)
    at org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:113)
    at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:61)
    ... 15 more
    Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Incompatible Bytes vector column and primitive category VOID
    at org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat$VectorizedParquetRecordReader.next(VectorizedParquetInputFormat.java:136)
    at org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat$VectorizedParquetRecordReader.next(VectorizedParquetInputFormat.java:49)
    at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:347)
    ... 21 more
     
  • 相关阅读:
    收藏网站代码
    将博客搬至CSDN
    AsyncTask源码浅析
    【转】Activity的launchMode分析 及 Intent.FLAG_NEW_TASK详解
    【转】Java异常:选择Checked Exception还是Unchecked Exception?
    springAOP基本概念和配置
    Java内部类的使用小结
    数据库索引浅析
    基于LinkedBlockingQueue源码自我实现阻塞队列
    【转】HTTP中Get与Post的区别——是我见过说得最全面的
  • 原文地址:https://www.cnblogs.com/cphmvp/p/6128838.html
Copyright © 2011-2022 走看看