zoukankan      html  css  js  c++  java
  • bigdata_hive_Issue of Vectorization on Parquet table

    When Vectorization is turned on in Hive:
    set hive.vectorized.execution.enabled=true;

    If the involved table is in parquet rather than orc format, you may see below error.
    This error appears in both "tez" and "mr" engine.

    Solution: Disable vectorization.


    Caused by: java.io.IOException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Incompatible Bytes vector column and primitive category VOID
    at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
    at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
    at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:352)
    at org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
    at org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
    at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:115)
    at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:126)
    at org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:113)
    at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:61)
    ... 15 more
    Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Incompatible Bytes vector column and primitive category VOID
    at org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat$VectorizedParquetRecordReader.next(VectorizedParquetInputFormat.java:136)
    at org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat$VectorizedParquetRecordReader.next(VectorizedParquetInputFormat.java:49)
    at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:347)
    ... 21 more
     
  • 相关阅读:
    电子证书 DER & PEM & CRT & CER
    更新显卡驱动
    隐马尔可夫模型拼音汉字输入法
    隐马尔可夫模型:HMM
    numpy 随机产生数字
    python(62):保留两位小数
    训练的时候 nvidia:自动刷新
    python(61):str 和 bytes 转换
    tensorflow 笔记12:函数区别:placeholder,variable,get_variable,参数共享
    tensorflow 笔记11:tf.nn.dropout() 的使用
  • 原文地址:https://www.cnblogs.com/cphmvp/p/6128838.html
Copyright © 2011-2022 走看看