zoukankan      html  css  js  c++  java
  • bigdata_hive_Issue of Vectorization on Parquet table

    When Vectorization is turned on in Hive:
    set hive.vectorized.execution.enabled=true;

    If the involved table is in parquet rather than orc format, you may see below error.
    This error appears in both "tez" and "mr" engine.

    Solution: Disable vectorization.


    Caused by: java.io.IOException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Incompatible Bytes vector column and primitive category VOID
    at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
    at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
    at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:352)
    at org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
    at org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
    at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:115)
    at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:126)
    at org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:113)
    at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:61)
    ... 15 more
    Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Incompatible Bytes vector column and primitive category VOID
    at org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat$VectorizedParquetRecordReader.next(VectorizedParquetInputFormat.java:136)
    at org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat$VectorizedParquetRecordReader.next(VectorizedParquetInputFormat.java:49)
    at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:347)
    ... 21 more
     
  • 相关阅读:
    Scala学习(八)---Scala继承
    Scala学习(九)---文件和正则表达式
    Scala学习(六)---Scala对象
    jemter----命令行模式(cmd)导出测试报告
    js数据类型
    定期任务,如每月清理(SpringBoot Mybatis-Plus)
    定时执行任务(SpringBoot Mybatis-Plus)
    图片服务映射(SpringBoot mybatis-plus)
    图片压缩上传(Springboot mybatis-plus)
    java生成pdf
  • 原文地址:https://www.cnblogs.com/cphmvp/p/6128838.html
Copyright © 2011-2022 走看看