zoukankan      html  css  js  c++  java
  • bigdata_hive_Issue of Vectorization on Parquet table

    When Vectorization is turned on in Hive:
    set hive.vectorized.execution.enabled=true;

    If the involved table is in parquet rather than orc format, you may see below error.
    This error appears in both "tez" and "mr" engine.

    Solution: Disable vectorization.


    Caused by: java.io.IOException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Incompatible Bytes vector column and primitive category VOID
    at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
    at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
    at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:352)
    at org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
    at org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
    at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:115)
    at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:126)
    at org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:113)
    at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:61)
    ... 15 more
    Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Incompatible Bytes vector column and primitive category VOID
    at org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat$VectorizedParquetRecordReader.next(VectorizedParquetInputFormat.java:136)
    at org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat$VectorizedParquetRecordReader.next(VectorizedParquetInputFormat.java:49)
    at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:347)
    ... 21 more
     
  • 相关阅读:
    学习:多项式算法----FWT
    学习:多项式算法----FFT
    学习:STL----优先队列
    Python中pip安装包出现Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection
    ELK集群redis服务因数据量太大导致内存使用率暴涨故障修复
    zabbix 历史数据存入elasticsearch
    elk+filebeat+redis日志系统部署
    Linux 服务守护脚本
    Nginx 配置临时维护页
    Linux DDos防御
  • 原文地址:https://www.cnblogs.com/cphmvp/p/6128838.html
Copyright © 2011-2022 走看看