zoukankan      html  css  js  c++  java
  • 排查Hive报错:org.apache.hadoop.hive.serde2.SerDeException: java.io.IOException: Start of Array expected

    CREATE TABLE json_nested_test (
        count string,
        usage string,
        pkg map<string,string>,
        languages array<string>,
        store map<string,array<map<string,string>>>)
    ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe'
    STORED AS TEXTFILE;

    以上述sql创建表json_nested_test后,查询时发现报错:Failed with exception java.io.IOException:org.apache.hadoop.hive.serde2.SerDeException: java.io.IOException: Start of Array expected

    于是进行如下测试,发现是org.apache.hive.hcatalog.data.JsonSerDe 对复杂类型支持不足造成,例如map<string,array<string>>,这个例子中就是不支持array作为map的value.
    CREATE TABLE json_nested_test_openx (
        count string,
        usage string,
        pkg map<string,string>,
        languages array<string>,
        store map<string,array<map<string,string>>>)
    ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
    STORED AS TEXTFILE;
    
    CREATE TABLE s1 (
        count string,
        usage string
    )
    ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe'
    STORED AS TEXTFILE;
    load data local inpath '/home/work/s1.txt' overwrite into table s1;
    select * from s1;
    CREATE TABLE s2 (
        count string,
        usage string,
        pkg map<string,string>
    )
    ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe'
    STORED AS TEXTFILE;
    load data local inpath '/home/work/s2.txt' overwrite into table s2;
    select * from s2;
    
    CREATE TABLE s3 (
        count string,
        usage string,
        pkg map<string,string>,
        languages array<string>
    )
    ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe'
    STORED AS TEXTFILE;
    load data local inpath '/home/work/s3.txt' overwrite into table s3;
    select * from s3;
    CREATE TABLE s4 (
        count string,
        usage string,
        pkg map<string,string>,
        languages array<string>,
        store map<string,array<map<string,string>>>
    )
    ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe'
    STORED AS TEXTFILE;
    load data local inpath '/home/work/s4.txt' overwrite into table s4;
    select * from s4;
    
    CREATE TABLE s5 (
        store map<string,array<map<string,string>>>
    )
    ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe'
    STORED AS TEXTFILE;
    load data local inpath '/home/work/s5.txt' overwrite into table s5;
    select * from s5;
    
    CREATE TABLE s6 (
        store map<string,array<string>>
    )
    ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe'
    STORED AS TEXTFILE;
    load data local inpath '/home/work/s6.txt' overwrite into table s6;
    select * from s6;

    这个Serde hive自带,路径为$HIVE_HOME/hcatalog/share/hcatalog/hive-hcatalog-core-1.2.1.jar,它也存在其它问题:不支持数据文件中的空行

  • 相关阅读:
    Microsoft .NET Framework 2.0实现发送邮件(Email)总结
    Microsoft .NET Framework 2.0对文件传输协议(FTP)操作(上传,下载,新建,删除,FTP间传送文件等)实现汇总
    抽象类
    WingIDE 单步调试 Uliweb Python 代码
    Android 4.0 SDK的离线方式安装
    .NET 3.5 中WCF客户端代理性能改进以及最佳实践
    在linux上部署Redmine
    认识jQuery mobile 框架,资源,书籍
    如何使用搜索技巧来成为一名高效的程序员
    Management Console 工具管理类软件通用开发框架(开放源码)
  • 原文地址:https://www.cnblogs.com/aprilrain/p/6916359.html
Copyright © 2011-2022 走看看