zoukankan      html  css  js  c++  java
  • SparkSql执行hive插入警告Unable to inherit permissions

    执行sparksql,插入hive 时候提示以下警告:

    2021-04-03 00:22:02 WARN  [org.apache.hadoop.hive.shims.HadoopShimsSecure setFullFileStatus():869]- Unable to inherit permissions for file hdfs://xxxx:8020/user/hive/warehouse/pdw_ls.db/relatives_ztry/part-00000 
    from file hdfs:
    //xxxx:8020/user/hive/warehouse/pdw_ls.db/relatives_ztry Permission denied. user=ccccc is not the owner of inode=part-00000 at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkOwner(DefaultAuthorizationProvider.java:188) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:174) at org.apache.sentry.hdfs.SentryAuthorizationProvider.checkPermission(SentryAuthorizationProvider.java:194) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:152) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3877) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3860) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkOwner(FSDirectory.java:3825) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOwner(FSNamesystem.java:6784) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setAcl(FSNamesystem.java:9296) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setAcl(NameNodeRpcServer.java:1642) at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.setAcl(AuthorizationProviderProxyClientProtocol.java:902) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setAcl(ClientNamenodeProtocolServerSideTranslatorPB.java:1347) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2281) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2277) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2275) 2021-04-03 00:22:02 WARN [org.apache.hadoop.hive.shims.HadoopShimsSecure setFullFileStatus():869]- Unable to inherit permissions for file hdfs://xxxx:8020/user/hive/warehouse/pdw_ls.db/relatives_ztry/part-00004 from file hdfs://xxxx:8020/user/hive/warehouse/pdw_ls.db/relatives_ztry Permission denied. user=ccccc is not the owner of inode=part-00004 at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkOwner(DefaultAuthorizationProvider.java:188) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:174) at org.apache.sentry.hdfs.SentryAuthorizationProvider.checkPermission(SentryAuthorizationProvider.java:194) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:152) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3877) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3860)

    处理方式:

    设置hive参数 hive.warehouse.subdir.inherit.perms:false,关闭hive的文件权限继承来规避该问题

    SparkConf sparkConf = new SparkConf();
    SparkSession session= SparkSession
                    .builder()
                    .config(sparkConf)
                    .config("hive.warehouse.subdir.inherit.perms","false")
                    .enableHiveSupport()
                    .getOrCreate();
  • 相关阅读:
    redis持久化之AOF和RDB
    pycharm快捷键
    500系列错误
    400系列错误
    Linux常用命令大全
    从浏览器输入一个地址,发送请求,经历了哪些过程(面试问题)
    401的错误分析
    301和302的区别
    数据库 语法收集 (主 mysql,次oracle,日常积累)
    python 发送邮件 smtplib.SMTPAuthenticationError: (535, b'Error: authentication failed') 以及授权码设置步骤
  • 原文地址:https://www.cnblogs.com/linkmust/p/14626623.html
Copyright © 2011-2022 走看看