zoukankan      html  css  js  c++  java
  • SparkSql执行hive插入警告Unable to inherit permissions

    执行sparksql,插入hive 时候提示以下警告:

    2021-04-03 00:22:02 WARN  [org.apache.hadoop.hive.shims.HadoopShimsSecure setFullFileStatus():869]- Unable to inherit permissions for file hdfs://xxxx:8020/user/hive/warehouse/pdw_ls.db/relatives_ztry/part-00000 
    from file hdfs:
    //xxxx:8020/user/hive/warehouse/pdw_ls.db/relatives_ztry Permission denied. user=ccccc is not the owner of inode=part-00000 at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkOwner(DefaultAuthorizationProvider.java:188) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:174) at org.apache.sentry.hdfs.SentryAuthorizationProvider.checkPermission(SentryAuthorizationProvider.java:194) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:152) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3877) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3860) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkOwner(FSDirectory.java:3825) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOwner(FSNamesystem.java:6784) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setAcl(FSNamesystem.java:9296) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setAcl(NameNodeRpcServer.java:1642) at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.setAcl(AuthorizationProviderProxyClientProtocol.java:902) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setAcl(ClientNamenodeProtocolServerSideTranslatorPB.java:1347) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2281) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2277) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2275) 2021-04-03 00:22:02 WARN [org.apache.hadoop.hive.shims.HadoopShimsSecure setFullFileStatus():869]- Unable to inherit permissions for file hdfs://xxxx:8020/user/hive/warehouse/pdw_ls.db/relatives_ztry/part-00004 from file hdfs://xxxx:8020/user/hive/warehouse/pdw_ls.db/relatives_ztry Permission denied. user=ccccc is not the owner of inode=part-00004 at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkOwner(DefaultAuthorizationProvider.java:188) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:174) at org.apache.sentry.hdfs.SentryAuthorizationProvider.checkPermission(SentryAuthorizationProvider.java:194) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:152) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3877) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3860)

    处理方式:

    设置hive参数 hive.warehouse.subdir.inherit.perms:false,关闭hive的文件权限继承来规避该问题

    SparkConf sparkConf = new SparkConf();
    SparkSession session= SparkSession
                    .builder()
                    .config(sparkConf)
                    .config("hive.warehouse.subdir.inherit.perms","false")
                    .enableHiveSupport()
                    .getOrCreate();
  • 相关阅读:
    神经网络学习笔记(2)
    从机器学习到深度学习资料整理
    在进行机器学习建模时,为什么需要验证集(validation set)?
    Python-绘制3D柱形图
    MIT FiveK图像转化--DNG到TIFF,TIFF到JPEG
    MIT-Adobe FiveK Dataset 图片自动下载
    matplotlib基础
    Numpy基础
    一个电磁感应小实验
    windows 代理服务器的搭建,提供Android 端访问公网.
  • 原文地址:https://www.cnblogs.com/linkmust/p/14626623.html
Copyright © 2011-2022 走看看