zoukankan      html  css  js  c++  java
  • SparkSql执行hive插入警告Unable to inherit permissions

    执行sparksql,插入hive 时候提示以下警告:

    2021-04-03 00:22:02 WARN  [org.apache.hadoop.hive.shims.HadoopShimsSecure setFullFileStatus():869]- Unable to inherit permissions for file hdfs://xxxx:8020/user/hive/warehouse/pdw_ls.db/relatives_ztry/part-00000 
    from file hdfs:
    //xxxx:8020/user/hive/warehouse/pdw_ls.db/relatives_ztry Permission denied. user=ccccc is not the owner of inode=part-00000 at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkOwner(DefaultAuthorizationProvider.java:188) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:174) at org.apache.sentry.hdfs.SentryAuthorizationProvider.checkPermission(SentryAuthorizationProvider.java:194) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:152) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3877) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3860) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkOwner(FSDirectory.java:3825) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOwner(FSNamesystem.java:6784) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setAcl(FSNamesystem.java:9296) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setAcl(NameNodeRpcServer.java:1642) at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.setAcl(AuthorizationProviderProxyClientProtocol.java:902) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setAcl(ClientNamenodeProtocolServerSideTranslatorPB.java:1347) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2281) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2277) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2275) 2021-04-03 00:22:02 WARN [org.apache.hadoop.hive.shims.HadoopShimsSecure setFullFileStatus():869]- Unable to inherit permissions for file hdfs://xxxx:8020/user/hive/warehouse/pdw_ls.db/relatives_ztry/part-00004 from file hdfs://xxxx:8020/user/hive/warehouse/pdw_ls.db/relatives_ztry Permission denied. user=ccccc is not the owner of inode=part-00004 at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkOwner(DefaultAuthorizationProvider.java:188) at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:174) at org.apache.sentry.hdfs.SentryAuthorizationProvider.checkPermission(SentryAuthorizationProvider.java:194) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:152) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3877) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:3860)

    处理方式:

    设置hive参数 hive.warehouse.subdir.inherit.perms:false,关闭hive的文件权限继承来规避该问题

    SparkConf sparkConf = new SparkConf();
    SparkSession session= SparkSession
                    .builder()
                    .config(sparkConf)
                    .config("hive.warehouse.subdir.inherit.perms","false")
                    .enableHiveSupport()
                    .getOrCreate();
  • 相关阅读:
    JavaScript内置对象
    微信小程序:实现可拖动悬浮图标(包括按钮角标的实现)
    项目1——博客系统
    在HTML中使用css3实现雪人动画效果
    Ajax请求后台发送String类型总是进入Error function解决办法总结
    原生js实现图片懒加载
    【Electron Playground 系列】自定义协议篇
    大规格文件的上传优化
    如何优雅的在react-hook中进行网络请求
    noip2008 双栈排序
  • 原文地址:https://www.cnblogs.com/linkmust/p/14626623.html
Copyright © 2011-2022 走看看