zoukankan      html  css  js  c++  java
  • ambari-环境调参

    1.Apache Tez

    (Tez is the next generation Hadoop Query Processing framework written on top of YARN. 它是一款基于YARN的新一代数据处理和分析框架,亮点:支持DAG调度作业,它能把多个有依赖的作业转换成一个作业,所以能大幅提升DAG性能,从而提升M-R查询性能)

    tez.am.resource.memory.mb=1024
    tez.task.resource.memory.mb=1024
    tez.runtime.io.sort.mb=400

    1.1 现象一

    ERROR : Failed to execute tez graph.
    org.apache.tez.dag.api.TezException: org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid resource request, requested resource type=[memory-mb] < 0 or greater than maximum allowed allocation. Requested resource=<memory:8192, vCores:1>, maximum allowed allocation=<memory:4096, vCores:32>, please note that maximum allowed allocation is calculated by scheduler based on maximum resource of registered NodeManagers, which might be less than configured maximum allocation=<memory:4096, vCores:32>
    原因是:
    ambari中设置Tez过大,下面两个参数被设置成了8192,启动时yarn单个container内存申请资源为8192,导致所需资源过大
    tez.am.resource.memory.mb=1024

    tez.task.resource.memory.mb=1024

    1.2 现象二

    设置的tez.runtime.io.sort.mb=512,大于设计的410MB

    tez.runtime.io.sort.mb 512 should be larger than 0 and should be less than 410MB

    1.3 现象三  

    当container以512MB启动时,执行过程内存超出
    Container [pid=27426,containerID=container_e07_1596508490512_0184_02_000001] is running 17154048B beyond the 'PHYSICAL' memory limit. Current usage: 528.4 MB of 512 MB physical memory used; 2.4 GB of 1.0 GB virtual memory used. Killing container.

    1.4 现象四

    yarn资源不足,可以调整 yarn.scheduler.capacity.maximum-am-resource-percent=0.5

     That your drivers have exceeded max memory configured in Max Application Master Resources. You can either increase max memory for AM or decrease driver memory in your jobs.

    Application is added to the scheduler and is not yet activated. Queue's AM resource limit exceeded. Details : AM Partition = 3ssystem; AM Resource Request = <memory:4096, vCores:1>; Queue Resource Limit for AM = <memory:6144, vCores:3>; User AM Resource Limit of the queue = <memory:6144, vCores:3>; Queue AM Resource Usage = <memory:4096, vCores:1>;
    

    1.5 现象五 hivesever2-site的权限验证

     去掉验证

      

    1.6 现象六 hdfs的权限问题

    dfs.permissions.enabled=false

  • 相关阅读:
    day26
    day 25
    java.io.IOException: java.net.ConnectException: Call From master/192.168.58.128 to master:10020 failed on connection exception: java.net.ConnectException: 拒绝连接;
    疫情可视化系统
    使用eclipse创建spring cloud的eureka客户端和eureka服务端
    连接虚拟机的hive时进程自动杀死
    在Ubuntu18.04的Docker中安装Oracle镜像及简单使用
    Ubuntu16.04 上Docker 中安装SQL Server 2017
    docker
    Docker镜像报错
  • 原文地址:https://www.cnblogs.com/long-yuan/p/13446741.html
Copyright © 2011-2022 走看看