zoukankan      html  css  js  c++  java
  • Hive 使用Tez引擎的配置

    前提

    Hive  需要是 2.0以上版本

    Tez配置

    下载好tar包后,将tar包上传到HDFS集群路径下

    hadoop fs -put /opt/software/apache-tez-0.9.1-bin.tar.gz/ /tez

    将Linux本机上的tar包解压

    Hive配置

    在hive/conf下新建tez-site.xml文件,并添加以下参数

    <?xml version="1.0" encoding="UTF-8"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
    <configuration>
    <!--指明hdfs集群上的tez的tar包,使Hadoop可以自动分布式缓存该jar包-->
        <property>
            <name>tez.lib.uris</name>
            <value>${fs.defaultFS}/tez/apache-tez-0.9.1-bin.tar.gz</value>
    </property>
    <!--tez是否可用Hadoop的jar包 -->
    
        <property>
             <name>tez.use.cluster.hadoop-libs</name>
             <value>true</value>
    </property>
    <!--以何种形式形成日志文件 -->
    
        <property>
             <name>tez.history.logging.service.class</name>        
             <value>org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService</value>
        </property>
    </configuration>

    在hive-env.sh文件中添加tez环境变量配置和依赖包环境变量配置

    # Set HADOOP_HOME to point to a specific hadoop install directory
    export HADOOP_HOME=/opt/module/hadoop-2.7.2
    
    # Hive Configuration Directory can be controlled by:
    export HIVE_CONF_DIR=/opt/module/hive/conf
    
    # Folder containing extra libraries required for hive compilation/execution can be controlled by:
    export TEZ_HOME=/opt/module/tez-0.9.1    #是你的tez的解压目录
    export TEZ_JARS=""
    for jar in `ls $TEZ_HOME |grep jar`; do
        export TEZ_JARS=$TEZ_JARS:$TEZ_HOME/$jar
    done
    for jar in `ls $TEZ_HOME/lib`; do
        export TEZ_JARS=$TEZ_JARS:$TEZ_HOME/lib/$jar
    done
    
    export HIVE_AUX_JARS_PATH=/opt/module/hadoop-2.7.2/share/hadoop/common/hadoop-lzo-0.4.20.jar$TEZ_JARS

    在hive-site.xml文件中添加如下配置,更改hive计算引擎

    <property>
        <name>hive.execution.engine</name>
        <value>tez</value>
    </property>

    yarn配置

    如果运行时报错

    Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1546781144082_0005 failed 2 times due to AM Container for appattempt_1546781144082_0005_000002 exited with  exitCode: -103
    For more detailed output, check application tracking page:http://hadoop103:8088/cluster/app/application_1546781144082_0005Then, click on links to logs of each attempt.
    Diagnostics: Container [pid=11116,containerID=container_1546781144082_0005_02_000001] is running beyond virtual memory limits. Current usage: 216.3 MB of 1 GB physical memory used; 2.6 GB of 2.1 GB virtual memory used. Killing container.

    修改yarn-site.xml,关掉虚拟内存检查

    <property>
        <name>yarn.nodemanager.vmem-check-enabled</name>
        <value>false</value>
    </property>
  • 相关阅读:
    Oracle 推出 ODAC for Entity Framework 和 LINQ to Entities Beta版
    Entity Framework Feature CTP 5系列文章
    MonoDroid相关资源
    MSDN杂志上的Windows Phone相关文章
    微软学Android Market推出 Web Windows Phone Marketplace
    使用 Visual Studio Agent 2010 进行负载压力测试的安装指南
    MonoMac 1.0正式发布
    Shawn Wildermuth的《Architecting WP7 》系列文章
    使用.NET Mobile API即51Degrees.mobi检测UserAgent
    MongoDB 客户端 MongoVue
  • 原文地址:https://www.cnblogs.com/yangxusun9/p/12609041.html
Copyright © 2011-2022 走看看