zoukankan      html  css  js  c++  java
  • 编写一个基于HBase的MR程序,结果遇到一个错:ERROR security.UserGroupInformation

    环境说明:Ubuntu12.04,使用CDH4.5,伪分布式环境

    Hadoop配置如下:

    core-site.xml:

    <configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
    <property>
        <name>fs.trash.interval</name>
        <value>10080</value>
    </property>
    <property>
        <name>fs.trash.checkpoint.interval</name>
        <value>10080</value>
    </property>
    </configuration>

    yarn-site.xml:

    <?xml version="1.0"?>
    <configuration>

    <!-- Site specific YARN configuration properties -->
    <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>localhost:8031</value>
    </property>
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>localhost:8032</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>localhost:8030</value>
    </property>
    <property>
        <name>yarn.resourcemanager.admin.address</name>
        <value>localhost:8033</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>localhost:8088</value>
    </property>
    <property>
        <description>Classpath for typical applications.</description>
        <name>yarn.application.classpath</name>
        <value>$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/share/hadoop/common/*,$HADOOP_COMMON_HOME/share/hadoop/common/lib/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,$YARN_HOME/share/hadoop/yarn/*,$YARN_HOME/share/hadoop/yarn/lib/*,$YARN_HOME/share/hadoop/mapreduce/*,$YARN_HOME/share/hadoop/mapreduce/lib/*</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce.shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
    <property>
        <name>yarn.nodemanager.local-dirs</name>
        <value>/home/hadoop/CDH4Dev/yarn/local</value>
    </property>
    <property>
        <name>yarn.nodemanager.log-dirs</name>
        <value>/home/hadoop/CDH4Dev/yarn/logs</value>
    </property>
    <property>
        <description>Where to aggregate logs</description>
        <name>yarn.nodemanager.remote-app-log-dir</name>
        <value>/home/hadoop/CDH4Dev/yarn/logs</value>
    </property>
    <property>
        <name>yarn.app.mapreduce.am.staging-dir</name>
        <value>/userhadoop</value>
    </property>
    </configuration>

    mapred-site.xml:

    <?xml version="1.0"?>
    <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

    <!-- Put site-specific property overrides in this file. -->

    <configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>localhost:10020</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>localhost:19888</value>
    </property>
    </configuration>

    hbase-site.xml

    <configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>localhost:10020</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>localhost:19888</value>
    </property>
    </configuration>

    在eclipse上,运行一个从 HBase指定table的指定行范围读取数据,经过MR后,再写到另一个table中的 MR程序,结果运行时出现:

    2014-03-29 21:07:36,081 WARN  [main] conf.Configuration (Configuration.java:warnOnceIfDeprecated(981)) - mapred.reduce.slowstart.completed.maps is deprecated. Instead, use mapreduce.job.reduce.slowstart.completedmaps
    2014-03-29 21:07:36,081 WARN  [main] conf.Configuration (Configuration.java:warnOnceIfDeprecated(981)) - dfs.secondary.http.address is deprecated. Instead, use dfs.namenode.secondary.http-address
    2014-03-29 21:07:36,082 WARN  [main] conf.Configuration (Configuration.java:warnOnceIfDeprecated(981)) - fs.checkpoint.period is deprecated. Instead, use dfs.namenode.checkpoint.period
    2014-03-29 21:07:36,082 WARN  [main] conf.Configuration (Configuration.java:warnOnceIfDeprecated(981)) - topology.node.switch.mapping.impl is deprecated. Instead, use net.topology.node.switch.mapping.impl
    2014-03-29 21:07:36,082 WARN  [main] conf.Configuration (Configuration.java:warnOnceIfDeprecated(981)) - mapred.tasktracker.tasks.sleeptime-before-sigkill is deprecated. Instead, use mapreduce.tasktracker.tasks.sleeptimebeforesigkill
    2014-03-29 21:07:36,082 WARN  [main] conf.Configuration (Configuration.java:warnOnceIfDeprecated(981)) - mapred.compress.map.output is deprecated. Instead, use mapreduce.map.output.compress
    2014-03-29 21:07:36,082 WARN  [main] conf.Configuration (Configuration.java:warnOnceIfDeprecated(981)) - mapred.merge.recordsBeforeProgress is deprecated. Instead, use mapreduce.task.merge.progress.records
    2014-03-29 21:07:36,082 WARN  [main] conf.Configuration (Configuration.java:warnOnceIfDeprecated(981)) - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
    2014-03-29 21:07:36,083 INFO  [main] mapreduce.TableOutputFormat (TableOutputFormat.java:setConf(203)) - Created table instance for tag_friend
    2014-03-29 21:07:36,086 WARN  [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    2014-03-29 21:07:36,098 WARN  [main] mapreduce.JobSubmitter (JobSubmitter.java:copyAndConfigureFiles(143)) - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
    2014-03-29 21:07:36,503 INFO  [main] mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(416)) - Cleaning up the staging area file:/tmp/hadoop-hadoop/mapred/staging/hadoop1813759830/.staging/job_local1813759830_0001
    2014-03-29 21:07:36,504 ERROR [main] security.UserGroupInformation (UserGroupInformation.java:doAs(1411)) - PriviledgedActionException as:hadoop (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not exist: hdfs://localhost:9000/home/hadoop/CDH4Dev/hbase/lib/zookeeper-3.4.5-cdh4.5.0.jar
    java.io.FileNotFoundException: File does not exist: hdfs://localhost:9000/home/hadoop/CDH4Dev/hbase/lib/zookeeper-3.4.5-cdh4.5.0.jar

    但是,实际上,home/hadoop/CDH4Dev/hbase/lib/zookeeper-3.4.5-cdh4.5.0.jar 是实际存在的。

    并且,上述程序export成jar包后,在hadoop上shell命令,是可以执行的

    无法继续下去了,请 大家指点

  • 相关阅读:
    Codeforces Round #526 (Div. 2) E. The Fair Nut and Strings
    Codeforces Round #526 (Div. 2) D. The Fair Nut and the Best Path
    S2SH项目实现分页功能
    S2SH项目实现分页功能
    HibernateDaoSupport类的使用
    object references an unsaved transient instance
    object references an unsaved transient instance
    启动Tomcat报WEB-INFlibj2ee.jar jar not loaded异常的解决办法
    启动Tomcat报WEB-INFlibj2ee.jar jar not loaded异常的解决办法
    解决eclipse中出现Resource is out of sync with the file system问题
  • 原文地址:https://www.cnblogs.com/lagujw/p/3633124.html
Copyright © 2011-2022 走看看