Hive的MoveTask错误

最近在部署Hive上线，结果在线上线下同时出现了MoveTask报错的现象，虽然两者错误的日志以及错误信息一样，但是经过分析解决又发现两者的原因是不一样的。

首先线下的错误日志：

2015-05-18 18:53:09,679 ERROR [main]: exec.Task (SessionState.java:printError(833)) - Failed with exception Unable to rename: hdfs://hadoop-master:9000/tmp/hive/hadoop/4d905c9f-ee65-4b1f-be96-93115b3aad61/hive_2015-05-18_18-51-42_401_2711668916550397051-1/-ext-10000 to: /user/hive/partitions/users_statis/dt_user_statis_behavior/event=play/period=0
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename: hdfs://hadoop-master:9000/tmp/hive/hadoop/4d905c9f-ee65-4b1f-be96-93115b3aad61/hive_2015-05-18_18-51-42_401_2711668916550397051-1/-ext-10000 to: /user/hive/partitions/users_statis/dt_user_statis_behavior/event=play/period=0
    at org.apache.hadoop.hive.ql.exec.MoveTask.moveFile(MoveTask.java:111)
    at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:213)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1604)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1364)
    at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1177)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1004)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:994)
    at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:247)
    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:199)
    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:410)
    at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:783)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:616)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

这个问题在http://blog.csdn.net/lucien_zong/article/details/10198533中有详细的描述

执行SQL时，最后一个任务是MoveTask,它的作用是将运行SQL生成的Mapeduce任务结果文件放到SQL中指定的存储查询结果的路径中，具体方法就是重命名
下面是 org.apache.hadoop.hive.ql.exec.MoveTask 中对结果文件重命名的一段代码：

//这个sourcePath参数就是存放Mapeduce结果文件的目录，所以它的值可能是
//hdfs://indigo:8020/tmp/hive-root/hive_2013-08-22_18-42-03_218_2856924886757165243/-ext-10000
if (fs.exists(sourcePath)) {
    Path deletePath = null;
    // If it multiple level of folder are there fs.rename is failing so first
    // create the targetpath.getParent() if it not exist
    if (HiveConf.getBoolVar(conf, HiveConf.ConfVars.HIVE_INSERT_INTO_MULTILEVEL_DIRS)) {
    deletePath = createTargetPath(targetPath, fs);
    }
    //这里targetPath的值就是指定的放置结果文件的目录，值可能是 result/userName154122639/4e574b5d9f894a70b074ccd3981ca0f1
    if (!fs.rename(sourcePath, targetPath)) {//上面产生的异常就是因为这里rename失败，进了if，throw了个异常
      try {
        if (deletePath != null) {
          fs.delete(deletePath, true);
        }
      } catch (IOException e) {
        LOG.info("Unable to delete the path created for facilitating rename"
            + deletePath);
      }
      throw new HiveException("Unable to rename: " + sourcePath
          + " to: " + targetPath);
    }
  }

rename的targetPath必须存在。

其实之前已经检查和创建targetPath了：

private Path createTargetPath(Path targetPath, FileSystem fs) throws IOException {
    Path deletePath = null;
    Path mkDirPath = targetPath.getParent();
    if (mkDirPath != null & !fs.exists(mkDirPath)) {
      Path actualPath = mkDirPath;
      while (actualPath != null && !fs.exists(actualPath)) {
        deletePath = actualPath;
        actualPath = actualPath.getParent();
      } <property> <name>hive.insert.into.multilevel.dirs</name> <value>true</value> </property> 
      fs.mkdirs(mkDirPath);
    }
    return deletePath;//返回新创建的最顶层的目录，万一失败用来删除用
  }

Apache出现过这个问题，已经解决掉了
CDH 竟然加了个参数 hive.insert.into.multilevel.dirs，默认是false，意思是我还有这BUG呢哈。
当你被坑了，想打个patch时，会发现改个配置就可以了。
意思是我保留这个BUG，但你要是被坑了也不能说我有BUG，自己改配置好了.
目前还没发现其他地方用到了这个参数，在这里唯一作用就是限制SQL中指定存放结果文件不存在的目录的深度不能大于1.
不过也没发现这有什么好处。

折腾半天，加个配置就可以了:

 <property> <name>hive.insert.into.multilevel.dirs</name> <value>true</value> </property> 

解决完线下的问题满心欢喜的以为可以解决线上的问题，结果发现了不行，仔细查看日志发现原来是因为缺包造成的

2015-05-18 19:22:03,799 ERROR [main]: exec.Task (SessionState.java:printError(861)) - Failed with exception Unable to move source hdfs://hadoop1:9000/tmp/hive/statistics/dt_statistics_content_daily/.hive-staging_hive_2015-05-18_19-11-45_323_132664610162390564-1/-ext-10000 to destination /tmp/hive/statistics/dt_statistics_content_daily
    org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move source hdfs://hadoop1:9000/tmp/hive/statistics/dt_statistics_content_daily/.hive-staging_hive_2015-05-18_19-11-45_323_132664610162390564-1/-ext-10000 to destination /tmp/hive/statistics/dt_statistics_content_daily
        at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2483)
        at org.apache.hadoop.hive.ql.exec.MoveTask.moveFile(MoveTask.java:105)
        at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:222)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1638)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1397)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1183)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1039)
        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:207)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:159)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:370)
        at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:754)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
    Caused by: java.io.IOException: Cannot find DistCp class package: org.apache.hadoop.tools.DistCp
        at org.apache.hadoop.hive.shims.Hadoop23Shims.runDistCp(Hadoop23Shims.java:1123)
        at org.apache.hadoop.hive.common.FileUtils.copy(FileUtils.java:553)
        at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2461)
        ... 21 more

查看源码发现在调用moveFile时候会去调用Hadoop的distcp接口

@Override
  public boolean runDistCp(Path src, Path dst, Configuration conf) throws IOException {
    int rc;

    // Creates the command-line parameters for distcp
    String[] params = {"-update", "-skipcrccheck", src.toString(), dst.toString()};

    try {
      Class clazzDistCp = Class.forName("org.apache.hadoop.tools.DistCp");
      Constructor c = clazzDistCp.getConstructor();
      c.setAccessible(true);
      Tool distcp = (Tool)c.newInstance();
      distcp.setConf(conf);
      rc = distcp.run(params);
    } catch (ClassNotFoundException e) {
      throw new IOException("Cannot find DistCp class package: " + e.getMessage());
    } catch (NoSuchMethodException e) {
      throw new IOException("Cannot get DistCp constructor: " + e.getMessage());
    } catch (Exception e) {
      throw new IOException("Cannot execute DistCp process: " + e, e);
    }

    return (0 == rc);
  }

使用命令发现Hive环境变量中并没有Hadoop-Distcp-2.6.0.jar

 hive -e 'set' | grep distcp 

于是只要把hadoop/share/tools/下的Hadoop-Distcp-2.6.0.jar加载下来就行。

需要说明的是线上的Hive是1.1.0版本，线下的hive是0.14版本。同时需要注意的是线下的move文件的方式是rename，线上则用distcp方式，这跟Hive的版本有关了。