在Hive的cli中执行show databases;报一下错误
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.util.Time.monotonicNowNanos()J
at org.apache.hadoop.util.Timer.monotonicNowNanos(Timer.java:59)
at org.apache.hadoop.util.StopWatch.start(StopWatch.java:63)
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:321)
at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextSplits(FetchOperator.java:372)
at org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:304)
at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:459)
at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:428)
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:147)
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2208)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:253)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
错误分析
自己在HADOOP的配置目录下的hadoop-env.sh中添加了
# Hive
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/home/duxing/DevTools/bigdata/hive-2.3.6/lib/*
# HBase
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/home/duxing/DevTools/bigdata/hbase-1.2.6/lib/*
其中HBase的lib下有hadoop-common-2.5.1.jar,我的HADOOP版本是2.8.5.
估计是Hive语句执行时调用的jar文件是hadoop-common-2.5.1.jar中的org.apache.hadoop.util.Time类,不是hadoop2.8.5的org.apache.hadoop.util.Time类.
把hadoop2.8.5 的hadoop-common的jar文件与HBase的hadoop-common-2.5.1.jar分别解压后,在idea中反编译.结果分别如下:
- hadoop-common-2.5.1.jar
//
// Source code recreated from a .class file by IntelliJ IDEA
// (powered by Fernflower decompiler)
//
package org.apache.hadoop.util;
import org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate;
import org.apache.hadoop.classification.InterfaceStability.Unstable;
@LimitedPrivate({"HDFS", "MapReduce"})
@Unstable
public final class Time {
public Time() {
}
public static long now() {
return System.currentTimeMillis();
}
public static long monotonicNow() {
long NANOSECONDS_PER_MILLISECOND = 1000000L;
return System.nanoTime() / 1000000L;
}
}
- Hadoop2.8.5的hadoop-common jar文件
//
// Source code recreated from a .class file by IntelliJ IDEA
// (powered by Fernflower decompiler)
//
package org.apache.hadoop.util;
import java.text.SimpleDateFormat;
import org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate;
import org.apache.hadoop.classification.InterfaceStability.Unstable;
@LimitedPrivate({"HDFS", "MapReduce"})
@Unstable
public final class Time {
private static final long NANOSECONDS_PER_MILLISECOND = 1000000L;
private static final ThreadLocal<SimpleDateFormat> DATE_FORMAT = new ThreadLocal<SimpleDateFormat>() {
protected SimpleDateFormat initialValue() {
return new SimpleDateFormat("yyyy-MM-dd HH:mm:ss,SSSZ");
}
};
public Time() {
}
public static long now() {
return System.currentTimeMillis();
}
public static long monotonicNow() {
return System.nanoTime() / 1000000L;
}
public static long monotonicNowNanos() {
return System.nanoTime();
}
public static String formatTime(long millis) {
return ((SimpleDateFormat)DATE_FORMAT.get()).format(millis);
}
}
对比发现hadoop-common-2.5.1.jar的Time中没有monotonicNowNanos方法
解决方法
把自己的hadoop配置目录下的hadoop-env.sh的
# Hive
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/home/duxing/DevTools/bigdata/hive-2.3.6/lib/*
# HBase
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/home/duxing/DevTools/bigdata/hbase-1.2.6/lib/*
替换为
# Hive
# export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/home/duxing/DevTools/bigdata/hive-2.3.6/lib/*
# HBase
# export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/home/duxing/DevTools/bigdata/hbase-1.2.6/lib/*
总结
不要在Classpath里边放有版本冲突的jar包,但是大数据框架之间的依赖关系注定这是不太可能的.只好自己注意这类的问题了.