Spark初始换HiveContext空指针异常

spark程序初始化HiveContext报错：空指针异常

版本说明：

	spark: 1.6.1
	scala: 2.10.8
	hive: 1.2.1

报错信息

	java.lang.RuntimeException: java.lang.NullPointerException
		at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
		at org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:204)
		at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:238)
		at org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:218)
		at org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:208)
		at org.apache.spark.sql.hive.HiveContext.setConf(HiveContext.scala:440)
		at org.apache.spark.sql.SQLContext$$anonfun$4.apply(SQLContext.scala:272)
		at org.apache.spark.sql.SQLContext$$anonfun$4.apply(SQLContext.scala:271)
		at scala.collection.Iterator$class.foreach(Iterator.scala:727)
		at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
		at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
		at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
		at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:271)
		at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:90)
		at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:101)
		at com.winner.clu.spark.batch.analysis.AccPresetConditionData.mainFun(AccPresetConditionData.scala:60)
		at com.winner.clu.spark.batch.BatchJobMain$.main(BatchJobMain.scala:53)
		at com.winner.clu.spark.batch.BatchJobMain.main(BatchJobMain.scala)

问题描述

idea本地开发环境进行调试程序，在通过 val htx=new HiveContext(sc)的时候，包空指针异常

问题原因

追踪源码的时候发现，在进行到下面这一步(创建本地临时目录)的时候，进行抛出异常

private void createPath(HiveConf conf, Path path, String permission, boolean isLocal, boolean isCleanUp) throws IOException {
        FsPermission fsPermission = new FsPermission(permission);
        Object fs;
        if (isLocal) {
            fs = FileSystem.getLocal(conf);
        } else {
            fs = path.getFileSystem(conf);
        }

        if (!((FileSystem)fs).exists(path)) {
            ((FileSystem)fs).mkdirs(path, fsPermission);
            String dirType = isLocal ? "local" : "HDFS";
            LOG.info("Created " + dirType + " directory: " + path.toString());
        }

        if (isCleanUp) {
            ((FileSystem)fs).deleteOnExit(path);
        }

    }

并且在程序启动的时候会由于本地hadoop/bin目录下没有winutils.exe而报错，故而想到可能又由于该问题引起的

处理方案

下载winutils.exe文件，copy到本地hadoop/bin目录下，如下图：

然后重新运行IDEA，问题解决

参考