Windows使用Idea编译spark源码

 1. 环境准备

JDK1.8

Scala2.11.8

Maven 3.3+

IDEA with scala plugin

2. 下载spark源码

下载地址 https://archive.apache.org/dist/spark/spark-2.0.0/spark-2.0.0.tgz

将下载好的spark源码解压到c:workspace

3. Idea 导入spark-2.0.0源码工程

 

一路next下去，最后点击finish。

 

 最后，进行项目的编译

4. 可能遇到的问题

4.1 not found: type SparkFlumeProtocol

sparkexternalflume-sinksrcmainscalaorgapachesparkstreamingflumesinkSparkAvroCallbackHandler.scala
Error:(45, 66) not found: type SparkFlumeProtocol

解决方案：

选中Spark Project External Flume Sink,并右键点击Generate Sources and Update Folders. 然后重新编译应该就会消失。

4.2 Error:(34, 45) object SqlBaseParser is not a member of package org.apache.spark.sql.catalyst.parser

sparksqlcatalystsrcmainscalaorgapachesparksqlcatalystparserAstBuilder.scala
Error:(34, 45) object SqlBaseParser is not a member of package org.apache.spark.sql.catalyst.parser   
import org.apache.spark.sql.catalyst.parser.SqlBaseParser._

解决方案:

选中Spark Project External  Catalyst,并右键点击Generate Sources and Update Folders. 然后重新编译应该就会消失.

4.3 Error:(52, 75) not found: value TCLIService

sparksqlhive-thriftserversrcmainjavaorgapachehiveservicecli	hriftThriftCLIService.java
Error:(52, 75) not found: value TCLIService
public abstract class ThriftCLIService extends AbstractService implements TCLIService.Iface, Runnable {………..

一般来讲，这几个问题解决之后，编译就会成功。

5. gitBash中进行编译

为什么使用gitbash,因为在idea中编译时会出现各种各种的报错，gitbash中拥有一些idea中没有的环境。可能出现如下错误，使用gitbash即可解决

使用gitbash进入项目的根目录下，执行下面3条命令

cd /c/Workspace/spark-2.0.0 
export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"
./build/mvn -Pyarn -Phadoop-2.7 -Dhadoop.version=2.7.3 -DskipTests clean package

最后编译出来的结果如下：

参考:
https://blog.csdn.net/make__It/article/details/84258916

http://dengfengli.com/blog/how-to-run-and-debug-spark-source-code-locally/