场景
配置完scala(2.11)和spark(2.2.0)的windows环境后,写了个worldcount案例。加入以下依赖,然后编写wordcount
<properties>
<spark-version>2.2.0</spark-version>
<scala-version>2.11</scala-version>
</properties>
<dependencies>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala-version}</artifactId>
<version>${spark-version}</version>
</dependency>
</dependencies>
import org.apache.spark.rdd.RDD
import org.apache.spark.{SparkConf, SparkContext}
/**
* @Author 晓风
* @Date 2021/10/8 22:37
* @Version 1.0
*/
object WorldCount {
def main(args: Array[String]): Unit = {
// 配置相关信息
val sparkConf:SparkConf = new SparkConf().setMaster("local").setAppName("wordCount")
// 获取上下文
val sc: SparkContext = new SparkContext(sparkConf)
/* // 读取文件
val lines: RDD[String] = sc.textFile("data/helloworld")
// 切分,flatmap
val worlds: RDD[String] = lines.flatMap(_.split(" "))
// map
val worldTuple: RDD[(String, Int)] = worlds.map((_, 1))
// reduce
val value: RDD[(String, Int)] = worldTuple.reduceByKey((_ + _))
// 打印
value.foreach(println)
*/
//hello scala
//hello java
//hello spark
//hello flink
//hello world
sc.textFile("data/helloworld").flatMap(_.split(" ")).map((_,1)).reduceByKey(_ + _).foreach(println)
// 关闭连接
sc.stop()
}
}
- 报错
Caused by: java.lang.NoClassDefFoundError: com/fasterxml/jackson/annotation/JsonMerge
- 加入依赖
<dependency> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-databind</artifactId> <version>2.9.0</version> </dependency> <dependency> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-core</artifactId> <version>2.9.0</version> </dependency> <dependency> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-annotations</artifactId> <version>2.9.0</version> </dependency>
- 报错
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.9.0
- 加入依赖
<dependency> <groupId>com.fasterxml.jackson.module</groupId> <artifactId>jackson-module-scala_2.11</artifactId> <version>2.9.0</version> </dependency>
最终依赖
```properties
<properties>
<spark-version>2.2.0</spark-version>
<scala-version>2.11</scala-version>
</properties>
<dependencies>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala-version}</artifactId>
<version>${spark-version}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.module</groupId>
<artifactId>jackson-module-scala_2.11</artifactId>
<version>2.9.0</version>
</dependency>
</dependencies>
```
原因
- jackson.databind版本与 jackson.module版本不匹配