CDH5.16Spark升级成Spark2后HistoryServer无法启动

报错信息

21/04/22 09:53:15 INFO history.HistoryServer: Started daemon with process name: 10548@cloudera-02
21/04/22 09:53:15 INFO util.SignalUtils: Registered signal handler for TERM
21/04/22 09:53:15 INFO util.SignalUtils: Registered signal handler for HUP
21/04/22 09:53:15 INFO util.SignalUtils: Registered signal handler for INT
21/04/22 09:53:16 WARN spark.SparkConf: The configuration key 'spark.history.fs.update.interval.seconds' has been deprecated as of Spark 1.4 and may be removed in the future. Please use the new key 'spark.history.fs.update.interval' instead.
21/04/22 09:53:16 INFO spark.SecurityManager: Changing view acls to: root
21/04/22 09:53:16 INFO spark.SecurityManager: Changing modify acls to: root
21/04/22 09:53:16 INFO spark.SecurityManager: Changing view acls groups to: 
21/04/22 09:53:16 INFO spark.SecurityManager: Changing modify acls groups to: 
21/04/22 09:53:16 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
21/04/22 09:53:16 INFO history.FsHistoryProvider: History server ui acls disabled; users with admin permissions: ; groups with admin permissions
Exception in thread "main" java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:280)
	at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)
Caused by: java.io.FileNotFoundException: Log directory specified does not exist: hdfs://cloudera-01:8020/user/spark/spark2ApplicationHistory
	at org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$startPolling(FsHistoryProvider.scala:267)
	at org.apache.spark.deploy.history.FsHistoryProvider.initialize(FsHistoryProvider.scala:211)
	at org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHistoryProvider.scala:207)
	at org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHistoryProvider.scala:86)
	... 6 more
Caused by: java.io.FileNotFoundException: File does not exist: hdfs://cloudera-01:8020/user/spark/spark2ApplicationHistory
	at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1270)
	at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1262)
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1262)
	at org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$startPolling(FsHistoryProvider.scala:257)
	... 9 more

解决方案

原来是Spark升级后，没有在HDFS上创建spark2ApplicationHistory目录，根据错误提示

Caused by: java.io.FileNotFoundException: Log directory specified does not exist: hdfs://cloudera-01:8020/user/spark/spark2ApplicationHistory

手动在HDFS上创建目录
使用hadoop dfs -mkdir /user/spark/spark2ApplicationHistory创建目录,但是发现权限不足

原来,CDH5.16的HDFS中，最高权限用户是root，我们需要修改HDFS权限，用来创建目录

1）为了创建目录，我们先进入到HDFS配置中，将dfs.permissions取消勾选（CDH默认会开启权限认证）

2)重启集群
3)然后再次使用hadoop dfs -mkdir /user/spark/spark2ApplicationHistory创建目录,
4)创建好后启动Spark2进行测试

5)Spark2运行成功后, 记得重新将dfs.permission勾选上

6)重启集群，问题解决!