报错信息
21/04/22 09:53:15 INFO history.HistoryServer: Started daemon with process name: 10548@cloudera-02
21/04/22 09:53:15 INFO util.SignalUtils: Registered signal handler for TERM
21/04/22 09:53:15 INFO util.SignalUtils: Registered signal handler for HUP
21/04/22 09:53:15 INFO util.SignalUtils: Registered signal handler for INT
21/04/22 09:53:16 WARN spark.SparkConf: The configuration key 'spark.history.fs.update.interval.seconds' has been deprecated as of Spark 1.4 and may be removed in the future. Please use the new key 'spark.history.fs.update.interval' instead.
21/04/22 09:53:16 INFO spark.SecurityManager: Changing view acls to: root
21/04/22 09:53:16 INFO spark.SecurityManager: Changing modify acls to: root
21/04/22 09:53:16 INFO spark.SecurityManager: Changing view acls groups to:
21/04/22 09:53:16 INFO spark.SecurityManager: Changing modify acls groups to:
21/04/22 09:53:16 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
21/04/22 09:53:16 INFO history.FsHistoryProvider: History server ui acls disabled; users with admin permissions: ; groups with admin permissions
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:280)
at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)
Caused by: java.io.FileNotFoundException: Log directory specified does not exist: hdfs://cloudera-01:8020/user/spark/spark2ApplicationHistory
at org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$startPolling(FsHistoryProvider.scala:267)
at org.apache.spark.deploy.history.FsHistoryProvider.initialize(FsHistoryProvider.scala:211)
at org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHistoryProvider.scala:207)
at org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHistoryProvider.scala:86)
... 6 more
Caused by: java.io.FileNotFoundException: File does not exist: hdfs://cloudera-01:8020/user/spark/spark2ApplicationHistory
at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1270)
at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1262)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1262)
at org.apache.spark.deploy.history.FsHistoryProvider.org$apache$spark$deploy$history$FsHistoryProvider$$startPolling(FsHistoryProvider.scala:257)
... 9 more
解决方案
原来是Spark升级后，没有在HDFS上创建spark2ApplicationHistory目录，根据错误提示
Caused by: java.io.FileNotFoundException: Log directory specified does not exist: hdfs://cloudera-01:8020/user/spark/spark2ApplicationHistory
手动在HDFS上创建目录
使用hadoop dfs -mkdir /user/spark/spark2ApplicationHistory创建目录,但是发现权限不足
原来,CDH5.16的HDFS中，最高权限用户是root，我们需要修改HDFS权限，用来创建目录
1）为了创建目录，我们先进入到HDFS配置中，将dfs.permissions取消勾选（CDH默认会开启权限认证）
2)重启集群
3)然后再次使用hadoop dfs -mkdir /user/spark/spark2ApplicationHistory创建目录,
4)创建好后启动Spark2进行测试
5)Spark2运行成功后, 记得重新将dfs.permission勾选上
6)重启集群，问题解决!