zoukankan      html  css  js  c++  java
  • 【原创】大叔经验分享(5)oozie提交spark任务如何添加依赖

    spark任务添加依赖的方式:

    1 如果是local方式运行,可以通过--jars来添加依赖;

    2 如果是yarn方式运行,可以通过spark.yarn.jars来添加依赖;

    这两种方式在oozie上都行不通,首先oozie上没办法也不应该通过local运行,其次通过spark.yarn.jars方式配置你会发现根本不会生效,来看为什么

    查看LauncherMapper的日志

    Spark Version 2.1.1

    Spark Action Main class        : org.apache.spark.deploy.SparkSubmit

    Oozie Spark action configuration

    =================================================================

    ...

                        --conf

                        spark.yarn.jars=hdfs://hdfs_name/jarpath/*.jar

                        --conf

                        spark.yarn.jars=hdfs://hdfs_name/oozie/share/lib_20180801121138/spark/spark-yarn_2.11-2.1.1.jar

    可见oozie会自己添加一个新的spark.yarn.jars配置,如果提供两个相同的key,spark会如何处理

    org.apache.spark.deploy.SparkSubmit

        val appArgs = new SparkSubmitArguments(args)

    org.apache.spark.launcher.SparkSubmitOptionParser

            if (!handle(name, value)) {

    org.apache.spark.deploy.SparkSubmitArguments

      override protected def handle(opt: String, value: String): Boolean = {

      ...

          case CONF =>

            value.split("=", 2).toSeq match {

              case Seq(k, v) => sparkProperties(k) = v

              case _ => SparkSubmit.printErrorAndExit(s"Spark config without '=': $value")

            }

    可见会直接覆盖,使用最后一个配置,即oozie的配置,而不是应用自己提供的配置,这样就需要应用自己将特殊依赖打包到应用jar中,具体使用maven的maven-assembly-plugin,配置其中的<dependencySets><dependencySet><includes><include>,详细配置如下:

    <assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0"

              xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

              xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0 http://maven.apache.org/xsd/assembly-1.1.0.xsd">

        <!-- TODO: a jarjar format would be better -->

        <id>jar-with-dependencies</id>

        <formats>

            <format>jar</format>

        </formats>

        <includeBaseDirectory>false</includeBaseDirectory>

        <dependencySets>

            <dependencySet>

                <outputDirectory>/</outputDirectory>

                <useProjectArtifact>true</useProjectArtifact>

                <unpack>true</unpack>

                <scope>runtime</scope>

                <includes>

                    <include>redis.clients:jedis</include>

                    <include>org.apache.commons:commons-pool2</include>

                </includes>

            </dependencySet>

        </dependencySets>

    </assembly>

    这里只是将默认提供的jar-with-dependencies.xml内容拷贝出来添加includes配置;

  • 相关阅读:
    BI之SSAS完整实战教程1 -- 开篇, BI简介 & SSAS简介
    MVC 5 + EF6 入门完整教程14 -- 动态生成面包屑导航
    MVC5+EF6 入门完整教程12--灵活控制Action权限
    JVM垃圾收集
    JVM类加载
    前端UI框架
    前端学习
    JVM字节码
    网络编程
    Java NIO
  • 原文地址:https://www.cnblogs.com/barneywill/p/10109352.html
Copyright © 2011-2022 走看看