zoukankan      html  css  js  c++  java
  • 【原创】大数据基础之Oozie(4)oozie使用的spark版本升级

    oozie默认使用的spark是1.6,一直没有升级,如果想用最新的2.4,需要自己手工升级

    首先看当前使用的spark版本的jar

    # oozie admin -oozie http://$oozie_server:11000/oozie -shareliblist
    # oozie admin -oozie http://$oozie_server:11000/oozie -shareliblist spark

    升级:

    1 手工修改hdfs上的sharelib目录,同时还要拷贝原有的oozie-sharelib-spark.jar

    # hdfs dfs -mv /user/oozie/share/lib/lib_20190424195113/spark /user/oozie/share/lib/lib_20190424195113/spark1
    # hdfs dfs -mkdir /user/oozie/share/lib/lib_20190424195113/spark
    # hdfs dfs -copyFromLocal $SPARK_HOME/jars/* /user/oozie/share/lib/lib_20190424195113/spark/
    # hdfs dfs -copyFromLocal $SPARK_HOME/python/lib/*.zip /user/oozie/share/lib/lib_20190424195113/spark/

    # hdfs dfs -cp /user/oozie/share/lib/lib_20190424195113/spark1/oozie-sharelib-spark.jar /user/oozie/share/lib/lib_20190424195113/spark/

    2 调用oozie的sharelibupdate命令

    # oozie admin -oozie http://$oozie_server:11000/oozie -sharelibupdate

    3 确认spark版本是否更新

    # oozie admin -oozie http://$oozie_server:11000/oozie -shareliblist spark
  • 相关阅读:
    My SQL
    弹窗
    DBDA
    ThinkPHP验证码与文件上传
    ThinkPHP表单验证
    ThinkPHP增删改
    ThinkPHP模型(查询)
    ThinkPHP跨控制器调用方法
    Superset安装
    Presto资源组配置
  • 原文地址:https://www.cnblogs.com/barneywill/p/11132512.html
Copyright © 2011-2022 走看看