zoukankan      html  css  js  c++  java
  • CDH6.2的spark访问oss

    CDH6配置oss后:
    spark的配置

    /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars
    /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hive/lib
    /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark/jars    #只是这个目录就可以
    /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop/lib
    这几个目录可能都要放jar包

    这个目录也可以要放:

    Caused by: java.lang.ClassNotFoundException: org.jdom.input.JDOMParseException

    cd /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark/jars/

    ls |grep ali
    rm -f aliyun-sdk-oss-2.8.3.jar
    rm -f hadoop-aliyun.jar
    ls |grep ali

    cp /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/aliyun-sdk-oss-2.8.3.jar /etc/hive/auxlib/
    cp /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/hadoop-aliyun-3.0.0-cdh6.2.0.jar /etc/hive/auxlib/

    cp /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/aliyun-sdk-oss-2.8.3.jar /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hive/lib/

    cp /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/hadoop-aliyun-3.0.0-cdh6.2.0.jar /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hive/lib/

    cp /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/aliyun-sdk-oss-2.8.3.jar /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark/jars/

    cp /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/hadoop-aliyun-3.0.0-cdh6.2.0.jar /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark/jars/

    cp /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/aliyun-sdk-oss-2.8.3.jar /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop/lib/

    cp /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/hadoop-aliyun-3.0.0-cdh6.2.0.jar /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop/lib/


    关于阿里云OSS报错-java.lang.ClassNotFoundException: org.jdom.input.JDOMParseException:
    https://tech.antfin.com/docs/2/97907

    cp /data/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/jdom-1.1.jar /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark/jars/

  • 相关阅读:
    如何选择开源许可证?(转)
    gdb的使用(转)
    git的使用
    2017-3-13 leetcode 4 11 15
    2017-3-12 leetcode 167 209 216
    2017-3-11 leetcode 217 219 228
    2017-3-10 leetcode 229 238 268
    1175: 零起点学算法82——find your present
    1174: 零起点学算法81——求整数绝对值
    1173: 零起点学算法80——求实数绝对值
  • 原文地址:https://www.cnblogs.com/hongfeng2019/p/12040919.html
Copyright © 2011-2022 走看看