1.打包python环境
建议使用conda conda克隆环境 conda create -n prod_env --clone base 进入conda的miniconda3/envs # 打包python环境 zip prod_env.zip prod_env
2.提交任务
spark-submit --master yarn --deploy-mode client --archives prod_env.zip#ENV # 会上传本地的prod_env.zip到hdfs上，作为缓存，运行时会解压到ENV目录，程序结束后自动删除; 如果python的库变动不大，可以提前做好zip包上传到hdfs上，然后指定hdfs:///your-path --conf spark.yarn.appMasterEnv.PYSPARK_PYTHON=./ENV/prod_env/bin/python3 #相对ENV目录 --conf spark.executorEnv.PYSPARK_PYTHON=./ENV/prod_env/bin/python3 hello.py
hello.py
from pyspark import SparkConf
from pyspark.sql import SparkSession
print("==========app start=========")
spark = SparkSession.builder.enableHiveSupport().getOrCreate()
df = spark.sql('show databases')
df.show()
print("==========app stop=========")
spark.stop()
上面使用的client模式。
本人是单机，以cluster提交时发生的错误，不知道怎么回事呀/ 日志中说找不到./ENV/prod_env/bin/python3
下面是错误日志
(base) demo@demo-PC:~/miniconda3/envs$ spark-submit --master yarn --deploy-mode cluster --archives hdfs:///user/demo/demo.zip#ENV --conf spark.yarn.appMasterEnv.PYSPARK_PYTHON=./ENV/demo/bin/python3 --conf spark.executorEnv.PYSPARK_PYTHON=./ENV/demo/bin/python3 hello.py
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/bigdata/spark-2.3.0-bin-hadoop2.6/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/bigdata/hadoop-2.6.5/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2020-09-19 21:20:12 WARN Utils:66 - Your hostname, demo-PC resolves to a loopback address: 127.0.1.1; using 192.168.185.144 instead (on interface ens33)
2020-09-19 21:20:12 WARN Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address
2020-09-19 21:20:13 INFO RMProxy:98 - Connecting to ResourceManager at /0.0.0.0:8032
2020-09-19 21:20:13 INFO Client:54 - Requesting a new application from cluster with 1 NodeManagers
2020-09-19 21:20:13 INFO Client:54 - Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
2020-09-19 21:20:13 INFO Client:54 - Will allocate AM container, with 1408 MB memory including 384 MB overhead
2020-09-19 21:20:13 INFO Client:54 - Setting up container launch context for our AM
2020-09-19 21:20:13 INFO Client:54 - Setting up the launch environment for our AM container
2020-09-19 21:20:13 INFO Client:54 - Preparing resources for our AM container
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/JavaEWAH-0.3.2.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/RoaringBitmap-0.5.11.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/ST4-4.0.4.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/activation-1.1.1.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/aircompressor-0.8.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/antlr-2.7.7.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/antlr-runtime-3.4.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/antlr4-runtime-4.7.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/aopalliance-1.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/aopalliance-repackaged-2.4.0-b34.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/apache-log4j-extras-1.2.17.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/apacheds-i18n-2.0.0-M15.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/apacheds-kerberos-codec-2.0.0-M15.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/api-asn1-api-1.0.0-M20.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/api-util-1.0.0-M20.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/arpack_combined_all-0.1.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/arrow-format-0.8.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/arrow-memory-0.8.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/arrow-vector-0.8.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/automaton-1.11-8.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/avro-1.7.7.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/avro-ipc-1.7.7.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/avro-mapred-1.7.7-hadoop2.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/base64-2.3.8.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/bcprov-jdk15on-1.58.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/bonecp-0.8.0.RELEASE.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/breeze-macros_2.11-0.13.2.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/breeze_2.11-0.13.2.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/calcite-avatica-1.2.0-incubating.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/calcite-core-1.2.0-incubating.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/calcite-linq4j-1.2.0-incubating.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/chill-java-0.8.4.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/chill_2.11-0.8.4.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/commons-beanutils-1.7.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/commons-beanutils-core-1.8.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/commons-cli-1.2.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/commons-codec-1.10.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/commons-collections-3.2.2.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/commons-compiler-3.0.8.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/commons-compress-1.4.1.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/commons-configuration-1.6.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/commons-crypto-1.0.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/commons-dbcp-1.4.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/commons-digester-1.8.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/commons-httpclient-3.1.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/commons-io-2.4.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/commons-lang-2.6.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/commons-lang3-3.5.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/commons-logging-1.1.3.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/commons-math3-3.4.1.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/commons-net-2.2.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/commons-pool-1.5.4.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/compress-lzf-1.0.3.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/core-1.1.2.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/curator-client-2.6.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/curator-framework-2.6.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/curator-recipes-2.6.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/datanucleus-api-jdo-3.2.6.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/datanucleus-core-3.2.10.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/datanucleus-rdbms-3.2.9.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/derby-10.12.1.1.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/eigenbase-properties-1.1.5.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/flatbuffers-1.2.0-3f79e055.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/generex-1.0.1.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/gson-2.2.4.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/guava-14.0.1.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/guice-3.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/guice-servlet-3.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/hadoop-annotations-2.6.5.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/hadoop-auth-2.6.5.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/hadoop-client-2.6.5.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/hadoop-common-2.6.5.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/hadoop-hdfs-2.6.5.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/hadoop-mapreduce-client-app-2.6.5.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/hadoop-mapreduce-client-common-2.6.5.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/hadoop-mapreduce-client-core-2.6.5.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/hadoop-mapreduce-client-jobclient-2.6.5.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/hadoop-mapreduce-client-shuffle-2.6.5.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/hadoop-yarn-api-2.6.5.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/hadoop-yarn-client-2.6.5.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/hadoop-yarn-common-2.6.5.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/hadoop-yarn-server-common-2.6.5.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/hadoop-yarn-server-web-proxy-2.6.5.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/hive-beeline-1.2.1.spark2.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/hive-cli-1.2.1.spark2.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/hive-exec-1.2.1.spark2.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/hive-jdbc-1.2.1.spark2.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/hive-metastore-1.2.1.spark2.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/hk2-api-2.4.0-b34.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/hk2-locator-2.4.0-b34.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/hk2-utils-2.4.0-b34.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/hppc-0.7.2.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/htrace-core-3.0.4.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/httpclient-4.5.4.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/httpcore-4.4.8.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/ivy-2.4.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/jackson-annotations-2.6.7.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/jackson-core-2.6.7.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/jackson-core-asl-1.9.13.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/jackson-databind-2.6.7.1.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/jackson-dataformat-yaml-2.6.7.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/jackson-jaxrs-1.9.13.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/jackson-mapper-asl-1.9.13.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/jackson-module-jaxb-annotations-2.6.7.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/jackson-module-paranamer-2.7.9.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/jackson-module-scala_2.11-2.6.7.1.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/jackson-xc-1.9.13.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/janino-3.0.8.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/java-xmlbuilder-1.1.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/javassist-3.18.1-GA.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/javax.annotation-api-1.2.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/javax.inject-1.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/javax.inject-2.4.0-b34.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/javax.servlet-api-3.1.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/javax.ws.rs-api-2.0.1.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/javolution-5.5.1.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/jaxb-api-2.2.2.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/jcl-over-slf4j-1.7.16.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/jdo-api-3.0.1.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/jersey-client-2.22.2.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/jersey-common-2.22.2.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/jersey-container-servlet-2.22.2.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/jersey-container-servlet-core-2.22.2.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/jersey-guava-2.22.2.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/jersey-media-jaxb-2.22.2.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/jersey-server-2.22.2.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/jets3t-0.9.4.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/jetty-6.1.26.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/jetty-util-6.1.26.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/jline-2.12.1.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/joda-time-2.9.3.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/jodd-core-3.5.2.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/jpam-1.1.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/json4s-ast_2.11-3.2.11.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/json4s-core_2.11-3.2.11.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/json4s-jackson_2.11-3.2.11.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/jsr305-1.3.9.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/jta-1.1.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/jtransforms-2.4.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/jul-to-slf4j-1.7.16.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/kryo-shaded-3.0.3.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/kubernetes-client-3.0.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/kubernetes-model-2.0.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/leveldbjni-all-1.8.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/libfb303-0.9.3.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/libthrift-0.9.3.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/log4j-1.2.17.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/logging-interceptor-3.8.1.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/lz4-java-1.4.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/machinist_2.11-0.6.1.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/macro-compat_2.11-1.1.1.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/mesos-1.4.0-shaded-protobuf.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/metrics-core-3.1.5.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/metrics-graphite-3.1.5.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/metrics-json-3.1.5.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/metrics-jvm-3.1.5.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/minlog-1.3.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/mysql-connector-java.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/netty-3.9.9.Final.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/netty-all-4.1.17.Final.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/objenesis-2.1.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/okhttp-3.8.1.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/okio-1.13.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/opencsv-2.3.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/orc-core-1.4.1-nohive.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/orc-mapreduce-1.4.1-nohive.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/oro-2.0.8.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/osgi-resource-locator-1.0.1.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/paranamer-2.8.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/parquet-column-1.8.2.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/parquet-common-1.8.2.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/parquet-encoding-1.8.2.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/parquet-format-2.3.1.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/parquet-hadoop-1.8.2.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/parquet-hadoop-bundle-1.6.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/parquet-jackson-1.8.2.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/protobuf-java-2.5.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/py4j-0.10.6.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/pyrolite-4.13.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/scala-compiler-2.11.8.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/scala-library-2.11.8.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/scala-parser-combinators_2.11-1.0.4.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/scala-reflect-2.11.8.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/scala-xml_2.11-1.0.5.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/scalap-2.11.8.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/shapeless_2.11-2.3.2.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/slf4j-api-1.7.16.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/slf4j-log4j12-1.7.16.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/snakeyaml-1.15.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/snappy-0.2.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/snappy-java-1.1.2.6.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/spark-catalyst_2.11-2.3.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/spark-core_2.11-2.3.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/spark-graphx_2.11-2.3.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/spark-hive-thriftserver_2.11-2.3.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/spark-hive_2.11-2.3.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/spark-kubernetes_2.11-2.3.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/spark-kvstore_2.11-2.3.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/spark-launcher_2.11-2.3.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/spark-mesos_2.11-2.3.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/spark-mllib-local_2.11-2.3.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/spark-mllib_2.11-2.3.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/spark-network-common_2.11-2.3.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/spark-network-shuffle_2.11-2.3.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/spark-repl_2.11-2.3.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/spark-sketch_2.11-2.3.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/spark-sql_2.11-2.3.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/spark-streaming_2.11-2.3.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/spark-tags_2.11-2.3.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/spark-unsafe_2.11-2.3.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/spark-yarn_2.11-2.3.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/spire-macros_2.11-0.13.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/spire_2.11-0.13.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/stax-api-1.0-2.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/stax-api-1.0.1.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/stream-2.7.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/stringtemplate-3.2.1.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/super-csv-2.2.0.jar
2020-09-19 21:20:14 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/univocity-parsers-2.5.9.jar
2020-09-19 21:20:15 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/validation-api-1.1.0.Final.jar
2020-09-19 21:20:15 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/xbean-asm5-shaded-4.4.jar
2020-09-19 21:20:15 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/xercesImpl-2.9.1.jar
2020-09-19 21:20:15 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/xmlenc-0.52.jar
2020-09-19 21:20:15 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/xz-1.0.jar
2020-09-19 21:20:15 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/zjsonpatch-0.3.0.jar
2020-09-19 21:20:15 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/zookeeper-3.4.6.jar
2020-09-19 21:20:15 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/hadoop/spark_jars/zstd-jni-1.3.2-2.jar
2020-09-19 21:20:15 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs://127.0.0.1:9000/user/demo/demo.zip
2020-09-19 21:20:15 INFO Client:54 - Uploading resource file:/home/demo/miniconda3/envs/hello.py -> hdfs://127.0.0.1:9000/user/demo/.sparkStaging/application_1600521219702_0002/hello.py
2020-09-19 21:20:15 INFO Client:54 - Uploading resource file:/opt/bigdata/spark-2.3.0-bin-hadoop2.6/python/lib/pyspark.zip -> hdfs://127.0.0.1:9000/user/demo/.sparkStaging/application_1600521219702_0002/pyspark.zip
2020-09-19 21:20:15 INFO Client:54 - Uploading resource file:/opt/bigdata/spark-2.3.0-bin-hadoop2.6/python/lib/py4j-0.10.6-src.zip -> hdfs://127.0.0.1:9000/user/demo/.sparkStaging/application_1600521219702_0002/py4j-0.10.6-src.zip
2020-09-19 21:20:15 INFO Client:54 - Uploading resource file:/tmp/spark-6c4f9ca0-5c52-4f58-9f9b-ff8c4e6b3336/__spark_conf__2718114163664064849.zip -> hdfs://127.0.0.1:9000/user/demo/.sparkStaging/application_1600521219702_0002/__spark_conf__.zip
2020-09-19 21:20:15 INFO SecurityManager:54 - Changing view acls to: demo
2020-09-19 21:20:15 INFO SecurityManager:54 - Changing modify acls to: demo
2020-09-19 21:20:15 INFO SecurityManager:54 - Changing view acls groups to:
2020-09-19 21:20:15 INFO SecurityManager:54 - Changing modify acls groups to:
2020-09-19 21:20:15 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(demo); groups with view permissions: Set(); users with modify permissions: Set(demo); groups with modify permissions: Set()
2020-09-19 21:20:15 INFO Client:54 - Submitting application application_1600521219702_0002 to ResourceManager
2020-09-19 21:20:15 INFO YarnClientImpl:251 - Submitted application application_1600521219702_0002
2020-09-19 21:20:16 INFO Client:54 - Application report for application_1600521219702_0002 (state: ACCEPTED)
2020-09-19 21:20:16 INFO Client:54 -
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1600521615900
final status: UNDEFINED
tracking URL: http://demo-PC:8088/proxy/application_1600521219702_0002/
user: demo
2020-09-19 21:20:17 INFO Client:54 - Application report for application_1600521219702_0002 (state: ACCEPTED)
2020-09-19 21:20:18 INFO Client:54 - Application report for application_1600521219702_0002 (state: ACCEPTED)
2020-09-19 21:20:19 INFO Client:54 - Application report for application_1600521219702_0002 (state: ACCEPTED)
2020-09-19 21:20:20 INFO Client:54 - Application report for application_1600521219702_0002 (state: ACCEPTED)
2020-09-19 21:20:21 INFO Client:54 - Application report for application_1600521219702_0002 (state: ACCEPTED)
2020-09-19 21:20:22 INFO Client:54 - Application report for application_1600521219702_0002 (state: ACCEPTED)
2020-09-19 21:20:23 INFO Client:54 - Application report for application_1600521219702_0002 (state: ACCEPTED)
2020-09-19 21:20:24 INFO Client:54 - Application report for application_1600521219702_0002 (state: FAILED)
2020-09-19 21:20:24 INFO Client:54 -
client token: N/A
diagnostics: Application application_1600521219702_0002 failed 2 times due to AM Container for appattempt_1600521219702_0002_000002 exited with exitCode: 13
For more detailed output, check application tracking page:http://demo-PC:8088/proxy/application_1600521219702_0002/Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1600521219702_0002_02_000001
Exit code: 13
Stack trace: ExitCodeException exitCode=13:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:575)
at org.apache.hadoop.util.Shell.run(Shell.java:478)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:766)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Container exited with a non-zero exit code 13
Failing this attempt. Failing the application.
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1600521615900
final status: FAILED
tracking URL: http://demo-PC:8088/cluster/app/application_1600521219702_0002
user: demo
Exception in thread "main" org.apache.spark.SparkException: Application application_1600521219702_0002 finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1159)
at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1518)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2020-09-19 21:20:24 INFO ShutdownHookManager:54 - Shutdown hook called
2020-09-19 21:20:24 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-6c4f9ca0-5c52-4f58-9f9b-ff8c4e6b3336
2020-09-19 21:20:24 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-29e8441f-4ddb-435e-85ba-0406a7eec752
2020-09-19 21:20:22 INFO SignalUtils:54 - Registered signal handler for TERM 2020-09-19 21:20:22 INFO SignalUtils:54 - Registered signal handler for HUP 2020-09-19 21:20:22 INFO SignalUtils:54 - Registered signal handler for INT 2020-09-19 21:20:22 WARN Utils:66 - Your hostname, demo-PC resolves to a loopback address: 127.0.1.1; using 192.168.185.144 instead (on interface ens33) 2020-09-19 21:20:22 WARN Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address 2020-09-19 21:20:22 INFO SecurityManager:54 - Changing view acls to: demo 2020-09-19 21:20:22 INFO SecurityManager:54 - Changing modify acls to: demo 2020-09-19 21:20:22 INFO SecurityManager:54 - Changing view acls groups to: 2020-09-19 21:20:22 INFO SecurityManager:54 - Changing modify acls groups to: 2020-09-19 21:20:22 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(demo); groups with view permissions: Set(); users with modify permissions: Set(demo); groups with modify permissions: Set() 2020-09-19 21:20:23 INFO ApplicationMaster:54 - Preparing Local resources 2020-09-19 21:20:23 INFO ApplicationMaster:54 - ApplicationAttemptId: appattempt_1600521219702_0002_000002 2020-09-19 21:20:23 INFO ApplicationMaster:54 - Starting the user application in a separate Thread 2020-09-19 21:20:23 INFO ApplicationMaster:54 - Waiting for spark context initialization... 2020-09-19 21:20:23 ERROR ApplicationMaster:91 - User class threw exception: java.io.IOException: Cannot run program "./ENV/demo/bin/python3": error=2, 没有那个文件或目录 java.io.IOException: Cannot run program "./ENV/demo/bin/python3": error=2, 没有那个文件或目录 at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) at org.apache.spark.deploy.PythonRunner$.main(PythonRunner.scala:91) at org.apache.spark.deploy.PythonRunner.main(PythonRunner.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:706) Caused by: java.io.IOException: error=2, 没有那个文件或目录 at java.lang.UNIXProcess.forkAndExec(Native Method) at java.lang.UNIXProcess.<init>(UNIXProcess.java:247) at java.lang.ProcessImpl.start(ProcessImpl.java:134) at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) ... 7 more 2020-09-19 21:20:23 INFO ApplicationMaster:54 - Final app status: FAILED, exitCode: 13, (reason: User class threw exception: java.io.IOException: Cannot run program "./ENV/demo/bin/python3": error=2, 没有那个文件或目录 at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) at org.apache.spark.deploy.PythonRunner$.main(PythonRunner.scala:91) at org.apache.spark.deploy.PythonRunner.main(PythonRunner.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:706) Caused by: java.io.IOException: error=2, 没有那个文件或目录 at java.lang.UNIXProcess.forkAndExec(Native Method) at java.lang.UNIXProcess.<init>(UNIXProcess.java:247) at java.lang.ProcessImpl.start(ProcessImpl.java:134) at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) ... 7 more ) 2020-09-19 21:20:23 ERROR ApplicationMaster:91 - Uncaught exception: org.apache.spark.SparkException: Exception thrown in awaitResult: at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:205) at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:486) at org.apache.spark.deploy.yarn.ApplicationMaster.org$apache$spark$deploy$yarn$ApplicationMaster$$runImpl(ApplicationMaster.scala:345) at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply$mcV$sp(ApplicationMaster.scala:260) at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply(ApplicationMaster.scala:260) at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply(ApplicationMaster.scala:260) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$5.run(ApplicationMaster.scala:800) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692) at org.apache.spark.deploy.yarn.ApplicationMaster.doAsUser(ApplicationMaster.scala:799) at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:259) at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:824) at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala) Caused by: java.io.IOException: Cannot run program "./ENV/demo/bin/python3": error=2, 没有那个文件或目录 at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) at org.apache.spark.deploy.PythonRunner$.main(PythonRunner.scala:91) at org.apache.spark.deploy.PythonRunner.main(PythonRunner.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:706) Caused by: java.io.IOException: error=2, 没有那个文件或目录 at java.lang.UNIXProcess.forkAndExec(Native Method) at java.lang.UNIXProcess.<init>(UNIXProcess.java:247) at java.lang.ProcessImpl.start(ProcessImpl.java:134) at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) ... 7 more 2020-09-19 21:20:23 INFO ApplicationMaster:54 - Deleting staging directory hdfs://127.0.0.1:9000/user/demo/.sparkStaging/application_1600521219702_0002 2020-09-19 21:20:23 INFO ShutdownHookManager:54 - Shutdown hook called