Spark创建空的DataFrame

前言

本文主要给出Spark创建空的DataFrame的代码示例，这里讲的空的DataFrame主要指有列名（可以自己随意指定），但是没有行的DataFrame，因为自己在开发过程中有这个需求，之前并不知道怎么创建，就查了一下，发现资料并不多，不知道因为太简单还是用的人少_{，至于具体什么需求就不阐述了，主要给有这方面需求的小伙伴参考一下。还有另一种空的DataFrame就是没有任何行任何列的DataFrame，不知道有什么用，反正贴在代码里，万一有人用呢}

1、代码

代码较简单，如下

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.types._
import org.apache.spark.sql.Row
import org.apache.spark.rdd.EmptyRDD

/**
 * Spark创建空DataFrame示例
 */
object EmptyDataFrame {

  def main(args: Array[String]): Unit = {

    val spark = SparkSession.builder().appName("EmptyDataFrame").master("local").getOrCreate()

    /**
     * 创建一个空的DataFrame，代表用户
     * 有四列，分别代表ID、名字、年龄、生日
     */
    val colNames = Array("id", "name", "age", "birth")
    //为了简单起见，字段类型都为String
    val schema = StructType(colNames.map(fieldName => StructField(fieldName, StringType, true)))
    //主要是利用了spark.sparkContext.emptyRDD
    val emptyDf = spark.createDataFrame(spark.sparkContext.emptyRDD[Row], schema)

    emptyDf.show

    /**
     * 也可以给每列指定相对应的类型
     */
    val schema1 = StructType(
      Seq(
        StructField("id", IntegerType, true),
        StructField("name", StringType, true),
        StructField("age", IntegerType, true),
        StructField("birth", StringType, true)))
    val emptyDf1 = spark.createDataFrame(spark.sparkContext.emptyRDD[Row], schema1)
    emptyDf1.show

    //还有一种空的DataFrame，没有任何行任何列
    spark.emptyDataFrame.show

    spark.stop()
  }

}

2、结果

D:devtoolsJavajdk1.8.0_191injava.exe "-javaagent:D:devtoolsIntelliJ IDEA Community Edition 2018.2.6libidea_rt.jar=64870:D:devtoolsIntelliJ IDEA Community Edition 2018.2.6in" -Dfile.encoding=UTF-8 -classpath D:devtoolsJavajdk1.8.0_191jrelibcharsets.jar;D:devtoolsJavajdk1.8.0_191jrelibdeploy.jar;D:devtoolsJavajdk1.8.0_191jrelibextaccess-bridge-64.jar;D:devtoolsJavajdk1.8.0_191jrelibextcldrdata.jar;D:devtoolsJavajdk1.8.0_191jrelibextdnsns.jar;D:devtoolsJavajdk1.8.0_191jrelibextjaccess.jar;D:devtoolsJavajdk1.8.0_191jrelibextjfxrt.jar;D:devtoolsJavajdk1.8.0_191jrelibextlocaledata.jar;D:devtoolsJavajdk1.8.0_191jrelibext
ashorn.jar;D:devtoolsJavajdk1.8.0_191jrelibextsunec.jar;D:devtoolsJavajdk1.8.0_191jrelibextsunjce_provider.jar;D:devtoolsJavajdk1.8.0_191jrelibextsunmscapi.jar;D:devtoolsJavajdk1.8.0_191jrelibextsunpkcs11.jar;D:devtoolsJavajdk1.8.0_191jrelibextzipfs.jar;D:devtoolsJavajdk1.8.0_191jrelibjavaws.jar;D:devtoolsJavajdk1.8.0_191jrelibjce.jar;D:devtoolsJavajdk1.8.0_191jrelibjfr.jar;D:devtoolsJavajdk1.8.0_191jrelibjfxswt.jar;D:devtoolsJavajdk1.8.0_191jrelibjsse.jar;D:devtoolsJavajdk1.8.0_191jrelibmanagement-agent.jar;D:devtoolsJavajdk1.8.0_191jrelibplugin.jar;D:devtoolsJavajdk1.8.0_191jrelib
esources.jar;D:devtoolsJavajdk1.8.0_191jrelib
t.jar;D:workspaceSpark_Test	argetclasses;C:Users一期一会.m2
epositoryorgapachesparkspark-core_2.112.4.0spark-core_2.11-2.4.0.jar;C:Users一期一会.m2
epositoryorgapacheavroavro1.8.2avro-1.8.2.jar;C:Users一期一会.m2
epositoryorgcodehausjacksonjackson-core-asl1.9.13jackson-core-asl-1.9.13.jar;C:Users一期一会.m2
epositoryorgcodehausjacksonjackson-mapper-asl1.9.13jackson-mapper-asl-1.9.13.jar;C:Users一期一会.m2
epositorycom	houghtworksparanamerparanamer2.7paranamer-2.7.jar;C:Users一期一会.m2
epositoryorgapachecommonscommons-compress1.8.1commons-compress-1.8.1.jar;C:Users一期一会.m2
epositoryorg	ukaanixz1.5xz-1.5.jar;C:Users一期一会.m2
epositoryorgapacheavroavro-mapred1.8.2avro-mapred-1.8.2-hadoop2.jar;C:Users一期一会.m2
epositoryorgapacheavroavro-ipc1.8.2avro-ipc-1.8.2.jar;C:Users一期一会.m2
epositorycommons-codeccommons-codec1.9commons-codec-1.9.jar;C:Users一期一会.m2
epositorycom	witterchill_2.110.9.3chill_2.11-0.9.3.jar;C:Users一期一会.m2
epositorycomesotericsoftwarekryo-shaded4.0.2kryo-shaded-4.0.2.jar;C:Users一期一会.m2
epositorycomesotericsoftwareminlog1.3.0minlog-1.3.0.jar;C:Users一期一会.m2
epositoryorgobjenesisobjenesis2.5.1objenesis-2.5.1.jar;C:Users一期一会.m2
epositorycom	witterchill-java0.9.3chill-java-0.9.3.jar;C:Users一期一会.m2
epositoryorgapachexbeanxbean-asm6-shaded4.8xbean-asm6-shaded-4.8.jar;C:Users一期一会.m2
epositoryorgapachehadoophadoop-client2.6.5hadoop-client-2.6.5.jar;C:Users一期一会.m2
epositoryorgapachehadoophadoop-common2.6.5hadoop-common-2.6.5.jar;C:Users一期一会.m2
epositorycommons-clicommons-cli1.2commons-cli-1.2.jar;C:Users一期一会.m2
epositoryxmlencxmlenc0.52xmlenc-0.52.jar;C:Users一期一会.m2
epositorycommons-httpclientcommons-httpclient3.1commons-httpclient-3.1.jar;C:Users一期一会.m2
epositorycommons-iocommons-io2.4commons-io-2.4.jar;C:Users一期一会.m2
epositorycommons-collectionscommons-collections3.2.2commons-collections-3.2.2.jar;C:Users一期一会.m2
epositorycommons-configurationcommons-configuration1.6commons-configuration-1.6.jar;C:Users一期一会.m2
epositorycommons-digestercommons-digester1.8commons-digester-1.8.jar;C:Users一期一会.m2
epositorycommons-beanutilscommons-beanutils1.7.0commons-beanutils-1.7.0.jar;C:Users一期一会.m2
epositorycommons-beanutilscommons-beanutils-core1.8.0commons-beanutils-core-1.8.0.jar;C:Users一期一会.m2
epositorycomgooglecodegsongson2.2.4gson-2.2.4.jar;C:Users一期一会.m2
epositoryorgapachehadoophadoop-auth2.6.5hadoop-auth-2.6.5.jar;C:Users一期一会.m2
epositoryorgapachehttpcomponentshttpclient4.2.5httpclient-4.2.5.jar;C:Users一期一会.m2
epositoryorgapachehttpcomponentshttpcore4.2.4httpcore-4.2.4.jar;C:Users一期一会.m2
epositoryorgapachedirectoryserverapacheds-kerberos-codec2.0.0-M15apacheds-kerberos-codec-2.0.0-M15.jar;C:Users一期一会.m2
epositoryorgapachedirectoryserverapacheds-i18n2.0.0-M15apacheds-i18n-2.0.0-M15.jar;C:Users一期一会.m2
epositoryorgapachedirectoryapiapi-asn1-api1.0.0-M20api-asn1-api-1.0.0-M20.jar;C:Users一期一会.m2
epositoryorgapachedirectoryapiapi-util1.0.0-M20api-util-1.0.0-M20.jar;C:Users一期一会.m2
epositoryorgapachecuratorcurator-client2.6.0curator-client-2.6.0.jar;C:Users一期一会.m2
epositoryorghtracehtrace-core3.0.4htrace-core-3.0.4.jar;C:Users一期一会.m2
epositoryorgapachehadoophadoop-hdfs2.6.5hadoop-hdfs-2.6.5.jar;C:Users一期一会.m2
epositoryorgmortbayjettyjetty-util6.1.26jetty-util-6.1.26.jar;C:Users一期一会.m2
epositoryxercesxercesImpl2.9.1xercesImpl-2.9.1.jar;C:Users一期一会.m2
epositoryxml-apisxml-apis1.3.04xml-apis-1.3.04.jar;C:Users一期一会.m2
epositoryorgapachehadoophadoop-mapreduce-client-app2.6.5hadoop-mapreduce-client-app-2.6.5.jar;C:Users一期一会.m2
epositoryorgapachehadoophadoop-mapreduce-client-common2.6.5hadoop-mapreduce-client-common-2.6.5.jar;C:Users一期一会.m2
epositoryorgapachehadoophadoop-yarn-client2.6.5hadoop-yarn-client-2.6.5.jar;C:Users一期一会.m2
epositoryorgapachehadoophadoop-yarn-server-common2.6.5hadoop-yarn-server-common-2.6.5.jar;C:Users一期一会.m2
epositoryorgapachehadoophadoop-mapreduce-client-shuffle2.6.5hadoop-mapreduce-client-shuffle-2.6.5.jar;C:Users一期一会.m2
epositoryorgapachehadoophadoop-yarn-api2.6.5hadoop-yarn-api-2.6.5.jar;C:Users一期一会.m2
epositoryorgapachehadoophadoop-mapreduce-client-core2.6.5hadoop-mapreduce-client-core-2.6.5.jar;C:Users一期一会.m2
epositoryorgapachehadoophadoop-yarn-common2.6.5hadoop-yarn-common-2.6.5.jar;C:Users一期一会.m2
epositoryjavaxxmlindjaxb-api2.2.2jaxb-api-2.2.2.jar;C:Users一期一会.m2
epositoryjavaxxmlstreamstax-api1.0-2stax-api-1.0-2.jar;C:Users一期一会.m2
epositoryorgcodehausjacksonjackson-jaxrs1.9.13jackson-jaxrs-1.9.13.jar;C:Users一期一会.m2
epositoryorgcodehausjacksonjackson-xc1.9.13jackson-xc-1.9.13.jar;C:Users一期一会.m2
epositoryorgapachehadoophadoop-mapreduce-client-jobclient2.6.5hadoop-mapreduce-client-jobclient-2.6.5.jar;C:Users一期一会.m2
epositoryorgapachehadoophadoop-annotations2.6.5hadoop-annotations-2.6.5.jar;C:Users一期一会.m2
epositoryorgapachesparkspark-launcher_2.112.4.0spark-launcher_2.11-2.4.0.jar;C:Users一期一会.m2
epositoryorgapachesparkspark-kvstore_2.112.4.0spark-kvstore_2.11-2.4.0.jar;C:Users一期一会.m2
epositoryorgfusesourceleveldbjnileveldbjni-all1.8leveldbjni-all-1.8.jar;C:Users一期一会.m2
epositorycomfasterxmljacksoncorejackson-core2.6.7jackson-core-2.6.7.jar;C:Users一期一会.m2
epositorycomfasterxmljacksoncorejackson-annotations2.6.7jackson-annotations-2.6.7.jar;C:Users一期一会.m2
epositoryorgapachesparkspark-network-common_2.112.4.0spark-network-common_2.11-2.4.0.jar;C:Users一期一会.m2
epositoryorgapachesparkspark-network-shuffle_2.112.4.0spark-network-shuffle_2.11-2.4.0.jar;C:Users一期一会.m2
epositoryorgapachesparkspark-unsafe_2.112.4.0spark-unsafe_2.11-2.4.0.jar;C:Users一期一会.m2
epositoryjavaxactivationactivation1.1.1activation-1.1.1.jar;C:Users一期一会.m2
epositoryorgapachecuratorcurator-recipes2.6.0curator-recipes-2.6.0.jar;C:Users一期一会.m2
epositoryorgapachecuratorcurator-framework2.6.0curator-framework-2.6.0.jar;C:Users一期一会.m2
epositorycomgoogleguavaguava16.0.1guava-16.0.1.jar;C:Users一期一会.m2
epositoryorgapachezookeeperzookeeper3.4.6zookeeper-3.4.6.jar;C:Users一期一会.m2
epositoryjavaxservletjavax.servlet-api3.1.0javax.servlet-api-3.1.0.jar;C:Users一期一会.m2
epositoryorgapachecommonscommons-lang33.5commons-lang3-3.5.jar;C:Users一期一会.m2
epositoryorgapachecommonscommons-math33.4.1commons-math3-3.4.1.jar;C:Users一期一会.m2
epositorycomgooglecodefindbugsjsr3051.3.9jsr305-1.3.9.jar;C:Users一期一会.m2
epositoryorgslf4jslf4j-api1.7.16slf4j-api-1.7.16.jar;C:Users一期一会.m2
epositoryorgslf4jjul-to-slf4j1.7.16jul-to-slf4j-1.7.16.jar;C:Users一期一会.m2
epositoryorgslf4jjcl-over-slf4j1.7.16jcl-over-slf4j-1.7.16.jar;C:Users一期一会.m2
epositorylog4jlog4j1.2.17log4j-1.2.17.jar;C:Users一期一会.m2
epositoryorgslf4jslf4j-log4j121.7.16slf4j-log4j12-1.7.16.jar;C:Users一期一会.m2
epositorycom
ingcompress-lzf1.0.3compress-lzf-1.0.3.jar;C:Users一期一会.m2
epositoryorgxerialsnappysnappy-java1.1.7.1snappy-java-1.1.7.1.jar;C:Users一期一会.m2
epositoryorglz4lz4-java1.4.0lz4-java-1.4.0.jar;C:Users一期一会.m2
epositorycomgithublubenzstd-jni1.3.2-2zstd-jni-1.3.2-2.jar;C:Users一期一会.m2
epositoryorg
oaringbitmapRoaringBitmap0.5.11RoaringBitmap-0.5.11.jar;C:Users一期一会.m2
epositorycommons-netcommons-net3.1commons-net-3.1.jar;C:Users一期一会.m2
epositoryorgjson4sjson4s-jackson_2.113.5.3json4s-jackson_2.11-3.5.3.jar;C:Users一期一会.m2
epositoryorgjson4sjson4s-core_2.113.5.3json4s-core_2.11-3.5.3.jar;C:Users一期一会.m2
epositoryorgjson4sjson4s-ast_2.113.5.3json4s-ast_2.11-3.5.3.jar;C:Users一期一会.m2
epositoryorgjson4sjson4s-scalap_2.113.5.3json4s-scalap_2.11-3.5.3.jar;C:Users一期一会.m2
epositoryorgscala-langmodulesscala-xml_2.111.0.6scala-xml_2.11-1.0.6.jar;C:Users一期一会.m2
epositoryorgglassfishjerseycorejersey-client2.22.2jersey-client-2.22.2.jar;C:Users一期一会.m2
epositoryjavaxws
sjavax.ws.rs-api2.0.1javax.ws.rs-api-2.0.1.jar;C:Users一期一会.m2
epositoryorgglassfishhk2hk2-api2.4.0-b34hk2-api-2.4.0-b34.jar;C:Users一期一会.m2
epositoryorgglassfishhk2hk2-utils2.4.0-b34hk2-utils-2.4.0-b34.jar;C:Users一期一会.m2
epositoryorgglassfishhk2externalaopalliance-repackaged2.4.0-b34aopalliance-repackaged-2.4.0-b34.jar;C:Users一期一会.m2
epositoryorgglassfishhk2externaljavax.inject2.4.0-b34javax.inject-2.4.0-b34.jar;C:Users一期一会.m2
epositoryorgglassfishhk2hk2-locator2.4.0-b34hk2-locator-2.4.0-b34.jar;C:Users一期一会.m2
epositoryorgjavassistjavassist3.18.1-GAjavassist-3.18.1-GA.jar;C:Users一期一会.m2
epositoryorgglassfishjerseycorejersey-common2.22.2jersey-common-2.22.2.jar;C:Users一期一会.m2
epositoryjavaxannotationjavax.annotation-api1.2javax.annotation-api-1.2.jar;C:Users一期一会.m2
epositoryorgglassfishjerseyundles
epackagedjersey-guava2.22.2jersey-guava-2.22.2.jar;C:Users一期一会.m2
epositoryorgglassfishhk2osgi-resource-locator1.0.1osgi-resource-locator-1.0.1.jar;C:Users一期一会.m2
epositoryorgglassfishjerseycorejersey-server2.22.2jersey-server-2.22.2.jar;C:Users一期一会.m2
epositoryorgglassfishjerseymediajersey-media-jaxb2.22.2jersey-media-jaxb-2.22.2.jar;C:Users一期一会.m2
epositoryjavaxvalidationvalidation-api1.1.0.Finalvalidation-api-1.1.0.Final.jar;C:Users一期一会.m2
epositoryorgglassfishjerseycontainersjersey-container-servlet2.22.2jersey-container-servlet-2.22.2.jar;C:Users一期一会.m2
epositoryorgglassfishjerseycontainersjersey-container-servlet-core2.22.2jersey-container-servlet-core-2.22.2.jar;C:Users一期一会.m2
epositoryio
etty
etty-all4.1.17.Final
etty-all-4.1.17.Final.jar;C:Users一期一会.m2
epositoryio
etty
etty3.9.9.Final
etty-3.9.9.Final.jar;C:Users一期一会.m2
epositorycomclearspringanalyticsstream2.7.0stream-2.7.0.jar;C:Users一期一会.m2
epositoryiodropwizardmetricsmetrics-core3.1.5metrics-core-3.1.5.jar;C:Users一期一会.m2
epositoryiodropwizardmetricsmetrics-jvm3.1.5metrics-jvm-3.1.5.jar;C:Users一期一会.m2
epositoryiodropwizardmetricsmetrics-json3.1.5metrics-json-3.1.5.jar;C:Users一期一会.m2
epositoryiodropwizardmetricsmetrics-graphite3.1.5metrics-graphite-3.1.5.jar;C:Users一期一会.m2
epositorycomfasterxmljacksoncorejackson-databind2.6.7.1jackson-databind-2.6.7.1.jar;C:Users一期一会.m2
epositorycomfasterxmljacksonmodulejackson-module-scala_2.112.6.7.1jackson-module-scala_2.11-2.6.7.1.jar;C:Users一期一会.m2
epositoryorgscala-langscala-reflect2.11.8scala-reflect-2.11.8.jar;C:Users一期一会.m2
epositorycomfasterxmljacksonmodulejackson-module-paranamer2.7.9jackson-module-paranamer-2.7.9.jar;C:Users一期一会.m2
epositoryorgapacheivyivy2.4.0ivy-2.4.0.jar;C:Users一期一会.m2
epositoryorooro2.0.8oro-2.0.8.jar;C:Users一期一会.m2
epository
et
azorvinepyrolite4.13pyrolite-4.13.jar;C:Users一期一会.m2
epository
etsfpy4jpy4j0.10.7py4j-0.10.7.jar;C:Users一期一会.m2
epositoryorgapachesparkspark-tags_2.112.4.0spark-tags_2.11-2.4.0.jar;C:Users一期一会.m2
epositoryorgapachecommonscommons-crypto1.0.0commons-crypto-1.0.0.jar;C:Users一期一会.m2
epositoryorgspark-projectsparkunused1.0.0unused-1.0.0.jar;C:Users一期一会.m2
epositoryorgapachesparkspark-sql_2.112.4.0spark-sql_2.11-2.4.0.jar;C:Users一期一会.m2
epositorycomunivocityunivocity-parsers2.7.3univocity-parsers-2.7.3.jar;C:Users一期一会.m2
epositoryorgapachesparkspark-sketch_2.112.4.0spark-sketch_2.11-2.4.0.jar;C:Users一期一会.m2
epositoryorgapachesparkspark-catalyst_2.112.4.0spark-catalyst_2.11-2.4.0.jar;C:Users一期一会.m2
epositoryorgscala-langmodulesscala-parser-combinators_2.111.1.0scala-parser-combinators_2.11-1.1.0.jar;C:Users一期一会.m2
epositoryorgcodehausjaninojanino3.0.9janino-3.0.9.jar;C:Users一期一会.m2
epositoryorgcodehausjaninocommons-compiler3.0.9commons-compiler-3.0.9.jar;C:Users一期一会.m2
epositoryorgantlrantlr4-runtime4.7antlr4-runtime-4.7.jar;C:Users一期一会.m2
epositoryorgapacheorcorc-core1.5.2orc-core-1.5.2-nohive.jar;C:Users一期一会.m2
epositoryorgapacheorcorc-shims1.5.2orc-shims-1.5.2.jar;C:Users一期一会.m2
epositorycomgoogleprotobufprotobuf-java2.5.0protobuf-java-2.5.0.jar;C:Users一期一会.m2
epositorycommons-langcommons-lang2.6commons-lang-2.6.jar;C:Users一期一会.m2
epositoryioairliftaircompressor0.10aircompressor-0.10.jar;C:Users一期一会.m2
epositoryorgapacheorcorc-mapreduce1.5.2orc-mapreduce-1.5.2-nohive.jar;C:Users一期一会.m2
epositoryorgapacheparquetparquet-column1.10.0parquet-column-1.10.0.jar;C:Users一期一会.m2
epositoryorgapacheparquetparquet-common1.10.0parquet-common-1.10.0.jar;C:Users一期一会.m2
epositoryorgapacheparquetparquet-encoding1.10.0parquet-encoding-1.10.0.jar;C:Users一期一会.m2
epositoryorgapacheparquetparquet-hadoop1.10.0parquet-hadoop-1.10.0.jar;C:Users一期一会.m2
epositoryorgapacheparquetparquet-format2.4.0parquet-format-2.4.0.jar;C:Users一期一会.m2
epositoryorgapacheparquetparquet-jackson1.10.0parquet-jackson-1.10.0.jar;C:Users一期一会.m2
epositoryorgapachearrowarrow-vector0.10.0arrow-vector-0.10.0.jar;C:Users一期一会.m2
epositoryorgapachearrowarrow-format0.10.0arrow-format-0.10.0.jar;C:Users一期一会.m2
epositoryorgapachearrowarrow-memory0.10.0arrow-memory-0.10.0.jar;C:Users一期一会.m2
epositoryjoda-timejoda-time2.9.9joda-time-2.9.9.jar;C:Users一期一会.m2
epositorycomcarrotsearchhppc0.7.2hppc-0.7.2.jar;C:Users一期一会.m2
epositorycomvlkanflatbuffers1.2.0-3f79e055flatbuffers-1.2.0-3f79e055.jar;C:Users一期一会.m2
epositoryorgscala-langscala-library2.11.0scala-library-2.11.0.jar com.hikvision.learn.TestEmptyDataFrame
+---+----+---+-----+
| id|name|age|birth|
+---+----+---+-----+
+---+----+---+-----+

+---+----+---+-----+
| id|name|age|birth|
+---+----+---+-----+
+---+----+---+-----+

++
||
++
++


Process finished with exit code 0