zoukankan      html  css  js  c++  java
  • [Spark][Python]DataFrame的左右连接例子

    [Spark][Python]DataFrame的左右连接例子

    $ hdfs dfs -cat people.json

    {"name":"Alice","pcode":"94304"}
    {"name":"Brayden","age":30,"pcode":"94304"}
    {"name":"Carla","age":19,"pcoe":"10036"}
    {"name":"Diana","age":46}
    {"name":"Etienne","pcode":"94104"}

    $ hdfs dfs -cat pcodes.json

    {"pcode":"10036","city":"New York","state":"NY"}
    {"pcode":"87501","city":"Santa Fe","state":"NM"}
    {"pcode":"94304","city":"Palo Alto","state":"CA"}
    {"pcode":"94104","city":"San Francisco","state":"CA"}

    $pyspark

    sqlContext = HiveContext(sc)
    peopleDF = sqlContext.read.json("people.json")
    peopleDF.limit(5).show()

    +----+-------+-----+-----+
    | age| name|pcode| pcoe|
    +----+-------+-----+-----+
    |null| Alice|94304| null|
    | 30|Brayden|94304| null|
    | 19| Carla| null|10036|
    | 46| Diana| null| null|
    |null|Etienne|94104| null|
    +----+-------+-----+-----+

    sqlContext = HiveContext(sc)
    pcodesDF = sqlContext.read.json("pcodes.json")
    pcodesDF.limit(5).show()

    +-------------+-----+-----+
    | city|pcode|state|
    +-------------+-----+-----+
    | New York|10036| NY|
    | Santa Fe|87501| NM|
    | Palo Alto|94304| CA|
    |San Francisco|94104| CA|
    +-------------+-----+-----+

    mydf000 = peopleDF.join(pcodesDF,"pcode")
    mydf000.limit(5).show()

    +-----+----+-------+----+-------------+-----+
    |pcode| age| name|pcoe| city|state|
    +-----+----+-------+----+-------------+-----+
    |94304|null| Alice|null| Palo Alto| CA|
    |94304| 30|Brayden|null| Palo Alto| CA|
    |94104|null|Etienne|null|San Francisco| CA|
    +-----+----+-------+----+-------------+-----+

    mydf001=peopleDF.join(pcodesDF,"pcode","leftsemi")
    mydf001.limit(5).show()

    +-----+----+-------+----+
    |pcode| age| name|pcoe|
    +-----+----+-------+----+
    |94304|null| Alice|null|
    |94304| 30|Brayden|null|
    |94104|null|Etienne|null|
    +-----+----+-------+----+

    mydf002=peopleDF.join(pcodesDF,"pcode","left_outer")
    mydf002.limit(5).show()

    +-----+----+-------+-----+-------------+-----+
    |pcode| age| name| pcoe| city|state|
    +-----+----+-------+-----+-------------+-----+
    |94304|null| Alice| null| Palo Alto| CA|
    |94304| 30|Brayden| null| Palo Alto| CA|
    | null| 19| Carla|10036| null| null|
    | null| 46| Diana| null| null| null|
    |94104|null|Etienne| null|San Francisco| CA|
    +-----+----+-------+-----+-------------+-----+

    mydf003=peopleDF.join(pcodesDF,"pcode","right_outer")
    mydf003.limit(5).show()

    +-----+----+-------+----+-------------+-----+
    |pcode| age| name|pcoe| city|state|
    +-----+----+-------+----+-------------+-----+
    |10036|null| null|null| New York| NY|
    |87501|null| null|null| Santa Fe| NM|
    |94304|null| Alice|null| Palo Alto| CA|
    |94304| 30|Brayden|null| Palo Alto| CA|
    |94104|null|Etienne|null|San Francisco| CA|
    +-----+----+-------+----+-------------+-----+
  • 相关阅读:
    windows7上使用docker容器
    centos7 docker镜像加速器配置
    用列表生成器打印九九乘法表
    -bash: wget: command not found的两种解决方法
    centos7 Dockerfile安装nginx
    centos6.5关闭防火墙命令
    centos7开机启动tomcat7
    centos7安装tomcat7
    CentOS7防火墙firewalld
    poj_3662 最小化第k大的值
  • 原文地址:https://www.cnblogs.com/gaojian/p/7633001.html
Copyright © 2011-2022 走看看