zoukankan      html  css  js  c++  java
  • pandas实现hive的lag和lead函数 以及 first_value和last_value函数

    lag和lead VS shift 

    该函数的格式如下:

    • 第一个参数为列名,
    • 第二个参数为往上第n行(可选,默认为1),
    • 第三个参数为默认值(当往上第n行为NULL时候,取默认值,如不指定,则为NULL)

    lag

    lag(字段名,N,默认值) over(partition by 分组字段 order by 排序字段 排序方式) 

    lead

    lead(字段名,N,默认值) over(partition by 分组字段 order by 排序字段 排序方式)

    案例:

    select 
      cookieid, 
      createtime, 
      url, 
      row_number() over (partition by cookieid order by createtime) as rn, 
      LAG(createtime,1,'1970-01-01 00:00:00') over (partition by cookieid order by createtime) as last_1_time, 
      LAG(createtime,2) over (partition by cookieid order by createtime) as last_2_time 
    from cookie.cookie4

    select 
      cookieid, 
      createtime, 
      url, 
      row_number() over (partition by cookieid order by createtime) as rn, 
      LEAD(createtime,1,'1970-01-01 00:00:00') over (partition by cookieid order by createtime) as next_1_time, 
      LEAD(createtime,2) over (partition by cookieid order by createtime) as next_2_time 
    from cookie.cookie4;

    窗口函数的pandas实现

    pandas中使用shift函数来实现lag/lead函数

    import pandas as pd
    df=pd.read_csv('c:/Users/WQBin/Desktop/data.csv',engine='python', names=['cookieid','createtime','url'])
    df['last_url'] = df.sort_values('createtime').groupby('cookieid')['url'].shift(1)
    df['next_url'] = df.sort_values('createtime').groupby('cookieid')['url'].shift(-1)
    df.sort_values(by=['cookieid','createtime'])

    first_value和 last_value VS first()和last()

    • FIRST_VALUE 返回组中数据窗口的第一个值
      •   FIRST_VALUE ( [scalar_expression )OVER ( [ partition_by_clause ] order_by_clause )
    • LAST_VALUE 返回组中数据窗口的最后一个值
      •    LAST_VALUE ( [scalar_expression )OVER ( [ partition_by_clause order_by_clause )
    select 
      cookieid, 
      createtime, 
      url, 
      row_number() over (partition by cookieid order by createtime) as rn, 
      first_value(url) over (partition by cookieid order by createtime) as first1 
    from cookie.cookie4;

    select 
      cookieid, 
      createtime, 
      url, 
      row_number() over (partition by cookieid order by createtime) as rn, 
      last_value(url) over (partition by cookieid order by createtime) as last1 
    from cookie.cookie4;

    窗口函数的pandas实现

    df.sort_values(['createtime'], ascending=[ 1]).groupby(['cookieid']).first()

    df.sort_values(['createtime'], ascending=[ 1]).groupby(['cookieid']).last()

  • 相关阅读:
    CH4302 Interval GCD
    20201017 模拟赛总结
    LOJ#2610. 「NOIP2013」货车运输
    POJ3696 The Luckiest number
    20200925 小练习总结
    更新一波状态
    AFO
    【扩展lucas定理】
    【LCT维护子树信息】uoj207 共价大爷游长沙
    [bzoj4589]Hard Nim(FWT快速沃尔什变化+快速幂)
  • 原文地址:https://www.cnblogs.com/wqbin/p/11987346.html
Copyright © 2011-2022 走看看