Python pandas Q&A video series by Data School
YouTube playlist and GitHub repository
Table of contents
- What is pandas?
- How do I read a tabular data file into pandas?
- How do I select a pandas Series from a DataFrame?
- Why do some pandas commands end with parentheses (and others don't)?
- How do I rename columns in a pandas DataFrame?
- How do I remove columns from a pandas DataFrame?
- How do I sort a pandas DataFrame or a Series?
- How do I filter rows of a pandas DataFrame by column value?
- How do I apply multiple filter criteria to a pandas DataFrame?
- Your pandas questions answered!
- How do I use the "axis" parameter in pandas?
- How do I use string methods in pandas?
- How do I change the data type of a pandas Series?
- When should I use a "groupby" in pandas?
- How do I explore a pandas Series?
- How do I handle missing values in pandas?
- What do I need to know about the pandas index? (Part 1)
- What do I need to know about the pandas index? (Part 2)
- How do I select multiple rows and columns from a pandas DataFrame?
- When should I use the "inplace" parameter in pandas?
- How do I make my pandas DataFrame smaller and faster?
- How do I use pandas with scikit-learn to create Kaggle submissions?
- More of your pandas questions answered!
- How do I create dummy variables in pandas?
- How do I work with dates and times in pandas?
- How do I find and remove duplicate rows in pandas?
- How do I avoid a SettingWithCopyWarning in pandas?
- How do I change display options in pandas?
- How do I create a pandas DataFrame from another object?
- How do I apply a function to a pandas Series or DataFrame?
In [1]:
# 传统方式
import pandas as pd
1. What is pandas? (video)
2. How do I read a tabular data file into pandas? (video)
In [2]:
# 直接从URL中读取Chipotle订单的数据集，并将结果存储在数据库中
url1 = "https://raw.githubusercontent.com/justmarkham/pandas-videos/master/data/chipotle.tsv"
#定义地址
orders =pd.read_table(url1)#使用read_table()打开
In [3]:
# 检查前5行
orders.head()
Out[3]:
Documentation for read_table
In [4]:
# 读取电影评论员的数据集（修改read_table的默认参数值）
user_cols = ['user_id','age','gender','occupation','zipcode']#定义列名
url2 = "https://raw.githubusercontent.com/justmarkham/pandas-videos/master/data/u.user"
#定义地址
#users=pd.read_table(url2,sep='|',header=None,names= user_clos,skiprows=2,skipfooter=3)
users = pd.read_table('http://bit.ly/movieusers', sep='|', header=None, names=user_cols)
#加入参数sep 分隔符，header 头部 标题，names 列名
In [5]:
# 检查前5行
users.head()
Out[5]:
3. How do I select a pandas Series from a DataFrame? (video)
In [6]:
# 将UFO报告的数据集读入DataFrame
url3 = "https://raw.githubusercontent.com/justmarkham/pandas-videos/master/data/ufo.csv"#定义列名
ufo = pd.read_table(url3, sep=',')
In [7]:
# #用read_table打开csv文件，区别是 read_csv直接是用逗号隔开
ufo = pd.read_csv(url3)
In [8]:
# 检查前5行
ufo.head()
Out[8]:
In [9]:
# #用括号法查看Series
ufo['City']
# #用点法查看Series，要注意 名字里面有空格或者是python专有字符的时候不能用，但是方便输入
ufo.City
Out[9]:
括号表示法总是有效，而点表示法有局限性：
- 如果系列名称中有空格，则点符号不起作用
- 如果系列与DataFrame方法或属性（如'head'或'shape'）具有相同的名称，则点符号不起作用
- 点符号不能用于定义新series的名
In [10]:
# #这里的拼接也不能用点的方法
ufo['Location'] = ufo.City + ', ' + ufo.State
ufo.head()
Out[10]:
4. Why do some pandas commands end with parentheses (and others don't)? (video)
In [11]:
# 将顶级IMDb电影的数据集读入DataFrame
url4="https://raw.githubusercontent.com/justmarkham/pandas-videos/master/data/imdb_1000.csv"
movies = pd.read_csv(url4)
#方法以括号结尾，而属性则没有：
In [12]:
# 示例方法：显示前5行
movies.head()
Out[12]:
In [13]:
#示例方法：计算摘要统计信息
movies.describe()
Out[13]:
In [14]:
movies.describe(include=['object'])
Out[14]:
In [15]:
# 示例属性：行数和列数
movies.shape
Out[15]:
In [16]:
# 示例属性：每列的数据类型
movies.dtypes
Out[16]:
In [17]:
# 使用describe方法的可选参数来仅汇总'object'列
movies.describe(include=['object'])
Out[17]:
Documentation for describe
5. How do I rename columns in a pandas DataFrame? (video)
In [18]:
# 检查列名称
ufo.columns
Out[18]:
In [19]:
# 使用'rename'方法重命名其中两列
ufo.rename(columns={'Colors Reported':'Colors_Reported', 'Shape Reported':'Shape_Reported'}, inplace=True)
ufo.columns
Out[19]:
Documentation for rename
In [20]:
# 通过覆盖'columns'属性替换所有列名
ufo = pd.read_table(url3, sep=',')
ufo_cols = ['city', 'colors reported', 'shape reported', 'state', 'time']
ufo.columns = ufo_cols
ufo.columns
Out[20]:
In [21]:
# 使用'names'参数替换文件读取过程中的列名
ufo = pd.read_csv(url3, header=0, names=ufo_cols)
ufo.columns
Out[21]:
Documentation for read_csv
In [22]:
ufo.columns = ufo.columns.str.replace(' ', '_') #如何批量修改替换使得列名无空格
ufo.columns
Out[22]:
Documentation for str.replace
6. How do I remove columns from a pandas DataFrame? (video)
In [35]:
ufo = pd.read_table(url3, sep=',')
ufo.head()
Out[35]:
In [37]:
# #axis=1 是纵向，inplace = True：不创建新的对象，直接对原始对象进行修改；
ufo.drop('Colors Reported', axis=1, inplace=True)
ufo.head()
Out[37]:
Documentation for drop
In [38]:
# 一次删除多个列
ufo.drop(['City', 'State'], axis=1, inplace=True)
ufo.head()
Out[38]:
In [39]:
# 一次删除多行（axis = 0表示行）
ufo.drop([0, 1], axis=0, inplace=True)
ufo.head()
#删除4行 按标签，axis=0 是横向，默认为横向，但建议写出来
Out[39]:
7. How do I sort a pandas DataFrame or a Series? (video)
In [40]:
movies.head()
Out[40]:
#注意：以下任何排序方法都不会影响基础数据。 （换句话说，排序是暂时的）。
In [41]:
#排序单个Series
movies.title.sort_values().head()
Out[41]:
In [42]:
# #排序单个Series 倒序
movies.title.sort_values(ascending=False).head()
Out[42]:
Documentation for sort_values for a Series. (Prior to version 0.17, use order instead.)
In [43]:
# #以单个Series排序DataFrame
movies.sort_values('title').head()
Out[43]:
In [44]:
# 改为按降序排序
movies.sort_values('title', ascending=False).head()
Out[44]:
Documentation for sort_values for a DataFrame. (Prior to version 0.17, use sort instead.)
In [45]:
# 首先按'content_rating'，然后按duration'排序DataFrame
movies.sort_values(['content_rating', 'duration']).head()
Out[45]:
Summary of changes to the sorting API in pandas 0.17
8. How do I filter rows of a pandas DataFrame by column value? (video)
In [46]:
movies.head()
Out[46]:
In [47]:
# 检查行数和列数
movies.shape
Out[47]:
##目标：过滤DataFrame行，仅显示“持续时间”至少为200分钟的电影
In [48]:
#
#先展示一个比较复杂的方法，用一个for循环制造一个和原数据一样行数，判断每一行是否符合条件，列表元素均为boolean
#创建一个列表，其中每个元素引用一个DataFrame行：如果行满足条件，则返回true，否则返回False
booleans = []
for length in movies.duration:
if length >= 200:
booleans.append(True)
else:
booleans.append(False)
In [49]:
# 确认列表与DataFrame的长度相同
len(booleans)
Out[49]:
In [50]:
# 检查前五个列表元素
booleans[0:5]
Out[50]:
In [51]:
# 将列表转换为Series
is_long = pd.Series(booleans)
is_long.head()
Out[51]:
In [52]:
# 使用带有布尔Series的括号表示法告诉DataFrame movies[is_long]要显示哪些行
movies[is_long]
Out[52]:
In [53]:
# 简化上面的步骤：不需要编写for循环来创建is_long'
is_long = movies.duration >= 200
movies[is_long]#运用这种写法，pandas就知道，按照这个series去筛选
# 或等效地，将其写在一行（无需创建'is_long'对象）
movies[movies.duration >= 200]
Out[53]:
In [54]:
# 从过滤后的DataFrame中选择“流派”系列
movies[movies.duration >= 200].genre
# 或者等效地，使用'loc'方法
movies.loc[movies.duration >= 200, 'genre']
Out[54]:
Documentation for loc
9. How do I apply multiple filter criteria to a pandas DataFrame? (video)
In [55]:
# read a dataset of top-rated IMDb movies into a DataFrame
movies = pd.read_csv('http://bit.ly/imdbratings')
movies.head()
Out[55]:
In [56]:
# 过滤DataFrame仅显示“持续时间”至少为200分钟的电影
movies[movies.duration >= 200]
Out[56]:
理解逻辑运算符：
and：仅当运算符的两边都为True时才为真or：如果运算符的任何一侧为True，则为真
In [57]:
print(True and True)
print(True and False)
print(False and False)
In [58]:
print(True or True)
print(True or False)
print(False or False)
在pandas中指定多个过滤条件的规则：
使用＆而不是和 使用|而不是或 在每个条件周围添加括号以指定评估顺序
Goal: Further filter the DataFrame of long movies (duration >= 200) to only show movies which also have a 'genre' of 'Drama'
In [59]:
# 使用'＆'运算符指定两个条件都是必需的
movies[(movies.duration >=200) & (movies.genre == 'Drama')]
Out[59]:
In [60]:
# I不正确：使用'|'运算符会展示长或戏剧的电影
movies[(movies.duration >=200) | (movies.genre == 'Drama')].head()
Out[60]:
##过滤原始数据框以显示“类型”为“犯罪”或“戏剧”或“动作”的电影
In [61]:
# 使用'|'运算符指定行可以匹配三个条件中的任何一个
movies[(movies.genre == 'Crime') | (movies.genre == 'Drama') | (movies.genre == 'Action')].head(10)
# 用isin等效
movies[movies.genre.isin(['Crime', 'Drama', 'Action'])].head(10)
Out[61]:
Documentation for isin
10. Your pandas questions answered! (video)
Question: When reading from a file, how do I read in only a subset of the columns?
In [62]:
url3 = "https://raw.githubusercontent.com/justmarkham/pandas-videos/master/data/ufo.csv"#定义列名
ufo = pd.read_csv(url3)#用read_csv打开csv文件
ufo.columns
Out[62]:
In [63]:
# 列名筛选
ufo = pd.read_csv(url3, usecols=['City', 'State'])
# 用位置切片等效
ufo = pd.read_csv(url3, usecols=[0, 4])
ufo.columns
Out[63]:
Question: When reading from a file, how do I read in only a subset of the rows?
In [64]:
# 读3行数据
ufo = pd.read_csv(url3, nrows=3)
ufo
Out[64]:
Documentation for read_csv
Question: How do I iterate through a Series?
In [65]:
# Series可直接迭代（如列表）
for c in ufo.City:
print(c)
Question: How do I iterate through a DataFrame?
In [66]:
# 可以使用各种方法迭代DataFrame
for index, row in ufo.iterrows():
print(index, row.City, row.State)
Documentation for iterrows
Question: How do I drop all non-numeric columns from a DataFrame?
In [67]:
# 将酒精消耗数据集读入DataFrame，并检查数据类型
url7= 'https://raw.githubusercontent.com/justmarkham/pandas-videos/master/data/drinks.csv'
drinks = pd.read_csv(url7)
drinks.dtypes
Out[67]:
In [68]:
# 仅包含DataFrame中的数字列
import numpy as np
drinks.select_dtypes(include=[np.number]).dtypes
Out[68]:
Documentation for select_dtypes
Question: How do I know whether I should pass an argument as a string or a list?
In [69]:
# 描述所有数字列
drinks.describe()
Out[69]:
In [70]:
# 传递字符串'all'来描述所有列
drinks.describe(include='all')
Out[70]:
In [71]:
# 传递数据类型列表以仅描述多个类型
drinks.describe(include=['object', 'float64'])
Out[71]:
In [72]:
# 即使您只想描述单个数据类型，也要传递一个列表
drinks.describe(include=['object'])
Out[72]:
Documentation for describe
11. How do I use the "axis" parameter in pandas? (video)
In [73]:
url7= 'https://raw.githubusercontent.com/justmarkham/pandas-videos/master/data/drinks.csv'
drinks = pd.read_csv(url7)
drinks.head()
Out[73]:
In [74]:
# drop a column (temporarily)
drinks.drop('continent', axis=1).head()
Out[74]:
Documentation for drop
In [75]:
# 删除一列（暂时）
drinks.drop(2, axis=0).head()
Out[75]:
使用axis参数引用行或列时：
axis 0表示行 axis 1指的是列
In [76]:
# 计算每个数字列的平均值
drinks.mean()
# 或等效地，明确指定轴
drinks.mean(axis=0)
Out[76]:
Documentation for mean
In [77]:
# 计算每一行的平均值
drinks.mean(axis=1).head()
Out[77]:
使用axis参数执行数学运算时：
- *axis0 *表示操作应“向下移动”行轴
- *axis1 *表示操作应“移过”列轴
In [78]:
# 'index' 等效 axis 0
drinks.mean(axis='index')
Out[78]:
In [79]:
# 'columns' 等效 axis 1
drinks.mean(axis='columns').head()
Out[79]:
12. How do I use string methods in pandas? (video)
In [80]:
url1 = "https://raw.githubusercontent.com/justmarkham/pandas-videos/master/data/chipotle.tsv"
#定义地址
orders =pd.read_table(url1)#使用read_table()打开
orders.head()
Out[80]:
In [81]:
# 在Python中访问字符串方法的常用方法
'hello'.upper()
Out[81]:
In [82]:
# spandas Series 的字符串方法通过'str'访问
orders.item_name.str.upper().head()
Out[82]:
In [83]:
# string方法'contains'检查子字符串并返回一个布尔Series
orders.item_name.str.contains('Chicken').head()
Out[83]:
In [84]:
# 布尔Series筛选DataFrame
orders[orders.item_name.str.contains('Chicken')].head()
Out[84]:
In [85]:
# 字符串方法可以链接在一起
orders.choice_description.str.replace('[', '').str.replace(']', '').head()
Out[85]:
In [86]:
# 许多pandas字符串方法支持正则表达式
orders.choice_description.str.replace('[[]]', '').head()
Out[86]:
String handling section of the pandas API reference
13. How do I change the data type of a pandas Series? (video)
In [87]:
url7= 'https://raw.githubusercontent.com/justmarkham/pandas-videos/master/data/drinks.csv'
drinks = pd.read_csv(url7)
drinks.head()
Out[87]:
In [88]:
# 检查每个系列的数据类型
drinks.dtypes
Out[88]:
In [89]:
# 更改现有系列的数据类型
drinks['beer_servings'] = drinks.beer_servings.astype(float)
drinks.dtypes
Out[89]:
Documentation for astype
In [90]:
# 或者，在读取文件时更改系列的数据类型
drinks = pd.read_csv(url7, dtype={'beer_servings':float})
drinks.dtypes
Out[90]:
In [91]:
orders = pd.read_table(url1)
orders.head()
Out[91]:
In [92]:
# 检查每个系列的数据类型
orders.dtypes
Out[92]:
In [93]:
# 将字符串转换为数字以进行数学运算
orders.item_price.str.replace('$', '').astype(float).mean()
Out[93]:
In [94]:
# 字符串方法'contains'检查子字符串并返回一个布尔系列
orders.item_name.str.contains('Chicken').head()
Out[94]:
In [95]:
# 将布尔系列转换为整数（False = 0，True = 1）
orders.item_name.str.contains('Chicken').astype(int).head()
Out[95]:
14. When should I use a "groupby" in pandas? (video)
In [96]:
drinks = pd.read_csv(url7)
drinks.head()
Out[96]:
In [97]:
# 计算整个数据集中的平均beer_servings
drinks.beer_servings.mean()
Out[97]:
In [98]:
# 计算非洲国家的平均beer_servings
drinks[drinks.continent=='Africa'].beer_servings.mean()
Out[98]:
In [99]:
#计算每个大陆的平均beer_servings
drinks.groupby('continent').beer_servings.mean()
Out[99]:
Documentation for groupby
In [100]:
# 其他聚合函数（例如'max'）也可以与groupby一起使用
drinks.groupby('continent').beer_servings.max()
Out[100]:
In [101]:
# 多个聚合函数可以同时应用
drinks.groupby('continent').beer_servings.agg(['count', 'mean', 'min', 'max'])
Out[101]:
Documentation for agg
In [102]:
# 不指定列，就会算出所有数值列
drinks.groupby('continent').mean()
Out[102]:
In [103]:
# 允许绘图出现在jupyter notebook中
%matplotlib inline
In [104]:
# 直接在上面的DataFrame的并排条形图
drinks.groupby('continent').mean().plot(kind='bar')
Out[104]:
Documentation for plot
15. How do I explore a pandas Series? (video)
In [105]:
# read a dataset of top-rated IMDb movies into a DataFrame
url4="https://raw.githubusercontent.com/justmarkham/pandas-videos/master/data/imdb_1000.csv"
movies = pd.read_csv(url4)
movies.head()
Out[105]:
In [106]:
# 检查数据类型
movies.dtypes
Out[106]:
探索非数字系列
In [107]:
# 计算最常见值的非空值，唯一值和频率
movies.genre.describe()
Out[107]:
Documentation for describe
In [108]:
# 数Series中每个值发生的次数
movies.genre.value_counts()
Out[108]:
Documentation for value_counts
In [109]:
# 显示百分比而不是原始计数
movies.genre.value_counts(normalize=True)
Out[109]:
In [110]:
# '输出的是一个Series
type(movies.genre.value_counts())
Out[110]:
In [111]:
# 可以使用Series方法
movies.genre.value_counts().head()
Out[111]:
In [112]:
# 显示Series中唯一值
movies.genre.unique()
Out[112]:
In [113]:
#数Series中唯一值的数量
movies.genre.nunique()
Out[113]:
In [114]:
# 两个Series的交叉列表
pd.crosstab(movies.genre, movies.content_rating)
Out[114]:
Documentation for crosstab
探索数字系列：
In [115]:
# 计算各种汇总统计
movies.duration.describe()
Out[115]:
In [116]:
# 许多统计数据都是作为Series方法实现的
movies.duration.mean()
Out[116]:
Documentation for mean
In [117]:
# 'value_counts' 主要用于分类数据，而不是数字数据
movies.duration.value_counts().head()
Out[117]:
In [118]:
# 允许绘图出现在jupyter notebook中
%matplotlib inline
In [119]:
# 'duration'Series的直方图（显示数值变量的分布）
movies.duration.plot(kind='hist')
Out[119]:
In [120]:
# 'genre'Series'value_counts'的条形图
movies.genre.value_counts().plot(kind='bar')
Out[120]:
Documentation for plot