import numpy as np
import pandas as pd
Data contained in pandas objects can be combined together in a number of ways:
-
pandas.merge connects rows in DataFrame based on one or more keys. This will be familiar to users of SQL or other relational databases, as it impliemnts(工具) database join oprations.
-
pandas.concat concatenates or "stacks" together objects along an axis.
-
The combine_first instance method enables splicing(拼接) together overlapping data to fill in missing values in one object with values from another.
I will address each of these and give a number of examples. They'll be utilized in examples throughout the rest of the book.
SQL风格的Join
merge or join operations combine datasets by linking rows using one or more keys. These operations are central to relational database(e.g. SQL-based). The merge function in pandas is the main entry point for using theses algorithms on your data.
Let's start with a simple example:
df1 = pd.DataFrame({
'key': 'b, b, a, c, a, a, b'.split(','),
'data1': range(7)
})
df2 = pd.DataFrame({
'key': ['a', 'b', 'd'],
'data2': range(3)
})
df1
df2
|
key |
data1 |
| 0 |
b |
0 |
| 1 |
b |
1 |
| 2 |
a |
2 |
| 3 |
c |
3 |
| 4 |
a |
4 |
| 5 |
a |
5 |
| 6 |
b |
6 |
|
key |
data2 |
| 0 |
a |
0 |
| 1 |
b |
1 |
| 2 |
d |
2 |
This is an example of a many to one join; the data in df1 has multiple rows labeled a and b, whereas(然而) df2 has only one row for each value in the key column. Calling merge with these objects we obtain:
"merge 默认是内连接, if 没有指定key..."
pd.merge(df1, df2) # data1, key, data2
'merge 默认是内连接, if 没有指定key...'
Note that I didn't specify which columns to join on. if that infomation is not specified, merge uses the overlapping columns names as keys. It's a good practice to specify explicitly, though:
(cj. 好像不是这样的哦)
"内连接走一波, 相同的记录才会保留哦, 跟作者的不一样"
pd.merge(df2, df1, on='key') # data1, key, data2
'内连接走一波, 相同的记录才会保留哦, 跟作者的不一样'
# cj test
pd.merge(df1, df2, on='key', how='left')
|
key |
data1 |
data2 |
| 0 |
b |
0 |
1.0 |
| 1 |
b |
1 |
NaN |
| 2 |
a |
2 |
NaN |
| 3 |
c |
3 |
NaN |
| 4 |
a |
4 |
NaN |
| 5 |
a |
5 |
NaN |
| 6 |
b |
6 |
NaN |
If the column names are different in each object, you can specify them separately:
(两个df的键不同, 进行合并时可以分别指定)
df3 = pd.DataFrame({
'lkey': 'a b a c a a b'.split(),
'data1': range(7)
})
df4 = pd.DataFrame({
'rkey': ['a', 'b', 'd'],
'data2': range(3)
})
pd.merge(df3, df4, left_on='lkey', right_on='rkey')
|
lkey |
data1 |
rkey |
data2 |
| 0 |
a |
0 |
a |
0 |
| 1 |
a |
2 |
a |
0 |
| 2 |
a |
4 |
a |
0 |
| 3 |
a |
5 |
a |
0 |
| 4 |
b |
1 |
b |
1 |
| 5 |
b |
6 |
b |
1 |
You may notice that the 'c' and 'd' values and associate data are missing from the result. By defualt merge does an inner join; the keys in the result are intersection. or the common set found in both tables. Other possible options are left, right and outer. The outer join takes the union of the keys, combining the effect of applying both left and right joins.
(merge 默认是内连接, 相关的还有左, 右, 外连接;
外连接是包含了左,右连接哦)
"默认以所有的键, 其实就是穷举所有的可能结果而已"
pd.merge(df1, df2, how='outer')
'默认以所有的键, 其实就是穷举所有的可能结果而已'
|
key |
data1 |
data2 |
| 0 |
b |
0.0 |
1.0 |
| 1 |
b |
1.0 |
NaN |
| 2 |
b |
6.0 |
NaN |
| 3 |
a |
2.0 |
NaN |
| 4 |
a |
4.0 |
NaN |
| 5 |
a |
5.0 |
NaN |
| 6 |
c |
3.0 |
NaN |
| 7 |
a |
NaN |
0.0 |
| 8 |
d |
NaN |
2.0 |
See Table 8-1 for a summary of the options for how.
| Option |
Behavior |
| 'inner' |
Use only the key combinations observed in both tables |
| 'left' |
Use all combinations found in the left table |
| 'right' |
Use all key combinations found in the right table |
| 'outer' |
Use all key combinations observed in both tables together |
Many-to-Many merges have well-defined, though not necessarily intuitive(直觉的), behavior. Here's an example:
df1 = pd.DataFrame({
'key': 'b b a c a b'.split(),
'data1': range(6)
})
df2 = pd.DataFrame({
'key': 'a b a b d'.split(),
'data2': range(5)
})
df1
df2
|
key |
data1 |
| 0 |
b |
0 |
| 1 |
b |
1 |
| 2 |
a |
2 |
| 3 |
c |
3 |
| 4 |
a |
4 |
| 5 |
b |
5 |
|
key |
data2 |
| 0 |
a |
0 |
| 1 |
b |
1 |
| 2 |
a |
2 |
| 3 |
b |
3 |
| 4 |
d |
4 |
pd.merge(df1, df2, how='inner')
|
key |
data1 |
data2 |
| 0 |
b |
0 |
1 |
| 1 |
b |
0 |
3 |
| 2 |
b |
1 |
1 |
| 3 |
b |
1 |
3 |
| 4 |
b |
5 |
1 |
| 5 |
b |
5 |
3 |
| 6 |
a |
2 |
0 |
| 7 |
a |
2 |
2 |
| 8 |
a |
4 |
0 |
| 9 |
a |
4 |
2 |
pd.merge(df1, df2, on='key', how='left')
|
key |
data1 |
data2 |
| 0 |
b |
0 |
1.0 |
| 1 |
b |
0 |
3.0 |
| 2 |
b |
1 |
1.0 |
| 3 |
b |
1 |
3.0 |
| 4 |
a |
2 |
0.0 |
| 5 |
a |
2 |
2.0 |
| 6 |
c |
3 |
NaN |
| 7 |
a |
4 |
0.0 |
| 8 |
a |
4 |
2.0 |
| 9 |
b |
5 |
1.0 |
| 10 |
b |
5 |
3.0 |
To merge with multiple keys, pass a list of columns names:
left = pd.DataFrame({'key1': ['foo', 'foo', 'bar'],
'key2': ['one', 'two', 'one'],
'lval': [1, 2, 3]})
right = pd.DataFrame({'key1': ['foo', 'foo', 'bar', 'bar'],
'key2': ['one', 'one', 'one', 'two'],
'rval': [4, 5, 6, 7]})
left
right
|
key1 |
key2 |
lval |
| 0 |
foo |
one |
1 |
| 1 |
foo |
two |
2 |
| 2 |
bar |
one |
3 |
|
key1 |
key2 |
rval |
| 0 |
foo |
one |
4 |
| 1 |
foo |
one |
5 |
| 2 |
bar |
one |
6 |
| 3 |
bar |
two |
7 |
"outer 所有可能的结果, 支持多个keys"
pd.merge(left, right, on=['key1', 'key2'], how='outer')
'outer 所有可能的结果, 支持多个keys'
|
key1 |
key2 |
lval |
rval |
| 0 |
foo |
one |
1.0 |
4.0 |
| 1 |
foo |
one |
1.0 |
5.0 |
| 2 |
foo |
two |
2.0 |
NaN |
| 3 |
bar |
one |
3.0 |
6.0 |
| 4 |
bar |
two |
NaN |
7.0 |
To determine which key combinations will appear in the result depending on the choice of merge method, think of the multiple keys as forming an array fo tuples to be used as a single join key.
When you are joining columns-on-columns, the indexes on the passed DataFrame objects are discarded.
pd.merge(left, right, on='key1')
|
key1 |
key2_x |
lval |
key2_y |
rval |
| 0 |
foo |
one |
1 |
one |
4 |
| 1 |
foo |
one |
1 |
one |
5 |
| 2 |
foo |
two |
2 |
one |
4 |
| 3 |
foo |
two |
2 |
one |
5 |
| 4 |
bar |
one |
3 |
one |
6 |
| 5 |
bar |
one |
3 |
two |
7 |
pd.merge(left, right, on='key1', suffixes=('_left', '_right'))
|
key1 |
key2_left |
lval |
key2_right |
rval |
| 0 |
foo |
one |
1 |
one |
4 |
| 1 |
foo |
one |
1 |
one |
5 |
| 2 |
foo |
two |
2 |
one |
4 |
| 3 |
foo |
two |
2 |
one |
5 |
| 4 |
bar |
one |
3 |
one |
6 |
| 5 |
bar |
one |
3 |
two |
7 |
See Table 8-2 for an argument reference on merge. Joining using the DataFrame's row index is the subject of the next section.
- left
- right
- how
- on
- left_on
- right_on
- left_index
- right_index
- sort
- suffixes 添加后缀
- copy
- indecator
按Index合并
In some cases, the merge key(s) in a DataFrame will be found on its index, In this case, you can pass left_index=True or right_index=True to indicate that the index should be used as the merge key:
left1 = pd.DataFrame({
'key': ['a', 'b', 'a', 'a', 'b', 'c'],
'value': range(6)
})
right1 = pd.DataFrame({'group_val':[3.5, 7]}, index=['a', 'b'])
left1
right1
|
key |
value |
| 0 |
a |
0 |
| 1 |
b |
1 |
| 2 |
a |
2 |
| 3 |
a |
3 |
| 4 |
b |
4 |
| 5 |
c |
5 |
pd.merge(left1, right1, left_on='key', right_index=True)
|
key |
value |
group_val |
| 0 |
a |
0 |
3.5 |
| 2 |
a |
2 |
3.5 |
| 3 |
a |
3 |
3.5 |
| 1 |
b |
1 |
7.0 |
| 4 |
b |
4 |
7.0 |
按轴水平/垂直合并
Another kind of data combination operation is referred to interchangeably as concat-enation, binding, or stacking, NumPy's concatenate function can do this with NumPy arrays:
arr = np.arange(12).reshape((3,4))
arr
array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]])
"直接水平拼接"
np.concatenate([arr, arr], axis=1)
'直接水平拼接'
array([[ 0, 1, 2, 3, 0, 1, 2, 3],
[ 4, 5, 6, 7, 4, 5, 6, 7],
[ 8, 9, 10, 11, 8, 9, 10, 11]])
不再继续往下扩展了, 就目前我工作中用得最多的还是Merge, Join,在处理表vlookup的场景下.还有就是涉及垂直/水平拼接的 pd.concat(), np.vstack() 和 np.hstack(), 结合SQL来配合使用,就非常灵活和高效了.