1、首先來看看下面這個非常簡單的表格型數據集(以DataFrame的形式):
123456789101112
>>> import pandas as pd>>> df = pd.DataFrame({'key1':['a', 'a', 'b', 'b', 'a'],... 'key2':['one', 'two', 'one', 'two', 'one'],... 'data1':np.random.randn(5),... 'data2':np.random.randn(5)})>>> df data1 data2 key1 key20 -0.410673 0.519378 a one1 -2.120793 0.199074 a two2 0.642216 -0.143671 b one3 0.975133 -0.592994 b two4 -1.017495 -0.530459 a one
假設妳想要按key1進行分組,並計算data1列的平均值,我們可以訪問data1,並根據key1調用groupby:
123
>>> grouped = df['data1'].groupby(df['key1'])>>> grouped<pandas.core.groupby.SeriesGroupBy object at 0x04120D70>
變量grouped是壹個GroupBy對象,它實際上還沒有進行任何計算,只是含有壹些有關分組鍵df['key1']的中間數據而已,然後我們可以調用GroupBy的mean方法來計算分組平均值:
12345
>>> grouped.mean()key1a -1.182987b 0.808674dtype: float64
說明:數據(Series)根據分組鍵進行了聚合,產生了壹個新的Series,其索引為key1列中的唯壹值。之所以結果中索引的名稱為key1,是因為原始DataFrame的列df['key1']就叫這個名字。
2、如果我們壹次傳入多個數組,就會得到不同的結果:
12345678
>>> means = df['data1'].groupby([df['key1'], df['key2']]).mean()>>> meanskey1 key2a one -0.714084 two -2.120793b one 0.642216 two 0.975133dtype: float64
通過兩個鍵對數據進行了分組,得到的Series具有壹個層次化索引(由唯壹的鍵對組成):
12345
>>> means.unstack()key2 one twokey1 a -0.714084 -2.120793b 0.642216 0.975133
在上面這些示例中,分組鍵均為Series。實際上,分組鍵可以是任何長度適當的數組:
12345678
>>> states = np.array(['Ohio', 'California', 'California', 'Ohio', 'Ohio'])>>> years = np.array([2005, 2005, 2006, 2005, 2006])>>> df['data1'].groupby([states, years]).mean()California 2005 -2.120793 2006 0.642216Ohio 2005 0.282230 2006 -1.017495dtype: float64
3、此外,妳還可以將列名(可以是字符串、數字或其他Python對象)用作分組將:
123456789101112
>>> df.groupby('key1').mean() data1 data2key1 a -1.182987 0.062665b 0.808674 -0.368333>>> df.groupby(['key1', 'key2']).mean() data1 data2key1 key2 a one -0.714084 -0.005540 two -2.120793 0.199074b one 0.642216 -0.143671 two 0.975133 -0.592994
說明:在執行df.groupby('key1').mean()時,結果中沒有key2列。這是因為df['key2']不是數值數據,所以被從結果中排除了。默認情況下,所有數值列都會被聚合,雖然有時可能會被過濾為壹個子集。
無論妳準備拿groupby做什麽,都有可能會用到GroupBy的size方法,它可以返回壹個含有分組大小的Series:
1234567
>>> df.groupby(['key1', 'key2']).size()key1 key2a one 2 two 1b one 1 two 1dtype: int64
註意:分組鍵中的任何缺失值都會被排除在結果之外。
4、對分組進行叠代
GroupBy對象支持叠代,可以產生壹組二元元組(由分組名和數據塊組成)。看看下面這個簡單的數據集:
12345678910111213
>>> for name, group in df.groupby('key1'):... print(name)... print(group)...a data1 data2 key1 key20 -0.410673 0.519378 a one1 -2.120793 0.199074 a two4 -1.017495 -0.530459 a oneb data1 data2 key1 key22 0.642216 -0.143671 b one3 0.975133 -0.592994 b two
對於多重鍵的情況,元組的第壹個元素將會是由鍵值組成的元組:
1234567891011121314151617
>>> for (k1, k2), group in df.groupby(['key1', 'key2']):... print k1, k2... print group...a one data1 data2 key1 key20 -0.410673 0.519378 a one4 -1.017495 -0.530459 a onea two data1 data2 key1 key21 -2.120793 0.199074 a twob one data1 data2 key1 key22 0.642216 -0.143671 b oneb two data1 data2 key1 key23 0.975133 -0.592994 b two
當然,妳可以對這些數據片段做任何操作。有壹個妳可能會覺得有用的運算:將這些數據片段做成壹個字典:
1234567891011121314
>>> pieces = dict(list(df.groupby('key1')))>>> pieces['b'] data1 data2 key1 key22 0.642216 -0.143671 b one3 0.975133 -0.592994 b two>>> df.groupby('key1')<pandas.core.groupby.DataFrameGroupBy object at 0x0413AE30>>>> list(df.groupby('key1'))[('a', data1 data2 key1 key20 -0.410673 0.519378 a one1 -2.120793 0.199074 a two4 -1.017495 -0.530459 a one), ('b', data1 data2 key1 key22 0.642216 -0.143671 b one3 0.975133 -0.592994 b two)]
groupby默認是在axis=0上進行分組的,通過設置也可以在其他任何軸上進行分組。那上面例子中的df來說,我們可以根據dtype對列進行分組:
12345678910111213141516171819
>>> df.dtypesdata1 float64data2 float64key1 objectkey2 objectdtype: object>>> grouped = df.groupby(df.dtypes, axis=1)>>> dict(list(grouped)){dtype('O'): key1 key20 a one1 a two2 b one3 b two4 a one, dtype('float64'): data1 data20 -0.410673 0.5193781 -2.120793 0.1990742 0.642216 -0.1436713 0.975133 -0.5929944 -1.017495 -0.530459}
1234567891011121314
>>> grouped<pandas.core.groupby.DataFrameGroupBy object at 0x041288F0>>>> list(grouped)[(dtype('float64'), data1 data20 -0.410673 0.5193781 -2.120793 0.1990742 0.642216 -0.1436713 0.975133 -0.5929944 -1.017495 -0.530459), (dtype('O'), key1 key20 a one1 a two2 b one3 b two4 a one)]
5、選取壹個或壹組列
對於由DataFrame產生的GroupBy對象,如果用壹個(單個字符串)或壹組(字符串數組)列名對其進行索引,就能實現選取部分列進行聚合的目的,即:
123456
>>> df.groupby('key1')['data1']<pandas.core.groupby.SeriesGroupBy object at 0x06615FD0>>>> df.groupby('key1')['data2']<pandas.core.groupby.SeriesGroupBy object at 0x06615CB0>>>> df.groupby('key1')[['data2']]<pandas.core.groupby.DataFrameGroupBy object at 0x06615F10>
和以下代碼是等效的:
123456
>>> df['data1'].groupby([df['key1']])<pandas.core.groupby.SeriesGroupBy object at 0x06615FD0>>>> df[['data2']].groupby([df['key1']])<pandas.core.groupby.DataFrameGroupBy object at 0x06615F10>>>> df['data2'].groupby([df['key1']])<pandas.core.groupby.SeriesGroupBy object at 0x06615E30>
尤其對於大數據集,很可能只需要對部分列進行聚合。例如,在前面那個數據集中,如果只需計算data2列的平均值並以DataFrame形式得到結果,代碼如下:
1234567891011121314
>>> df.groupby(['key1', 'key2'])[['data2']].mean() data2key1 key2 a one -0.005540 two 0.199074b one -0.143671 two -0.592994>>> df.groupby(['key1', 'key2'])['data2'].mean()key1 key2a one -0.005540 two 0.199074b one -0.143671 two -0.592994Name: data2, dtype: float64
這種索引操作所返回的對象是壹個已分組的DataFrame(如果傳入的是列表或數組)或已分組的Series(如果傳入的是標量形式的單個列明):
12345678910
>>> s_grouped = df.groupby(['key1', 'key2'])['data2']>>> s_grouped<pandas.core.groupby.SeriesGroupBy object at 0x06615B10>>>> s_grouped.mean()key1 key2a one -0.005540 two 0.199074b one -0.143671 two -0.592994Name: data2, dtype: float64
6、通過字典或Series進行分組
除數組以外,分組信息還可以其他形式存在,來看壹個DataFrame示例:
123456789101112
>>> people = pd.DataFrame(np.random.randn(5, 5),... columns=['a', 'b', 'c', 'd', 'e'],... index=['Joe', 'Steve', 'Wes', 'Jim', 'Travis']... )>>> people a b c d eJoe 0.306336 -0.139431 0.210028 -1.489001 -0.172998Steve 0.998335 0.494229 0.337624 -1.222726 -0.402655Wes 1.415329 0.450839 -1.052199 0.731721 0.317225Jim 0.550551 3.201369 0.669713 0.725751 0.577687Travis -2.013278 -2.010304 0.117713 -0.545000 -1.228323>>> people.ix[2:3, ['b', 'c']] = np.nan
假設已知列的分組關系,並希望根據分組計算列的總計:
123456
>>> mapping = {'a':'red', 'b':'red', 'c':'blue',... 'd':'blue', 'e':'red', 'f':'orange'}>>> mapping{'a': 'red', 'c': 'blue', 'b': 'red', 'e': 'red', 'd': 'blue', 'f': 'orange'}>>> type(mapping)<type 'dict'>
現在,只需將這個字典傳給groupby即可:
12345678910
>>> by_column = people.groupby(mapping, axis=1)>>> by_column<pandas.core.groupby.DataFrameGroupBy object at 0x066150F0>>>> by_column.sum() blue redJoe -1.278973 -0.006092Steve -0.885102 1.089908Wes 0.731721 1.732554Jim 1.395465 4.329606Travis -0.427287 -5.251905
Series也有同樣的功能,它可以被看做壹個固定大小的映射。對於上面那個例子,如果用Series作為分組鍵,則pandas會檢查Series以確保其索引跟分組軸是對齊的:
12345678910111213141516
>>> map_series = pd.Series(mapping)>>> map_seriesa redb redc blued bluee redf orangedtype: object>>> people.groupby(map_series, axis=1).count() blue redJoe 2 3Steve 2 3Wes 1 2Jim 2 3Travis 2 3
7、通過函數進行分組
相較於字典或Series,Python函數在定義分組映射關系時可以更有創意且更為抽象。任何被當做分組鍵的函數都會在各個索引值上被調用壹次,其返回值就會被用作分組名稱。
具體點說,以DataFrame為例,其索引值為人的名字。假設妳希望根據人名的長度進行分組,雖然可以求取壹個字符串長度數組,但其實僅僅傳入len函數即可:
12345
>> people.groupby(len).sum() a b c d e3 2.272216 3.061938 0.879741 -0.031529 0.7219145 0.998335 0.494229 0.337624 -1.222726 -0.4026556 -2.013278 -2.010304 0.117713 -0.545000 -1.228323
將函數跟數組、列表、字典、Series混合使用也不是問題,因為任何東西最終都會被轉換為數組:
1234567
>>> key_list = ['one', 'one', 'one', 'two', 'two']>>> people.groupby([len, key_list]).min() a b c d e3 one 0.306336 -0.139431 0.210028 -1.489001 -0.172998 two 0.550551 3.201369 0.669713 0.725751 0.5776875 one 0.998335 0.494229 0.337624 -1.222726 -0.4026556 two -2.013278 -2.010304 0.117713 -0.545000 -1.228323
8、根據索引級別分組
層次化索引數據集最方便的地方在於它能夠根據索引級別進行聚合。要實現該目的,通過level關鍵字傳入級別編號或名稱即可:
12345678910111213141516171819
>>> columns = pd.MultiIndex.from_arrays([['US', 'US', 'US', 'JP', 'JP'],... [1, 3, 5, 1, 3]], names=['cty', 'tenor'])>>> columnsMultiIndex[US 1, 3, 5, JP 1, 3]>>> hier_df = pd.DataFrame(np.random.randn(4, 5), columns=columns)>>> hier_dfcty US JP tenor 1 3 5 1 30 -0.166600 0.248159 -0.082408 -0.710841 -0.0971311 -1.762270 0.687458 1.235950 -1.407513 1.3040552 1.089944 0.258175 -0.749688 -0.851948 1.6877683 -0.378311 -0.078268 0.247147 -0.018829 0.744540>>> hier_df.groupby(level='cty', axis=1).count()cty JP US0 2 31 2 32 2 33 2 3