如何解决熊猫：最快的分组方式，可对分组进行最大值和求和
这是我要实现的目标：
input:
B C D
A
x z 1 10
x z 2 11
x z 3 12
y s 4 13
y s 5 14
output:
B C D sum
A
x z 3 12 33
y s 5 14 27
我有以下代码。
import pandas as pd
df = pd.DataFrame({'A': ['x','x','y','y'],'B': ['z','z','s','s'],'C': [1,2,3,4,5],'D': [10,11,12,13,14]})
df = df.set_index('A')
df['sum'] = df.groupby('A')['D'].transform('sum')
idx = df.groupby(['A'])['C'].transform(max) == df['C']
df= df[idx]
我正在相当大的Dataframe上执行此操作。但是要花很长时间，尤其是第一批人。 有什么办法可以加快这个过程？ 由于我要做的就是将总和放在一个组中，并将该行保留在最大不同列的位置。
解决方法
总的来说，我相信您的方法是可行的，除了一些改进：
java.lang.NullPointerException
at java.util.Objects.requireNonNull(Objects.java:203)
at com.example.filemanager.MainActivity$FileAdaptor.getView(MainActivity.java:500)
at android.widget.AbsListView.obtainView(AbsListView.java:3271)
at android.widget.ListView.makeAndAddView(ListView.java:2238)
at android.widget.ListView.fillDown(ListView.java:838)
at android.widget.ListView.fillFromTop(ListView.java:900)
at android.widget.ListView.layoutChildren(ListView.java:1974)
at android.widget.AbsListView.onLayout(AbsListView.java:3041)
at android.view.View.layout(View.java:23754)
at android.view.ViewGroup.layout(ViewGroup.java:7277)
at android.widget.LinearLayout.setChildFrame(LinearLayout.java:1829)
at android.widget.LinearLayout.layoutHorizontal(LinearLayout.java:1818)
at android.widget.LinearLayout.onLayout(LinearLayout.java:1584)
at android.view.View.layout(View.java:23754)
at android.view.ViewGroup.layout(ViewGroup.java:7277)
at android.widget.RelativeLayout.onLayout(RelativeLayout.java:1103)
at android.view.View.layout(View.java:23754)
at android.view.ViewGroup.layout(ViewGroup.java:7277)
at androidx.drawerlayout.widget.DrawerLayout.onLayout(DrawerLayout.java:1231)
at android.view.View.layout(View.java:23754)
at android.view.ViewGroup.layout(ViewGroup.java:7277)
at android.widget.FrameLayout.layoutChildren(FrameLayout.java:332)
at android.widget.FrameLayout.onLayout(FrameLayout.java:270)
at android.view.View.layout(View.java:23754)
at android.view.ViewGroup.layout(ViewGroup.java:7277)
at androidx.appcompat.widget.ActionBarOverlayLayout.onLayout(ActionBarOverlayLayout.java:530)
at android.view.View.layout(View.java:23754)
at android.view.ViewGroup.layout(ViewGroup.java:7277)
at android.widget.FrameLayout.layoutChildren(FrameLayout.java:332)
at android.widget.FrameLayout.onLayout(FrameLayout.java:270)
at android.view.View.layout(View.java:23754)
at android.view.ViewGroup.layout(ViewGroup.java:7277)
at android.widget.LinearLayout.setChildFrame(LinearLayout.java:1829)
at android.widget.LinearLayout.layoutVertical(LinearLayout.java:1673)
at android.widget.LinearLayout.onLayout(LinearLayout.java:1582)
at android.view.View.layout(View.java:23754)
at android.view.ViewGroup.layout(ViewGroup.java:7277)
at android.widget.FrameLayout.layoutChildren(FrameLayout.java:332)
at android.widget.FrameLayout.onLayout(FrameLayout.java:270)
at com.android.internal.policy.DecorView.onLayout(DecorView.java:1059)
at android.view.View.layout(View.java:23754)
at android.view.ViewGroup.layout(ViewGroup.java:7277)
at android.view.ViewRootImpl.performLayout(ViewRootImpl.java:3679)
at android.view.ViewRootImpl.performTraversals(ViewRootImpl.java:3139)
at android.view.ViewRootImpl.doTraversal(ViewRootImpl.java:2200)
at android.view.ViewRootImpl$TraversalRunnable.run(ViewRootImpl.java:8960)
at android.view.Choreographer$CallbackRecord.run(Choreographer.java:996)
at android.view.Choreographer.doCallbacks(Choreographer.java:794)
at android.view.Choreographer.doFrame(Choreographer.java:729)
at android.view.Choreographer$FrameDisplayEventReceiver.run(Choreographer.java:981)
at android.os.Handler.handleCallback(Handler.java:883)
at android.os.Handler.dispatchMessage(Handler.java:100)
at android.os.Looper.loop(Looper.java:237)
at android.app.ActivityThread.main(ActivityThread.java:7814)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:493)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1068)
另一个改进是您可以进行懒惰分组：
For X=2 To 300000
If Cells(x,9) = "M" Then
Cells(x,10) = "1"
Else
Cells(x,10) = "0"
End If
Next x
df.groupby('B').agg(B=('C','max'),C=('D',Sum=('D','sum')).rename_axis('A',axis=0)
B C Sum
A
s 5 14 27
z 3 12 33
尝试一下：
tmp = df.groupby('A').agg(
idx = ('C','idxmax'),D = ('D','sum')
)
result = df.loc[tmp['idx']].set_index('A').assign(D=tmp['D'])
wwnde 似乎是迄今为止最快的解决方案。
我的贡献（比原始方法快，但比其他方法慢）：
df['sum'] = df.groupby('A')['D'].transform('sum')
df = df.loc[df.groupby('A').C.idxmax()]
使用@Quang Hoang提示可以使速度更快：
groups = df.groupby('A')
df['sum'] = groups['D'].transform('sum')
df = df.loc[ groups.C.idxmax()].set_index('A')
基准
# Import libraries
import numpy as np
import pandas as pd
from time import time
import seaborn as sns
import matplotlib.pyplot as plt
# Make fake data with 10M rows and 10 target-groups
values = np.arange(10**7)
groups = [f'group{i}' for i in range(1,11) for j in range(int(len(values)/10))]
unused_col = [letter for letter in 'abcdefghij' for j in range(int(len(values)/10))]
df = pd.DataFrame(dict(A=groups,B=unused_col,C=values*0.01,D=values))
# Define functions
def caina_max(df):
df = df.copy()
groups = df.groupby('A')
df['sum'] = groups['D'].transform('sum')
df = df.loc[ groups.C.idxmax()].set_index('A')
return df
def Code_Different(df):
df = df.copy()
tmp = df.groupby('A').agg(
idx = ('C','sum'))
df = df.loc[tmp['idx']].set_index('A').assign(Sum=tmp['D'])
return df
def Muriel(df):
df = df.copy()
df = df.set_index('A')
df1 = df.groupby(['A','B']).max()
df2 = df.groupby('A')['D'].sum()
df = df1.join(df2,lsuffix='_caller',rsuffix='_other')
df = df.reset_index(level=1).rename(columns={'D_caller': 'D','D_other': 'Sum'})
return df
def Quang_Hoang(df):
df = df.copy()
groups = df.groupby('A')
df['sum'] = groups['D'].transform('sum')
idx = groups['C'].transform('max') == df['C']
df = df[idx].set_index('A')
return df
def valenzio(df):
df.copy()
df = df.set_index('A')
df['sum'] = df.groupby('A')['D'].transform('sum')
idx = df.groupby(['A'])['C'].transform(max) == df['C']
df= df[idx]
return df
def wwnde(df):
df = df.copy()
df = df.groupby('B').agg(B=('C',axis=0)
return df
# Benchmark
functions = caina_max,Code_Different,Muriel,Quang_Hoang,valenzio,wwnde
times = {f.__name__: [] for f in functions}
for func in functions:
fname = func.__name__
for i in range(100): # reduce this range for faster reproducibility
t0=time()
func(df)
t1=time()
times[fname].append((t1-t0))
# Benchmark table
df_benchmark = pd.DataFrame(times).agg([np.mean,np.std,max,min]).T.sort_values('mean').round(3)
df_benchmark.index.name = 'Approach'
# Benchmark figure
plt.figure(figsize=(12,8))
sns.boxplot(data=pd.melt(pd.DataFrame(times)),x='variable',y='value',)
plt.xticks(rotation=45)
plt.title(label='Benchmark',fontweight="bold",pad=20)
plt.ylabel('Time in seconds',labelpad=10)
plt.xlabel('')
plt.show()
输出：
mean std max min
Approach
wwnde 1.165 0.009 1.198 1.148
Quang_Hoang 1.488 0.039 1.659 1.439
Code_Different 1.532 0.027 1.638 1.500
caina_max 1.680 0.030 1.813 1.641
valenzio 2.847 0.036 3.030 2.805
Muriel 3.598 0.025 3.666 3.549
这应该更快：
df = pd.DataFrame({'A': ['x','x','y','y'],'B': ['z','z','s','s'],'C': [1,2,3,4,5],'D': [10,11,12,13,14]})
df = df.set_index('A')
df1 = df.groupby(['A','B']).max()
df2 = df.groupby('A')['D'].sum()
df1.join(df2,rsuffix='_other')
版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 dio@foxmail.com 举报，一经查实，本站将立刻删除。
