How To Calculate The Ewm Correlation Coefs After Groupby

E.G, I have following csv data (There are more than one group g in practice): G,T,x,y g,1,3,4 g,2,4,5 g,3,6,1 g,4,7,2 g,5,8,3 g,6,9,8 I want to calculate the exponential weighted

Solution 1:

The problem is that corr returns the correlation matrix. So when you do ewm.corr it returns a panel. So you need to extract the extra diagonal components to get the correlation coefficient.

An explicit solution with a loop is:

res = pd.concat([el.ewm(halflife = 3).corr().xs('x', axis = 1).loc['y', :] for key, el in dat.groupby(level = 'G')])

This is clearer if you inspect el.ewm(halflife = 3).corr():

el.ewm(halflife = 3).corr()
Out[54]: 
<class 'pandas.core.panel.Panel'>
Dimensions: 6 (items) x 2 (major_axis) x 2 (minor_axis)
Items axis: (g, 1) to (g, 6)
Major_axis axis: x to y
Minor_axis axis: x to y

Following this answer I realised you can avoid the loop by using the expression above but within an apply rather than transform method on the grouped object.

dat.groupby(level='G').apply(lambda x: x.ewm(halflife=3).corr().xs('x', axis = 1).loc['y', :]).T

In both cases, I obtain the expected output:

res
Out[55]: 
G  T
g  1         NaN
   2    1.000000
   3   -0.867510
   4   -0.792758
   5   -0.510885
   6    0.413379
Name: y, dtype: float64