Enhancement: add "sum" to pandas.describe()

[darindillon]

Feature request: can we add "sum" to the pandas.describe() method? Not every dataset cares about sum, of course, but enough datasets do care about it that it seems worth adding to the default describe() method. (For instance, financial applications where you have multiple columns representing different scenarios, and you want to see how much profit would be earned of each scenario).

[jreback]

xref #7014

I am a bit - on this ATM as .aggregate/.agg will be easily able to do this pretty generically.

[jorisvandenbossche]

I also leaning to "not needed to add it" (you already have many statistics, adding one more makes the output more complex).

(for reference, R's summary also does not include sum, not that we cannot deviate from that of course)

[darindillon]

Well yes, you can always call .describe() to get some of the statistics and also call .agg() to get other statistics and alsocall something else to get other statistics and then merge them all together; but of course that's more work. Obviously, .describe() can't contain everything, it should contains the "commonly used" ones; and what's "commonly used" for you may be different than for me.
But I would argue that (especially for financial applications) "sum" is a very common metric it's worth including. (If this were an exotic metric like "kurtosis" that most people don't understand, we wouldn't include it; but "sum" is useful to lots of people).

If I did all the work to add this (including test cases, etc) and submitted a PR, would you be willing to consider including it?

[jorisvandenbossche]

It is difficult to assess, as I am also biased by the type of data I am working with, but I think sum is less generally applicable as the others.
For example, I work with time series data of concentrations. For such a case, all current statistics are useful (or at least mean something), while a sum makes no sense. And I have the feeling that this can be generalized, meaning that the current (distribution) statistics almost always make some sense, while for sum it much more depends on the type of data.

[TomAugspurger]

I'm also -0 on this. I think focusing on DataFrame.agg would be more useful than modifying describe.

[jorisvandenbossche]

@darindillon I am going to close this issue, as there does not seem much enthusiasm to add this. But have a look at #14668 as a possible way to more easily define a custom aggregation function.

[jreback]

In [9]: s = Series(range(5))

In [10]: s.iloc[2] = np.nan

In [11]: s.describe()
Out[11]: 
count    4.000000
mean     2.000000
std      1.825742
min      0.000000
25%      0.750000
50%      2.000000
75%      3.250000
max      4.000000
dtype: float64

In [12]: len(s)
Out[12]: 5

count already describes the number of non-nans

however you can see that len (or size) is 5 here.
So would not object to adding: len (or size), maybe nnulls (len - count).

[TomAugspurger]

I'm still -0 on adding sum, and I don't have a strong opinion on including the length or number of nulls. That information is included in DataFrame.info()

Changing the output of describe will be an API breaking change I think, as people could have been relying on the position of each statistic. I'd say the cost of breaking API isn't worth it in this case, but am willing to be convinced otherwise.

[palewire]

With an eye towards capturing the sentiments of the voices in this thread, I took a run at rewriting the documentation for the describe method to be more precise.

That effort was recently merged into the master branch and you can read it here.

The top-line change is to shift the definition of the method's mission from the broad "Generate various summary statistics, excluding NaN value" to something more specific.

Aiming for something that fit the current output and set down a boundary that would rationally exclude sum and any number of arbitrary statistics, the new definition is below. (My emphasis).

"Generates descriptive statistics that summarize the central tendency, dispersion and shape of a dataset’s distribution, excluding NaN values."

My hope is that it can provide a guide for evaluating future additions to the method that aren't based on the use cases of different users. Take a look and let me know what you think.

To me, if we accept this as the definition, we have a pretty strong case for excluding sum, but it raises for me the question of whether it might make sense to add common statistics that do match the definition, like mode, to the mix. If we appended them to the bottom of the current output, I think we could avoid the backwards breaking changes that @TomAugspurger rightly warns of.