Summary Statistics in Pandas:

  • Let’s revisit the climate dataset that we looked at previously.
  • Below is the code to read the data from the CSV file and display the summary of the DataFrame.
python
1import pandas as pd
2
3dfc = pd.read_csv('./data/IthacaDailyClimate2018.csv')
4
5dfc.info()

Output:

1<class 'pandas.core.frame.DataFrame'>
2RangeIndex: 365 entries, 0 to 364
3Data columns (total 7 columns):
4  #   Column               Non-Null Count  Dtype
5 ---  ------               --------------  -----
6 0   Date                 365 non-null    object
7 1   Maximum Temperature  365 non-null    int64
8 2   Minimum Temperature  365 non-null    int64
9 3   Average Temperature  365 non-null    float64
10 4   Precipitation        365 non-null    float64
11 5   Snowfall             365 non-null    float64
12 6   Snow Depth           365 non-null    float64
13dtypes: float64(4), int64(2), object(1)
14memory usage: 20.1+ KB
  • Next, the head() method is used to display the first few rows of the DataFrame.
python
5dfc.head()

Output:

1         Date  Maximum Temperature  ...  Snowfall  Snow Depth
20  2018-01-01                    5  ...       1.0         3.0
31  2018-01-02                   13  ...       0.6         4.0
42  2018-01-03                   19  ...       0.0         4.0
53  2018-01-04                   22  ...       0.0         3.0
64  2018-01-05                   18  ...       1.2         4.0
7
8[5 rows x 7 columns]
  • With the exception of the Date column, the rest of the data in df are all numerical, either ints or floats.
  • Therefore, we can get useful statistical summaries of these different numerical data columns.

The min() method:

  • Let’s imagine we want to know the smallest value in each of the columns of the dataframe.
python
5minvals = dfc.min()
6print(minvals)

Output:

1Date                   2018-01-01
2Maximum Temperature             2
3Minimum Temperature            -9
4Average Temperature          -0.5
5Precipitation                 0.0
6Snowfall                      0.0
7Snow Depth                    0.0
8dtype: object

The mean() method:

  • To compute the mean of each column, we can use the mean() method.
python
5meanvals = dfc.mean(numeric_only=True)
6print(meanvals)

Output:

1Maximum Temperature    55.832877
2Minimum Temperature    37.389041
3Average Temperature    46.610959
4Precipitation           0.114137
5Snowfall                0.209041
6Snow Depth              0.843836
7dtype: float64

The sum() method:

  • To compute the sum of each column, we can use the sum() method.
python
5sumvals = dfc.sum()
6print(sumvals)

Output:

1Date                   2018-01-012018-01-022018-01-032018...
2Maximum Temperature                                    20379
3Minimum Temperature                                    13647
4Average Temperature                                  17013.0
5Precipitation                                          41.66
6Snowfall                                                76.3
7Snow Depth                                             308.0
8dtype: object

  • Which of the entries in sumvals involves the concatenation of multiple strings instead of the summation of numerical values?

  • Which two entries in sumvals are most useful?

The describe() method:

  • The describe method on a dataframe combines several different summary statistics into one object, a new DataFrame.
python
5dfcdesc = dfc.describe()
6print(dfcdesc)

Output:

1       Maximum Temperature  Minimum Temperature  ...    Snowfall
2count           365.000000           365.000000  ...  365.000000
3mean             55.832877            37.389041  ...    0.209041
4std              21.403939            18.927185  ...    0.813857
5min               2.000000            -9.000000  ...    0.000000
625%              37.000000            25.000000  ...    0.000000
750%              57.000000            37.000000  ...    0.000000
875%              75.000000            54.000000  ...    0.000000
9max              94.000000            70.000000  ...    8.600000
10
11[8 rows x 6 columns]
  • The entries in dfcdesc contain summaries of several quantities, including “count”, “mean”, “std”, “min”, “25%”, “50%”, “75%”, and “max”.
  • Collectively, these quantities help to describe the dataset.
  • The rows indicating “25%”, “50%”, and “75%” are quantile values, meaning that they describe how the data separate into groups of different sizes.