Understand what the Apply, Applymap, and Aggregate functions do in Pandas
The Apply, Applymap, and Aggregate functions are frequently used to transform variables or the entire data in a way the user desires. I personally call these functions “Broadcasting Functions” because they allow us to broadcast a certain logic, say a custom function, to all the data points in the variable or data. In this article, I explain to you how these three functions differ and some examples to illustrate those points. We use the iconic Titanic Disaster dataset for those examples. Specifically, I used the dataset featured in OpenML which has a Public license.
We first import the pandas package and store the titanic training data in a variable named “df”.
# Dataset Source: OpenML; License(CC): Publicimport pandas as pd
df = pd.read_csv("../input/titanic/train.csv")
Simply put, the apply function in pandas is a variable level function where you can apply various transformations to transform a variable. Here, you can utilize the lambda function or custom function you make to create the transformation logic you want to apply. For instance, if you want to multiply by a 100 on the “Fare” variable for some reason, you can run the following code:
df['Fare'] = df['Fare'].apply(lambda x: x * 100)
With this mind, you can perform all kinds of cool transformations as long as you are able to wisely craft the lambda or custom function accurately in the way you want. The following is a code example which extracts month and day information from some string dates with a xxxx/mm/dd format.
data['last_review_month'] = data['last_review'].apply(lambda x: datetime.datetime.strptime(x, "%Y-%m-%d").month)data['last_review_day'] = data['last_review'].apply(lambda x: datetime.datetime.strptime(x, "%Y-%m-%d").day)
The Applymap function is the all data version of apply where the transformation logic is applied on every data point in the data (e.g. every cell in data view).
Say we want to change all the passenger names into lower case. For demonstration purposes, let us create a separate dataframe that is a subset of the original with just the “Name” variable in it.
df_name = df.copy()[['Name']]df_name.head()
Now, we use the Applymap function to accomplish what we want.
df_name = df_name.applymap(lambda x: x.lower() if type(x) == str else x)
Note that the if-else statement can be written as above within the lambda function. You can see below that all the names are now lower case!
Say we want to replace categories (in string format) into corresponding integers. Can we use the Applymap function to do this? Although the Apply function may be more relevant for this, we can still use the Applymap function to accomplish the same results.
We have a dictionary that maps the genders, male and female to 0 and 1 respectively.
mapping = {"male":0, "female":1}df.applymap(mapping.get)
As you can see from the output above, Applymap function, as stated above, applies the transformation logic to every data point in every variable. Hence, we see that all other cells that are irrelevant from the “Sex” variable were replaced with None. We do not want that. In order to achieve what we want, we can craft the lambda function to replace the values only when the value in the cell is one of the mapping keys, which, in this case, are strings ‘male’ and ‘female’.
df.applymap(lambda x: mapping[x] if x in mapping.keys() else x)
Now we see that only the “Sex” variable has been transformed while the other variables are intact.
Aggregation
Lastly, but not least, the Aggregation function, unlike the Apply and Applymap functions, returns a new dataframe that includes the aggregated summary statistics that the user specifies. Aggregated summary statistics refer to statistics including maximum value, minimum value, mean, median and mode. Here, we calculate the average age, maximum age and the survival rate of the passengers.
df.groupby("Pclass").agg(avg_age = ("Age", "mean"),
max_age = ("Age", "max"),
survival_rate = ("Survived", "mean"))
As you can see from the snippet above, using the aggregation function in conjunction with the Groupby function becomes a powerful tool for calculating aggregations for different groups of data points.
In this article, I used the Titanic Disaster dataset to illustrate what the three most commonly used transformation / broadcasting functions do and how they differ from one another. Stay tuned for more of my articles on data cleaning, machine learning, deep learning, natural language processing and more.
If you found this post helpful, consider supporting me by signing up on medium via the following link : )
You will have access to so many useful and interesting articles and posts from not only me but also other authors!
Data Scientist. 1st Year PhD student in Informatics at UC Irvine.
Former research area specialist at the Criminal Justice Administrative Records System (CJARS) economics lab at the University of Michigan, working on statistical report generation, automated data quality review, building data pipelines and data standardization & harmonization. Former Data Science Intern at Spotify. Inc. (NYC).
He loves sports, working-out, cooking good Asian food, watching kdramas and making / performing music and most importantly worshiping Jesus Christ, our Lord. Checkout his website!
