What’s New ?

The Top 10 favtutor Features You Might Have Overlooked

Read More

PySpark GroupBy function Explained (With Examples)

  • Sep 23, 2023
  • 7 Minutes Read
  • Why Trust Us
    We uphold a strict editorial policy that emphasizes factual accuracy, relevance, and impartiality. Our content is crafted by top technical writers with deep knowledge in the fields of computer science and data science, ensuring each piece is meticulously reviewed by a team of seasoned editors to guarantee compliance with the highest standards in educational content creation and publishing.
  • By Abhisek Ganguly
PySpark GroupBy function Explained (With Examples)

In the realm of big data processing, PySpark has emerged as a powerful tool, allowing data scientists and engineers to perform complex data manipulations and analyses efficiently. PySpark offers a versatile and high-performance solution for this task with its groupBy operation. In this article, we will dive deep into the world of PySpark groupBy, exploring its capabilities, use cases, and best practices.

Introduction to PySpark groupBy

PySpark is an open-source Python library that provides an interface for Apache Spark, a powerful distributed data processing framework. Spark allows users to process large-scale datasets in parallel across a cluster of computers, making it a popular choice for big data analytics. 

The groupBy operation in PySpark allows you to group data based on one or more columns in a DataFrame. Once grouped, you can perform various aggregation operations, such as summing, counting, averaging, or applying custom aggregation functions, on the grouped data.

Let's start by exploring the basic syntax of the groupBy operation in PySpark:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("PySparkGroupBy").getOrCreate()

data = [("Alice", "Sales", 5000),
        ("Bob", "Engineering", 6000),
        ("Charlie", "Sales", 4500),
        ("David", "Engineering", 7000),
        ("Eva", "HR", 5500)]
columns = ["Name", "Department", "Salary"]
df = spark.createDataFrame(data, columns)

grouped_by_department = df.groupBy("Department")

grouped_by_dept_salary = df.groupBy("Department", "Salary")


In the code above, we first create a SparkSession and load a sample DataFrame. Then, we demonstrate how to use groupBy to group data by a single column ("Department") and by multiple columns ("Department" and "Salary").



Department Salary
Sales 5000
Engineering 6000
Sales 4500
Engineering 7000
HR 5500


Now, let's explore some key aspects of PySpark groupBy.

Aggregating Data with groupBy

Once you've grouped your data, you often want to compute aggregates on the grouped data. PySpark provides a wide range of aggregation functions that you can use with groupBy.

Here are some common aggregation functions:

1. agg

The agg function allows you to specify one or more aggregation functions to apply to each group. For example, to calculate the total salary expenditure for each department:

from pyspark.sql import functions as F

total_salary_by_dept = df.groupBy("Department").agg(F.sum("Salary").alias("TotalSalary")) 


2. count

The count function counts the number of rows in each group. For instance, to count the number of employees in each department:

employee_count_by_dept = df.groupBy("Department").count()


3. sum, avg, min, and max

These functions calculate the sum, average, minimum, and maximum values of a numeric column within each group, respectively. For instance, to find the average salary in each department:

avg_salary_by_dept = df.groupBy("Department").agg(F.avg("Salary").alias("AvgSalary"))


4. Custom Aggregation Functions

You can also define custom aggregation functions using agg. This is useful when you need to perform complex calculations. For example, to find the standard deviation of salaries within each department:

from pyspark.sql.functions import stddev stddev_salary_by_dept = df.groupBy("Department").agg(stddev("Salary").alias("StdDevSalary"))

Applying Multiple Aggregations

In many cases, you may want to apply multiple aggregation functions in a single groupBy operation. PySpark makes this straightforward:

from pyspark.sql.functions import sum, avg, max, min

result = df.groupBy("Department").agg(


In this example, we calculate the total, average, maximum, and minimum salary for each department in a single groupBy operation.

Department TotalSalary AvgSalary MaxSalary MinSalary
Sales 9500 4750.0 5000 4500
Engineering 13000 6500.0 7000 6000
HR 5500 5500.0 5500 5500

Applying Filters and Conditions

You can also apply filters and conditions to the grouped data using the filter or where method. This allows you to further refine the groups based on specific criteria. For example, to find the average salary of employees in the Sales department who earn more than $4,000:

from pyspark.sql.functions import avg

result = df.groupBy("Department").agg(
).filter("Department = 'Sales' AND Salary > 4000")


Department AvgSalary
Sales 5000.0

GroupBy with Window Functions

Window functions are powerful tools for performing calculations across rows in a group of rows. You can use window functions in combination with groupBy to achieve complex analytical tasks. Here's a simplified example of calculating the rank of employees within their department based on salary:

from pyspark.sql.window import Window
from pyspark.sql.functions import rank

window_spec = Window.partitionBy("Department").orderBy(df["Salary"].desc())

ranked_employees = df.withColumn("Rank", rank().over(window_spec))


In this example, we create a window specification that partitions the data by department and orders it by salary in descending order. Then, we use the rank window function to assign a rank to each employee within their department.

Name Department Salary Rank
David Engineering 7000 1
Bob Engineering 6000 2
Alice Sales 5000 1
Charlie Sales 4500 2
Eva HR 5500 1

Performance Optimization

When working with large datasets, performance is a critical consideration. PySpark offers several ways to optimize groupBy operations:

1. Caching and Persisting

Cache or persist intermediate DataFrames that you reuse in multiple transformations or actions. This reduces the need to recompute the same data multiple times.

df.cache()  # Caches the DataFrame in memory
df.persist()  # Persists the DataFrame, allowing more control over storage options


2. Using Spark's Catalyst Optimizer

PySpark leverages Spark's Catalyst optimizer to optimize query execution. By default, Spark will optimize your DataFrame operations to improve performance.

3. Proper Partitioning

Ensure that your data is properly partitioned. When you perform a groupBy, Spark can leverage data locality for faster processing.

4. Avoiding Shuffles

Minimize data shuffling operations, as they can be expensive. Shuffles occur when data needs to be reorganized across partitions during operations like groupBy. Avoid unnecessary shuffles by using appropriate transformations.


The groupBy operation in PySpark is a powerful tool for data manipulation and aggregation. It allows you to group data based on one or more columns and perform various aggregations and calculations on the grouped data. With PySpark's groupBy, you can confidently tackle complex data analysis challenges and derive valuable insights from your data.

In this article, we've covered the fundamental concepts and usage of groupBy in PySpark, including syntax, aggregation functions, multiple aggregations, filtering, window functions, and performance optimization.

FavTutor - 24x7 Live Coding Help from Expert Tutors!

About The Author
Abhisek Ganguly
Passionate machine learning enthusiast with a deep love for computer science, dedicated to pushing the boundaries of AI through academic research and sharing knowledge through teaching.