Breaking News: Grepper is joining You.com. Read the official announcement!

pyspark long and wide dataframe

Innocent Iguana answered on March 4, 2024 Popularity 1/10 Helpfulness 8/10

answer pyspark long and wide dataframe

pyspark long and wide dataframe

Comment

# Typically can be done by explode() and pivot() method
# Say, you have a pyspark dataframe wide_df:
# +---+------+-------+-------+
# | id|value1| value2| value3|
# +---+------+-------+-------+
# |  C|     3|     30|    300|
# |  B|     2|     20|    200|
# |  A|     1|     10|    100|
# +---+------+-------+-------+
# Melt from wide DataFrame to long format
long_df = wide_df.select("id", explode(array([
    ("value1", col("value1")),
    ("value2", col("value2")),
    ("value3", col("value3"))
])).alias("variable", "value"))

# The long_df will now look like:
# +---+--------+-----+
# | id|variable|value|
# +---+--------+-----+
# |  A|  value1|    1|
# |  A|  value2|   10|
# |  A|  value3|  100|
# |  B|  value1|    2|
# |  B|  value2|   20|
# |  B|  value3|  200|
# |  C|  value1|    3|
# |  C|  value2|   30|
# |  C|  value3|  300|
# +---+--------+-----+

# Pivot from long DataFrame to wide format
wide_df = long_df.groupBy("id").pivot("variable").agg({"value": "max"})

Popularity 1/10 Helpfulness 8/10 Language python

Source: Grepper

Tags: dataframe pyspark python

Link to this answer
Share Copy Link

Contributed on Mar 04 2024

Innocent Iguana

0 Answers Avg Quality 2/10

pyspark long and wide dataframe

Contents

More Related Answers

pyspark long and wide dataframe

Grepper

Documentation

Social

Legal

Contact

Oops, You will need to install Grepper and log-in to perform this action.