Breaking News: Grepper is joining You.com. Read the official announcement!

How to Drop a DataFrame/Dataset column in pyspark

Add Answer

Soumyaranjan Rout answered on September 22, 2021 Popularity 9/10 Helpfulness 3/10

answer How to Drop a DataFrame/Dataset column in pyspark

related drop columns pyspark

related Pyspark Drop columns

related how to drop column in spark dataframe

How to Drop a DataFrame/Dataset column in pyspark

Comment

To drop a column in a PySpark DataFrame, you can use the drop method.
This method takes two arguments:

col: The name of the column you want to drop.
axis: The axis along which you want to drop the column. 
In this case, you should set axis=1 to indicate that you want to drop a column.

Here's an example of how you can use the drop method to drop a column in a 
PySpark DataFrame:


from pyspark.sql import SparkSession

# Create a SparkSession
spark = SparkSession.builder.appName("Drop Column").getOrCreate()

# Load a DataFrame
df = spark.read.csv("path/to/data.csv", header=True)

# Drop a column
df = df.drop("col_name", axis=1)
This will drop the column with the name "col_name" from the DataFrame. 
If you want to drop multiple columns, you can pass a list of column names 
to the col argument.

df = df.drop(["col_name_1", "col_name_2"], axis=1)

Keep in mind that the drop method returns a new DataFrame with the specified
column(s) removed. It does not modify the original DataFrame.

Popularity 9/10 Helpfulness 3/10 Language python

Source: Grepper

Tags: drop pyspark python

Link to this answer
Share Copy Link

Contributed on Dec 19 2022

Soumyaranjan Rout

0 Answers Avg Quality 2/10

Closely Related Answers

drop columns pyspark

Comment

df.drop("firstname") \
  .printSchema()
""" import col is required """  
df.drop(col("firstname")) \
  .printSchema()  
  
df.drop(df.firstname) \
  .printSchema()

Popularity 10/10 Helpfulness 6/10 Language python

Source: sparkbyexamples.com

Tags: drop

Link to this answer
Share Copy Link

Contributed on Sep 22 2021

Super Shrew

0 Answers Avg Quality 2/10

Pyspark Drop columns

Comment

list = ['Employee ID','Employee NAME','Company Name']
  
# delete two columns
dataframe = dataframe.drop(*list)
dataframe.show()

Popularity 9/10 Helpfulness 4/10 Language whatever

Source: www.geeksforgeeks.org

Tags: drop

Link to this answer
Share Copy Link

Contributed on Jul 27 2022

Auckland Lauwo

0 Answers Avg Quality 2/10

how to drop column in spark dataframe

Comment

# Use drop() method
foo3 = foo2.drop("delay") 
foo3.show()

Popularity 9/10 Helpfulness 3/10 Language python

Source: Grepper

Tags: drop python

Link to this answer
Share Copy Link

Contributed on Sep 22 2022

MazelTov27

0 Answers Avg Quality 2/10

How to Drop a DataFrame/Dataset column in pyspark

Contents

More Related Answers

How to Drop a DataFrame/Dataset column in pyspark

Closely Related Answers

drop columns pyspark

Pyspark Drop columns

how to drop column in spark dataframe

Grepper

Documentation

Social

Legal

Contact

Oops, You will need to install Grepper and log-in to perform this action.