Breaking News: Grepper is joining You.com. Read the official announcement!

PySpark find columns with null values

Scarlet Macaw answered on October 9, 2021 Popularity 8/10 Helpfulness 8/10

answer PySpark find columns with null values

related pyspark check all columns for null values

PySpark find columns with null values

Comment

# Find Count of Null, None, NaN of All DataFrame Columns
from pyspark.sql import functions as F
df.select([F.count(F.when(F.isnan(c) | F.col(c).isNull(), c)).alias(c) \
           for c in df.columns]).show()

Popularity 8/10 Helpfulness 8/10 Language python

Source: sparkbyexamples.com

Tags: find get null pyspark python

Link to this answer
Share Copy Link

Contributed on Jul 15 2022

Scarlet Macaw

0 Answers Avg Quality 2/10

Closely Related Answers

pyspark check all columns for null values

Comment

from pyspark.sql.functions import isnan, when, count, col

df.select([count(when(isnan(c), c)).alias(c) for c in df.columns]).show()
+-------+----------+---+
|session|timestamp1|id2|
+-------+----------+---+
|      0|         0|  3|
+-------+----------+---+

Popularity 8/10 Helpfulness 6/10 Language python

Source: stackoverflow.com

Tags: null pyspark python

Link to this answer
Share Copy Link