Breaking News: Grepper is joining You.com. Read the official announcement!

spark find duplicate rows

Cristian Agudelo answered on December 23, 2022 Popularity 6/10 Helpfulness 1/10

answer spark find duplicate rows

spark find duplicate rows

Comment

scala> import spark.implicits._
import spark.implicits._

scala> val data1 = Seq(10, 20, 20, 30, 30, 30, 40)
data1: Seq[Int] = List(10, 20, 20, 30, 30, 30, 40)

scala> val df1 = data1.toDF()
df1: org.apache.spark.sql.DataFrame = [value: int]

scala> val dupes = df1.groupBy("value").count.filter("count > 1")
dupes: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [value: int, count: bigint]

scala> dupes.show
+-----+-----+
|value|count|
+-----+-----+
|   20|    2|
|   30|    3|
+-----+-----+

Popularity 6/10 Helpfulness 1/10 Language whatever

Source: www.hadoopinrealworld.com

Tags: find rows whatever

Link to this answer
Share Copy Link