pyspark dataframe from pandas

Caio Henrique answered on November 30, 2022 Popularity 9/10 Helpfulness 4/10

answer pyspark dataframe from pandas

pyspark dataframe from pandas

Comment

from pyspark.sql.types import StructType,StructField, StringType, IntegerType
#Create User defined Custom Schema using StructType
mySchema = StructType([ StructField("First Name", StringType(), True)\
                       ,StructField("Age", IntegerType(), True)])

#Create DataFrame by changing schema
sparkDF2 = spark.createDataFrame(pandasDF,schema=mySchema)
sparkDF2.printSchema()
sparkDF2.show()

#Outputs below schema & DataFrame

root
 |-- First Name: string (nullable = true)
 |-- Age: integer (nullable = true)

+----------+---+
|First Name|Age|
+----------+---+
|     Scott| 50|
|      Jeff| 45|
|    Thomas| 54|
|       Ann| 34|
+----------+---+

Popularity 9/10 Helpfulness 4/10 Language python

Source: sparkbyexamples.com

Tags: pandas python

Link to this answer
Share Copy Link

Contributed on Nov 30 2022

Caio Henrique

0 Answers Avg Quality 2/10

pyspark dataframe from pandas

Contents

More Related Answers

pyspark dataframe from pandas

Grepper

Documentation

Social

Legal

Contact

Oops, You will need to install Grepper and log-in to perform this action.