Tutorial-2 Pyspark DataFrame FileFormats

Here we are going to discuss about reading and writing different file formats and sources like parquet,json,carbon, mysql(RDBMS),S3 etc.

Q-1 How to read the parquet file from hdfs and after some transformations, write again into hdfs only as a parquet file?
Ans:

#Read and write Parquet file from hdfs

df=spark.read.parquet(“parquet file path”)

#Perform transformation on df

df.write.parquet(“data.parquet”) #hdfs path

Q-2 How to read the json file from hdfs and after some transformations, write again into hdfs only as a json file?
Ans:

#Read and write json file from hdfs

df=spark.read.json(“json file path”)

#Perform transformation on df

df.write.json(“data.json”) #hdfs path

Q-3 How to read the data from mysql or other rdbms database and write the transformed data into hdfs as a parquet file ?
Ans:

url=”jdbc:mysql://localhost:3306/test”
tbl_name=”tbl_test”
properties=
{
“driver”:”com.mysql.jdbc.Driver”
“username”:”root”
“password”:”xxx”
}

df=spark.read.jdbc(url=url, table=tbl_name,properties=properties)

#Perform the transformations on df

df.write.parquet(“data.parquet”) #hdfs path