Here we are going to discuss about reading and writing different file formats and sources like parquet,json,carbon, mysql(RDBMS),S3 etc.

Q-1 How to read the parquet file from hdfs and after some transformations, write again into hdfs only as a parquet file?
Ans:

#Read and write Parquet file from hdfs

df=spark.read.parquet(“parquet file path”)

#Perform transformation on df

df.write.parquet(“data.parquet”) #hdfs path

Q-2 How to read the json file from hdfs and after some transformations, write again into hdfs only as a json file?
Ans:

#Read and write json file from hdfs

df=spark.read.json(“json file path”)

#Perform transformation on df

df.write.json(“data.json”) #hdfs path

Q-3 How to read the data from mysql or other rdbms database and write the transformed data into hdfs as a parquet file ?
Ans:


url=”jdbc:mysql://localhost:3306/test”
tbl_name=”tbl_test”
properties=
{
“driver”:”com.mysql.jdbc.Driver”
“username”:”root”
“password”:”xxx”
}

df=spark.read.jdbc(url=url, table=tbl_name,properties=properties)

#Perform the transformations on df

df.write.parquet(“data.parquet”) #hdfs path


8 Comments

Radmila · June 5, 2020 at 3:41 pm

Thank you for this post. Its very inspiring.

Sildena · June 5, 2020 at 3:43 pm

Thanks a lot for the article post.Much thanks again. Fantastic.

Rya · June 5, 2020 at 4:01 pm

Thanks so much for the post.Much thanks again. Really Cool.

Chandra · June 9, 2020 at 7:39 am

Thanks for the article post.Really thank you! Great.

Barakh · June 12, 2020 at 9:54 am

Thanks so much for the post.Really thank you! Keep writing.

Himani · June 15, 2020 at 2:26 pm

Thank you for your blog post.Really thank you! Awesome.

Genelia · June 26, 2020 at 6:48 am

Thanks so much for the post.Really thank you! Keep writing.

Personal Tax Accountants Near Me · July 3, 2020 at 11:43 pm

I truly love your blog.. Excellent colors & theme. Did you make this website yourself? Please reply back as Iā€™m trying to create my very own blog and would love to learn where you got this from or just what the theme is named. Appreciate it!

Leave a Reply

Your email address will not be published. Required fields are marked *

Insert math as
Block
Inline
Additional settings
Formula color
Text color
#333333
Type math using LaTeX
Preview
\({}\)
Nothing to preview
Insert