Here we are going to discuss about reading and writing different file formats and sources like parquet,json,carbon, mysql(RDBMS),S3 etc.

Q-1 How to read the parquet file from hdfs and after some transformations, write again into hdfs only as a parquet file?
Ans:

#Read and write Parquet file from hdfs

df=spark.read.parquet(“parquet file path”)

#Perform transformation on df

df.write.parquet(“data.parquet”) #hdfs path

Q-2 How to read the json file from hdfs and after some transformations, write again into hdfs only as a json file?
Ans:

#Read and write json file from hdfs

df=spark.read.json(“json file path”)

#Perform transformation on df

df.write.json(“data.json”) #hdfs path

Q-3 How to read the data from mysql or other rdbms database and write the transformed data into hdfs as a parquet file ?
Ans:


url=”jdbc:mysql://localhost:3306/test”
tbl_name=”tbl_test”
properties=
{
“driver”:”com.mysql.jdbc.Driver”
“username”:”root”
“password”:”xxx”
}

df=spark.read.jdbc(url=url, table=tbl_name,properties=properties)

#Perform the transformations on df

df.write.parquet(“data.parquet”) #hdfs path


8 Comments

Radmila · June 5, 2020 at 3:41 pm

Thank you for this post. Its very inspiring.

Sildena · June 5, 2020 at 3:43 pm

Thanks a lot for the article post.Much thanks again. Fantastic.

Rya · June 5, 2020 at 4:01 pm

Thanks so much for the post.Much thanks again. Really Cool.

Chandra · June 9, 2020 at 7:39 am

Thanks for the article post.Really thank you! Great.

Barakh · June 12, 2020 at 9:54 am

Thanks so much for the post.Really thank you! Keep writing.

Himani · June 15, 2020 at 2:26 pm

Thank you for your blog post.Really thank you! Awesome.

Genelia · June 26, 2020 at 6:48 am

Thanks so much for the post.Really thank you! Keep writing.

Personal Tax Accountants Near Me · July 3, 2020 at 11:43 pm

I truly love your blog.. Excellent colors & theme. Did you make this website yourself? Please reply back as I’m trying to create my very own blog and would love to learn where you got this from or just what the theme is named. Appreciate it!

Leave a Reply

Your email address will not be published. Required fields are marked *

Insert math as
Block
Inline
Additional settings
Formula color
Text color
#333333
Type math using LaTeX
Preview
\({}\)
Nothing to preview
Insert