Hey there!! Have you heard about Data Lake or used any in your projects? Might be. Let me help you recall it. Data Lake is a repository that helps to store structured and unstructured data, like HDFS, Azure Data Lake, or AWS S3. But are you known to DELTA LAKE? Read more…
Hey folks 🙂 You guys must to eager to know how can you update your existing Big Data files using Spark. So, we’ll try some hands-on on Spark Dataframe. After rigorously working on spark dataframes and visiting my previous articles, you must be familiar with the fact that: IT’S DIFFICULT Read more…
Have you ever wondered if we could apply joins on Spark Dataframes as we do on SQL tables? Would it be possible? Woohoo!! You guessed it right. Here we have with us, a spark module called SPARK SQL for structured data processing. Spark SQL supports all kinds of SQL joins. Read more…
Spark contains loads of aggregate functions to extract out the statistical information leveraging group by, cube and rolling DataFrames. Today, we’ll be checking out some aggregate functions to ease down the operations on Spark DataFrames. Before moving ahead, let’s create a dataframe to work with. I have created an Item Read more…
What if you need to find the name of the employee with the highest salary. Or get the names of the total employees in each department from the employee table. How would you apply operations on dataframes to get these results? Now, here comes “Spark Aggregate Functions” into the picture. Read more…
Have you ever thought of using SQL statements in Spark Dataframe? Is it possible to provide conditions in Spark to get the desired outputs in the dataframe? We have studied the case and switch statements in any programming language we practiced. And yes, here too Spark leverages to provides us Read more…
Spark filter() function is used to filter rows from the dataframe based on given condition or expression. If you are familiar with SQL, then it would be much simpler for you to filter out rows according to your requirements. For example, a list of students who got marks more than Read more…
In this article, I will walk you through commonly used dataframe column operations. Spark withcolumn() is used to rename, drop, change the value of an existing column and to create a new column too. Let’s create a dataframe first. Suppose you want to calculate the Percentage of the Student using Read more…
Hey there!! You might be knowing that data type conversion is an important step while doing the transformation of the dataframe. Let’s say we would like to add a number to the dataframe column and the column data type is String. Can we perform addition now ? Answer is No. Read more…