Monitoring in Spark using spark.sql

It is common that we need to see how our spark application runs and we need to collect spark metrics. Instead of seeing through web UI we can use see it in more structured way using spark Monitoring.In this metrics we will see regarding

Step1:Starting Spark History server

STEP2:Configuring spark History Server

On seeing the log file I found out that spark requires /tmp/spark-events

Then I created the directory in hdfs and then restarted the history server

Now I found spark History server listening under localhost:18080 port

Step3:Viewing Spark logs with Spark

The event logs of running and finished application is available under /tmp/spark-logs in json format.So I will be reading the directory using spark.read.json as shown below,Before that I am ensuring spark.eventLog is enable and logdir is set

Various spark Metrics of Running application can be found by reading the directory as shown Below.Some of the notable metrics include

Since I have not ran any task most of my metrics is null in above screenshot

Uses:

We can have this for monitoring spark apps and we can have a detailed log analysis of our Jobs in a structured way

2.We can use this metrics to read the application metrics and user metrics of cluster utilisation

That’s all for the day !! :)

Please post me with topics in spark which I have to cover and provide me with suggestion for improving my writing :)

Learn and let others Learn!!

Related posts:

When it is cleaned and properly groomed, it’s nearly like that sense of order is imparted to the dog. Smoothes softens and conditions unruly beard hair whereas hydrating the skin beneath the beard…

So tonight I was watching the Post starring Meryl Streep and Tom Hanks which discusses the publication of the Pentagon papers. Besides Meryl’s constant laughing I drew comparison between the…

The Game Shares ICO starts on May 20th, 2018 at 12:00 PM UTC. Copy ethereum address from pb.game.com/ico website. Please use an ERC-20 compatible Ethereum wallet when you are contributing to the Game…