In this blog we will see how to make a local spark with a master and three workers followed by launching a spark-shell on it.
Creating the Master Class:
I have already installed the spark (spark-3.1.1) and now we will start the master class using the following command(Should be executed inside the bin folder):
spark-class org.apache.spark.deploy.master.Master
Note: Do not Close These Terminal. Closing it will end the master.
You can see the master web UI by copying and pasting the URL in a web-browser.
Master Web UI:
Creating the Workers:
Copy paste the following command in different terminals to create multiple workers
spark-class org.apache.spark.deploy.worker.Worker (Master URL)
After executing this command, you will be able to see ‘successfully registered’ text in the end
After creating the workers, you will be able to see this in your command prompt where the master class is running.
Refresh the webUI page to view about the workers. Since the workers do not communicate with each other, each worker thinks they are running on a seperate machine but in actual they are running on the same machine. So each will show the machine’s full resource.
Launching the Spark-shell:
To launch the spark-shell, use the following command.
spark-shell --master (Master URL)
After executing this command, you will be able to use the spark-shell that is connected to the master.
We can view the Job UI using the URL.
After executing it and by refreshing the master web UI, we can see there will be a new application running in the name of spark shell. By clicking it we can see more information about the application.
Hope this blog is helpful and feel free to send feedbacks and also to comment on blogs for further improvements in the future. You can also comment you queries which will be helped out ASAP.
RarelyKnows 