Deployment Architecture in Spark

Deploy Graph

1. One Master node is in charge of multiple Worker nodes.

2. Worker nodes communicate with Master, and manage executors.

Each worker has multiple ExecutorBackend processes.

Each ExecutorBackend process has a Executor object, which has a task thread pool.

3. Driver is the process running the main() function of the application and create the SparkContext.

4. Each application has a driver and multiple executors.

All tasks in an executor belong to the same application.

Logic plan

Reference:

The contents of this artical are from https://github.com/JerryLead/SparkInternals