Spark WARN: Initial job has not accepted any resources

$spark-shell --driver-memory 2g --executor-memory 6g

WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster ui
to ensure that workers are registered and have sufficient memory

This message will pop up any time an application is requesting more resources from the cluster than the cluster can currently provide. What resources you might ask? Well Spark is only looking for two things: Cores and Ram. Cores represents the number of open executor slots that your cluster provides for execution. Ram refers to the amount of free Ram required on any worker running your application. Note for both of these resources the maximum value is not your System’s max, it is the max as set by the your Spark configuration.

In CM, choose Spark Configuration.

1. Master Default Group

Java Heap Size of Master in Bytes  master_max_heapsize = 2G Maximum size for the Java process heap memory. Passed to Java -Xmx. Measured in bytes.

2. Worker Default Group

Java Heap Size of Worker in Bytes  worker_max_heapsize = 2G Maximum size for the Java process heap memory. Passed to Java -Xmx. Measured in bytes. Total Java Heap Sizes of Worker's Executors in Bytes  executor_total_max_heapsize=16G

Memory available to the Worker's Executors. This is the maximum sum total of all the Executors' Java heap sizes on this Worker node. Passed to Java -Xmx. Measured in bytes.

Reference:
Spark web UI: http://www.datastax.com/dev/blog/common-spark-troubleshooting