Issues to submit Spark job in YARN

1.  The job is accepted, but hanging until to fail.

15/03/25 20:28:53 INFO yarn.Client: Application report for application_1427016337843_0030 (state: ACCEPTED)

15/03/25 20:28:54 INFO yarn.Client: Application report for application_1427016337843_0030 (state: FAILED)

15/03/25 20:28:54 INFO yarn.Client:

client token: N/A

diagnostics: Application application_1427016337843_0030 failed 2 times due to AM Container for appattempt_1427016337843_0030_000002 exited with exitCode: 15

Diagnostics: Exception from container-launch.

Container id: container_1427016337843_0030_02_000001

Exit code: 15

Stack trace: ExitCodeException exitCode=15:

at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)

at org.apache.hadoop.util.Shell.run(Shell.java:455)

Solution:

Should remove ".setMaster("local")" in the code, and add "--master yarn-cluster" as parameter in the command line.

2.  Incompatible scala version

5/03/26 02:37:49 INFO yarn.Client: Application report for application_1427016337843_0047 (state: RUNNING)

15/03/26 02:37:50 INFO yarn.Client: Application report for application_1427016337843_0047 (state: FINISHED)

15/03/26 02:37:50 INFO yarn.Client:

client token: N/A

diagnostics: User class threw exception: scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;

ApplicationMaster host: ip-192-168-8-110.ap-southeast-1.compute.internal

ApplicationMaster RPC port: 0

queue: default

start time: 1427337486613

final status: FAILED

Solution:

The official released Spark1.3 is complited by scala 2.10. So the spark job should be compiled by 2.10 rather than 2.11+. Or you can recompile Spark by 2.11.

http://spark.apache.org/docs/latest/building-spark.html#building-for-scala-211

http://stackoverflow.com/questions/28487305/nosuchmethoderror-while-running-spark-streaming-job-on-hdp-2-2