Sunday, 28 February 2016

Set Log4j Conf in Spark Job

Normally, spark job gives us a lot of logs in console, which hide our prints.
Put below under resources folder of your project.

log4j.rootCategory=WARN, console
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n$exprTyper=WARN$SparkILoopInterpreter=WARN

To use a custom log4j configuration for the application master or executors, here are the options:

add -Dlog4j.configuration=<location of configuration file> to spark.driver.extraJavaOptions (for the driver) or spark.executor.extraJavaOptions (for executors). Note that if using a file, the file: protocol should be explicitly provided, and the file needs to exist locally on all the nodes.
Note this step should be in front of  --class

$SPARK_HOME/bin/spark-submit --jars $JARS \
--files ${HOME}/conf/ \
--driver-java-options "-Dlog4j.configuration=file://${HOME} /conf/"  \
--class $APP_CLASS $APP_JAR \
--master $SPARK_MASTER \


No comments:

Post a Comment