Friday, 21 August 2015

Azkaban Workflow

Inherited Parameters

Any included .properties files will be treated as properties that are shared amongst the individual jobs of the flow. The properties are resolved in a hierarchical manner by directory.
system.properties
baz.job
myflow/
   myflow.properties
   foo.job
That directory structure will be preserved when running in Azkaban. The baz job will inherit only from system.properties. The jobsfoo will inherit from myflow.properties, which in turn will inherit from system.properties.
Parameter Substitution
Azkaban allows for replacing of parameters. Whenever a ${parameter} is found in a properties or job file, Azkaban will attempt to replace that parameter. 
# shared.properties
replaceparameter=bar
# myjob.job
param1=mytest
foo=${replaceparameter}

param2=${param1}
In the previous example, before myjob is run, foo will equal bar and param2 will equal mytest.


Embedded flow

A flow can also be included as a node in other flow as an embedded flow. To create an embedded flow, simply create a .job file withtype=flow and flow.name set to the name of the end action of the embedded flow.

# subflow.job
type=flow
flow.name=endSubflow
dependencies=startMainflow



Reference:
http://azkaban.github.io/azkaban/docs/latest/

1 comment:

  1. How I can fetch the logs of the inner nodes in case the project has embedded flows?

    ReplyDelete