Apache NiFi Cluster Setup

Setup a basic NiFi Cluster with three nodes.

Step 1 Download NiFi package

sudo wget http://apache.mirror.vexxhost.com/nifi/0.7.0/nifi-0.7.0-bin.tar.gz

sudo tar -xvf nifi-0.7.0-bin.tar.gz

Step 2 Configure NiFi Cluster

sudo vi /opt/nifi-0.7.0/conf/nifi.properties

Node 1. NCM(NiFi Cluster Manager)

# web properties #
nifi.web.http.host=172.31.48.1
nifi.web.http.port=8080

# cluster manager properties (only configure for cluster manager) #
nifi.cluster.is.manager=true
nifi.cluster.manager.address=172.31.48.1
nifi.cluster.manager.protocol.port=8090

Node 2. Primary Node(the first slave node)

Node 3. Slave Node 2

# web properties #

nifi.web.http.host=172.31.48.2

nifi.web.http.port=8080

# cluster node properties (only configure for cluster nodes) #

nifi.cluster.is.node=true

nifi.cluster.node.address=172.31.48.2

nifi.cluster.node.protocol.port=8070

# if multicast is not used, nifi.cluster.node.unicast.xxx must have same values as nifi.cluster.manager.xxx #

nifi.cluster.node.unicast.manager.address=172.31.48.1

nifi.cluster.node.unicast.manager.protocol.port=8090

Step 3. Configure State Managemetn with Zookeeper

An embedded ZooKeeper server cannot be run on the NCM. So only configure zookeeper in worker nodes.

In conf/nifi.properties,

nifi.state.management.embedded.zookeeper.start=false

In conf/zookeeper.properties

Add the broker list:

server.1=172.31.48.1:2888:3888

server.2=172.31.48.2:2888:3888

Creating a file named myid and placing it in ZooKeeper's data directory. The contents of this file should be the index of the server as specific by the server.<number>.

mkdir state/zookeeper
echo 1 > state/zookeeper/myid


State is stored only in ZooKeeper (whichever instance is configured in your conf/state-mangement.xml file). If there is no ZooKeeper instance that is properly configured there, it will fail to store any state and you'll end up re-listing those files when NiFi restarts.

ERROR [Timer-Driven Process Thread-10] o.a.nifi.processors.standard.FetchSFTP FetchSFTP[id=5cdfac90-2d07-443e-97b6-b06a1a883a22] FetchSFTP[id=5cdfac90-2d07-443e-97b6-b06a1a883a22] failed to process due to org.apache.nifi.processor.exception.ProcessException: IOException thrown from FetchSFTP[id=5cdfac90-2d07-443e-97b6-b06a1a883a22]: java.io.IOException: error; rolling back session: org.apache.nifi.processor.exception.ProcessException: IOException thrown from FetchSFTP[id=5cdfac90-2d07-443e-97b6-b06a1a883a22]: java.io.IOException: error 2016-08-09 18:59:48,093 ERROR [Timer-Driven Process Thread-10] o.a.nifi.processors.standard.FetchSFTP org.apache.nifi.processor.exception.ProcessException: IOException thrown from FetchSFTP[id=5cdfac90-2d07-443e-97b6-b06a1a883a22]: java.io.IOException: error

In conf/state-management.xml

Add zookeeper broker list for NiFi to fetch state in zookeeper
"Connect String": 172.31.48.1:2181,172.31.48.2:2181

Step 4. Start NiFi Service

sudo ./nifi-0.7.0/bin/nifi.sh start

Check Service Running:

ps -ef | grep nifi

Note:
If nifi.web.http.host is not explicitly set, this will default to localhost which then be interpreted by the NCM incorrectly when transmitted with the heartbeat.

An error will show up:
An unexpected error has occurred
No nodes were able to process this request.

Reference:
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#clustering
https://community.hortonworks.com/questions/11891/how-to-troubleshoot-nifi-non-secure-cluster.html