Step 1 Download NiFi package
sudo wget http://apache.mirror.vexxhost.com/nifi/0.7.0/nifi-0.7.0-bin.tar.gz
sudo tar -xvf nifi-0.7.0-bin.tar.gz
Step 2 Configure NiFi Cluster
sudo vi /opt/nifi-0.7.0/conf/nifi.properties
Node 1. NCM(NiFi Cluster Manager)
# web properties #
nifi.web.http.host=172.31.48.1
nifi.web.http.port=8080
nifi.cluster.is.manager=true
nifi.cluster.manager.address=172.31.48.1
nifi.cluster.manager.protocol.port=8090
Node 2. Primary Node(the first slave node)
Node 3. Slave Node 2
# web properties #
nifi.web.http.host=172.31.48.2
nifi.web.http.port=8080
# cluster node properties (only configure for cluster nodes) #
nifi.cluster.is.node=true
nifi.cluster.node.address=172.31.48.2
nifi.cluster.node.protocol.port=8070
# if multicast is not used, nifi.cluster.node.unicast.xxx must have same values as nifi.cluster.manager.xxx #
nifi.cluster.node.unicast.manager.address=172.31.48.1
nifi.cluster.node.unicast.manager.protocol.port=8090
Step 3. Configure State Managemetn with Zookeeper
In conf/nifi.properties,
nifi.state.management.embedded.zookeeper.start=false
In conf/zookeeper.properties
Add the broker list:
server.1=172.31.48.1:2888:3888
server.2=172.31.48.2:2888:3888
Creating a file named myid and placing it in ZooKeeper's data directory. The contents of this file should be the index of the server as specific by the
server.<number>
.
mkdir state/zookeeper
echo 1 > state/zookeeper/myid
State is stored only in ZooKeeper (whichever instance is configured in your conf/state-mangement.xml file). If there is no ZooKeeper instance that is properly configured there, it will fail to store any state and you'll end up re-listing those files when NiFi restarts.
ERROR [Timer-Driven Process Thread-10] o.a.nifi.processors.standard.FetchSFTP FetchSFTP[id=5cdfac90-2d07-443e-97b6-b06a1a883a22] FetchSFTP[id=5cdfac90-2d07-443e-97b6-b06a1a883a22] failed to process due to org.apache.nifi.processor.exception.ProcessException: IOException thrown from FetchSFTP[id=5cdfac90-2d07-443e-97b6-b06a1a883a22]: java.io.IOException: error; rolling back session: org.apache.nifi.processor.exception.ProcessException: IOException thrown from FetchSFTP[id=5cdfac90-2d07-443e-97b6-b06a1a883a22]: java.io.IOException: error 2016-08-09 18:59:48,093 ERROR [Timer-Driven Process Thread-10] o.a.nifi.processors.standard.FetchSFTP org.apache.nifi.processor.exception.ProcessException: IOException thrown from FetchSFTP[id=5cdfac90-2d07-443e-97b6-b06a1a883a22]: java.io.IOException: error
In conf/state-management.xml
Add zookeeper broker list for NiFi to fetch state in zookeeper
"Connect String": 172.31.48.1:2181,172.31.48.2:2181
Step 4. Start NiFi Service
sudo ./nifi-0.7.0/bin/nifi.sh start
Check Service Running:
ps -ef | grep nifi
Note:
If nifi.web.http.host is not explicitly set, this will default to localhost which then be interpreted by the NCM incorrectly when transmitted with the heartbeat.
An error will show up:
An unexpected error has occurred
No nodes were able to process this request.
Reference:
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#clustering
https://community.hortonworks.com/questions/11891/how-to-troubleshoot-nifi-non-secure-cluster.html
No comments:
Post a Comment