Data Transfer Across NiFi Cluster

I have one NiFi cluster and one NiFi instances deployed in AWS. 

The idea is to collect data from remote edge instance, then send to central cluster for processing.

US-East Cluster: 172.31.48.1

Dataflow: RGP->PutFile

# Site to Site properties

nifi.remote.input.socket.host=172.31.48.1

nifi.remote.input.socket.port=9090

nifi.remote.input.secure=false

# web properties #

nifi.web.http.host=172.31.48.1

nifi.web.http.port=8080

Singapore Edge instance: 172.31.11.2

Dataflow: GetFile->output_port

# Site to Site properties

nifi.remote.input.socket.host=172.31.11.2

nifi.remote.input.socket.port=9090

nifi.remote.input.secure=false

# web properties #

nifi.web.http.host=172.31.11.2

nifi.web.http.port=8080

The GetFile processor can fetch files and queue them in Singapore instance.

The RGP can successfully detect the "output_port" by setting the url as "http://sig-public-ip:8080/nifi" in US-east instance.

However, RGP log shows :

14:38:14 UTC ERROR e93c67c6-3472-44e4-aa3e-78613ce37e47 us-ease-public-ip:8080

RemoteGroupPort[name=output_port,target=http://sig-public-ip:8080/nifi] failed to communicate with http://sig-public-ip:8080/nifi due to java.net.SocketTimeoutException

Analyze Logs:

By setting <logger name="org.apache.nifi.remote" level="DEBUG"/> in your logback.xml on nodes of both clusters. We can find below logs in nifi/logs/nifi-app.log from the RPG worker node.

2016-08-12 15:05:43,885 DEBUG [Timer-Driven Process Thread-5] o.a.n.r.c.socket.EndpointConnectionPool EndpointConnectionPool[Cluster URL=http://54.179.173.154:8080/nifi] Establishing site-to-site connection with PeerStatus[hostname=172.31.11.203,port=9090,secure=false,flowFileCount=1]

2016-08-12 15:05:45,738 INFO [Clustering Tasks Thread-2] org.apache.nifi.cluster.heartbeat Heartbeat created at 2016-08-12 15:05:45,625 and sent at 2016-08-12 15:05:45,738; send took 0 millis

2016-08-12 15:05:48,811 ERROR [Timer-Driven Process Thread-10] o.a.nifi.remote.StandardRemoteGroupPort RemoteGroupPort[name=eu_port,target=http://52.48.156.238:8080/nifi] failed to communicate with http://52.48.156.238:8080/nifi due to java.net.SocketTimeoutException

2016-08-12 15:05:48,811 ERROR [Timer-Driven Process Thread-10] o.a.nifi.remote.StandardRemoteGroupPort

java.net.SocketTimeoutException: null

        at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:118) ~[na:1.8.0_101]

        at org.apache.nifi.remote.client.socket.EndpointConnectionPool.establishSiteToSiteConnection(EndpointConnectionPool.java:708) ~[nifi-site-to-site-client-0.7.0.jar:0.7.0]

This means the Site-to-Site connection on [hostname=172.31.11.203,port=9090,secure=false,flowFileCount=1] isn't established.

The solution is:

Use public ip instead of private ip in the edge instance's site-to-site property.

Since, NCM connects NiFi instance through 8080 port, then collect info of site-to-site protocol.

The private ip can't be recognized and used to build connection.