This page explains how to configure and deploy MiNiFi agents.
The Java agent is able to run most of NiFi's available processors, but is a larger binary distribution (49MB) and consumes greater system resources (24MB max JVM heapsize by default). If you need maximum flexibility to make routing and processing decisions at your data's point of origin, the Java agent is a good fit.
The C++ agent is a smaller binary (3.2MB), consumes low system memory (about 5MB at idle) but has a limited subset of processors. If your primary concern is gathering and pushing data to downstream consumers and minimizing system impact, the C++ agent is a good fit.
Install the appropriate OS level dependencies:
flowStatus and dump are available only for MiNiFi Java.
For MiNiFi Java, the agent logs to $MINIFI_HOME/logs/minifi-app.log.
For MiNiFi C++, the agent logs to $MINIFI_HOME/minifi-app.log.txt
Agents execute the flow defined in $MINIFI_HOME/conf/config.yml. The shipped config.yml file has an "empty" flow.
While it's possible to create and modify config.yml programmatically or by hand, the more common scenario is to:
What follows is an illustrated walkthrough of configuring MiNiFi and NiFi to generate and collect log data.
First, check that $NIFI_HOME/conf/nifi.properties has a Site to Site input socket port specified:
# Site to Site properties nifi.remote.input.host= nifi.remote.input.secure=false nifi.remote.input.socket.port=1026 nifi.remote.input.http.enabled=true nifi.remote.input.http.transaction.ttl=30 sec
Restart NiFi so changes take effect.
On NiFi's web UI, place an Input Port "From MiNiFi" and connect it to a downstream processor:
Create a new process group, and double click on it to enter it. Working inside this new processor group will let us create a clean template that includes flow components meant to run in MiNiFi.
Inside the process group, create a Remote Process Group (RPG) and give it the URL of your NiFi instance. The RPG will connect to NiFi and discover the input port created earlier.
Create a TailFile processor and connect it to the RPG, selecting the "From MiNiFi" input port.
Now create a template, download it as an XML file, and download the Converter ToolKit.
From the toolkit directory:
[user@host minifi-toolkit-0.0.1]# ./bin/config.sh transform /path/to/template.xml /output/path/config.yml Java home: /usr/lib/jvm/jre-1.8.0-openjdk MiNiFi Toolkit home: /minifi-toolkit-0.0.1 No validation errors found in template.
Open the config.yml file and add Port and Host to a Properties section at the end of the Input Ports section (at the end of the file):
Remote Processing Groups: - name: http://localhost:8080/nifi url: http://localhost:8080/nifi comment: '' timeout: 30 sec yield period: 10 sec Input Ports: - id: AUTOGENERATED_NIFI_PORT_ID_HERE name: MiNiFi-input comment: '' max concurrent tasks: 1 use compression: false Properties: # Deviates from spec and will later be removed when this is autonegotiated Port: 1026 Host Name: localhost
Copy the generated config.yml file into $MINIFI/conf/config.yml, and start MiNiFi:
[user@host nifi-minifi-cpp-0.1.0]# ./bin/minifi.sh start Starting MiNiFi with PID 463 and pid file /nifi-minifi-cpp-0.1.0/bin/.minifi.pid
After a few seconds, you should see NiFi receiving data on the input port:
If you do not see data in the queue, it may have already been consumed by the downstream processor (PutFile in this case). If no data is arriving on the input port at all, check the MiNiFi log files, and confirm that the file being tailed exists and is not empty.