Apache NiFi Toolkit Guide

Overview

The NiFi Toolkit contains several command line utilities to setup and support NiFi in standalone and clustered environments. The utilities include:

CLI — The cli tool enables administrators to interact with NiFi and NiFi Registry instances to automate tasks such as deploying versioned flows and managing process groups and cluster nodes.
Encrypt Config — The encrypt-config tool encrypts the sensitive keys in the nifi.properties file to facilitate the setup of a secure NiFi instance.

The utilities are executed with scripts found in the bin folder of your NiFi Toolkit installation.

The NiFi Toolkit is downloaded separately from NiFi (see the Apache NiFi Downloads page).

Prerequisites for Running in a Secure Environment

For secured nodes and clusters, two policies should be configured in advance:

Access the controller – A user that will have access to these utilities should be authorized in NiFi by creating an “access the controller” policy (/controller) with both view and modify rights
Proxy user request – If not previously set, node’s identity (the DN value of the node’s certificate) should be authorized to proxy requests on behalf of a user

When executing either the Notify or Node Manager tools in a secured environment the proxyDN flag option should be used in order to properly identify the user that was authorized to execute these commands. In non-secure environments, or if running the status operation on the Node Manager tool, the flag is ignored.

NiFi CLI

This tool offers a CLI focused on interacting with NiFi and NiFi Registry in order to automate tasks, such as deploying flows from a NIFi Registy to a NiFi instance or managing process groups and cluster nodes.

Usage

The CLI toolkit can be executed in standalone mode to execute a single command, or interactive mode to enter an interactive shell.

To execute a single command:

./bin/cli.sh <command> <args>

To launch the interactive shell:

./bin/cli.sh

To show help:

./bin/cli.sh -h

The following are available commands:

demo quick-import
nifi current-user
nifi cluster-summary
nifi connect-node
nifi delete-node
nifi disconnect-node
nifi get-root-id
nifi get-node
nifi get-nodes
nifi offload-node
nifi list-reg-clients
nifi create-reg-client
nifi update-reg-client
nifi get-reg-client-id
nifi pg-import
nifi pg-start
nifi pg-stop
nifi pg-create
nifi pg-get-vars
nifi pg-set-var
nifi pg-get-version
nifi pg-change-version
nifi pg-get-all-versions
nifi pg-list
nifi pg-status
nifi pg-get-services
nifi pg-create-service
nifi pg-enable-services
nifi pg-disable-services
nifi pg-get-param-context
nifi pg-set-param-context
nifi pg-replace
nifi get-services
nifi get-service
nifi create-service
nifi enable-services
nifi disable-services
nifi get-reporting-tasks
nifi get-reporting-task
nifi create-reporting-task
nifi delete-reporting-task
nifi start-reporting-tasks
nifi stop-reporting-tasks
nifi list-users
nifi create-user
nifi list-user-groups
nifi create-user-group
nifi update-user-group
nifi get-policy
nifi update-policy
nifi list-param-contexts
nifi get-param-context
nifi create-param-context
nifi delete-param-context
nifi set-inherited-param-contexts
nifi remove-inherited-param-contexts
nifi set-param
nifi delete-param
nifi export-param-context
nifi import-param-context
nifi merge-param-context
nifi list-param-providers
nifi get-param-provider
nifi create-param-provider
nifi delete-param-provider
nifi fetch-params
nifi set-param-provider-property
nifi get-access-token
nifi get-access-token-spnego
nifi logout-access-token
nifi get-controller-configuration
nifi update-controller-configuration
registry current-user
registry list-buckets
registry create-bucket
registry delete-bucket
registry list-flows
registry create-flow
registry delete-flow
registry list-flow-versions
registry export-flow-version
registry import-flow-version
registry sync-flow-versions
registry transfer-flow-version
registry diff-flow-versions
registry upload-bundle
registry upload-bundles
registry list-bundle-groups
registry list-bundle-artifacts
registry list-bundle-versions
registry download-bundle
registry get-bundle-checksum
registry list-extension-tags
registry list-extensions
registry list-users
registry create-user
registry update-user
registry list-user-groups
registry create-user-group
registry update-user-group
registry get-policy
registry update-policy
registry update-bucket-policy
registry get-access-token
registry get-access-token-spnego
registry logout-access-token
registry export-all-flows
registry import-all-flows
session keys
session show
session get
session set
session remove
session clear
exit
help

To display extensive help for a specific command:

./bin/cli.sh <command> -h

Property/Argument Handling

Most commands will require specifying a baseUrl for the NiFi or NiFi Registry instance.

An example command to list the buckets in a NiFi Registry instance would be the following:

./bin/cli.sh registry list-buckets -u http://localhost:18080

In order to avoid specifying the URL (and possibly other optional arguments for TLS) on every command, you can define a properties file containing the repetitive arguments.

An example properties file for a local NiFi Registry instance would look like the following:

 baseUrl=http://localhost:18080
 keystore=
 keystoreType=
 keystorePasswd=
 keyPasswd=
 truststore=
 truststoreType=
 truststorePasswd=
 proxiedEntity=

This properties file can then be used on a command by specifying -p:

./bin/cli.sh registry list-buckets -p /path/to/local-nifi-registry.properties

You could then maintain a properties file for each environment you plan to interact with, such as Dev, QA, and Prod.

In addition to specifying a properties file on each command, you can setup a default properties file to be used in the event that no properties file is specified.

The default properties file is specified using the session concept, which persists to the users home directory in a file called .nifi-cli.config.

An example of setting the default property files for NiFi would be the following:

./bin/cli.sh session set nifi.props /path/to/local-nifi.properties

An example for NiFi Registry would be the following:

./bin/cli.sh session set nifi.reg.props /path/to/local-nifi-registry.properties

This will write the above properties into the .nifi-cli.config in the user’s home directory and will allow commands to be executed without specifying a URL or properties file:

./bin/cli.sh registry list-buckets

The above command will now use the baseUrl from local-nifi-registry.properties.

The order of resolving an argument is the following:

A direct argument overrides anything in a properties file or session
A properties file argument (-p) overrides the session
The session is used when nothing else is specified

Security Configuration

If NiFi and NiFi Registry are secured, then commands executed from the CLI will need to make a TLS connection and authenticate as a user with permissions to perform the desired action.

Currently the CLI supports authenticating with a client certificate and an optional proxied-entity. A common scenario would be running the CLI from one of the nodes where NiFi or NiFi Registry is installed, which allows the CLI to use the same keystore and truststore as the NiFi/NiFi Registry instance.

The security configuration can be specified per-command, or in one of the properties files described in the previous section.

The examples below are for NiFi Registry, but the same concept applies for NiFi commands.

Example - Secure NiFi Registry without Proxied-Entity

Assuming we have a keystore containing the certificate for "CN=user1, OU=NIFI", an example properties file would be the following:

 baseUrl=https://localhost:18443
 keystore=/path/to/keystore.jks
 keystoreType=JKS
 keystorePasswd=changeme
 keyPasswd=changeme
 truststore=/path/to/truststore.jks
 truststoreType=JKS
 truststorePasswd=changeme

In this example, commands will be executed as "CN=user1, OU=NIFI". This user would need to be a user in NiFi Registry, and commands accessing buckets would be restricted to buckets this user has access to.

Example - Secure NiFi Registry with Proxied-Entity

Assuming we have access to the keystore of NiFi Registry itself, and that NiFi Registry is also configured to allow Kerberos or LDAP authentication, an example properties file would be the following:

 baseUrl=https://localhost:18443
 keystore=/path/to/keystore.jks
 keystoreType=JKS
 keystorePasswd=changeme
 keyPasswd=changeme
 truststore=/path/to/truststore.jks
 truststoreType=JKS
 truststorePasswd=changeme
 proxiedEntity=user1@NIFI.COM

In this example, the certificate in keystore.jks would be for the NiFi Registry server, for example "CN=localhost, OU=NIFI". This identity would need to be defined as a user in NiFi Registry and given permissions to 'Proxy'.

"CN=localhost, OU=NIFI" would be proxying commands to be executed as user1@NIFI.COM.

Interactive Usage

In interactive mode the tab key can be used to perform auto-completion.

For example, typing tab at an empty prompt should display possible commands for the first argument:

#>
demo       exit       help       nifi       registry   session

Typing "nifi " and then a tab will show the sub-commands for NiFi:

#> nifi
cluster-summary         export-param-context    list-users              pg-set-param-context
connect-node            get-node                merge-param-context     pg-set-var
create-param-context    get-nodes               offload-node            pg-start
create-reg-client       get-param-context       pg-change-version       pg-status
create-reporting-task   get-policy              pg-create-service       pg-stop
create-service          get-reg-client-id       pg-create-service       set-param
create-user             get-reporting-task      pg-disable-services     start-reporting-tasks
create-user-group       get-reporting-tasks     pg-enable-services      stop-reporting-tasks
current-user            get-root-id             pg-get-all-versions     update-policy
delete-node             get-service             pg-get-param-context    update-reg-client
delete-param            get-services            pg-get-services         update-user-group
delete-param-context    import-param-context    pg-get-vars             delete-reporting-task
disable-services        list-param-contexts     pg-get-version
disconnect-node         list-reg-clients        pg-import
enable-services         list-user-groups        pg-list

Arguments that represent a path to a file, such as -p or when setting a properties file in the session, will auto-complete the path being typed:

#> session set nifi.props /tmp/
dir1/   dir2/   dir3/

Output

Most commands support the ability to specify an --outputType argument, or -ot for short.

Currently the output type may be simple or json.

The default output type in interactive mode is simple, and the default output type in standalone mode is json.

Example of simple output for list-buckets:

#> registry list-buckets -ot simple
My Bucket - 3c7b7467-0012-4d8f-a918-6aa42b6b9d39

Example of json output for list-buckets:

#> registry list-buckets -ot json
[ {
  "identifier" : "3c7b7467-0012-4d8f-a918-6aa42b6b9d39",
  "name" : "My Bucket",
  "createdTimestamp" : 1516718733854,
  "permissions" : {
    "canRead" : true,
    "canWrite" : true,
    "canDelete" : true
  },
  "link" : {
    "params" : {
      "rel" : "self"
    },
    "href" : "buckets/3c7b7467-0012-4d8f-a918-6aa42b6b9d39"
  }
} ]

Back-Referencing

When using the interactive CLI, a common scenario will be using an id from a previous result as the input to the next command. Back-referencing provides a shortcut for referencing a result from the previous command via a positional reference.

Not every command produces back-references. To determine if a command supports back-referencing, check the usage.

#> registry list-buckets help
Lists the buckets that the current user has access to.
PRODUCES BACK-REFERENCES

A common scenario for utilizing back-references would be the following:

User starts by exploring the available buckets in a registry instance

#> registry list-buckets
#   Name           Id                                     Description
-   ------------   ------------------------------------   -----------
1   My Bucket      3c7b7467-0012-4d8f-a918-6aa42b6b9d39   (empty)
2   Other Bucket   175fb557-43a2-4abb-871f-81a354f47bc2   (empty)

User then views the flows in one of the buckets using a back-reference to the bucket id from the previous result in position 1

#> registry list-flows -b &1
Using a positional back-reference for 'My Bucket'
#   Name      Id                                     Description
-   -------   ------------------------------------   ----------------
1   My Flow   06acb207-d2f1-447f-85ed-9b8672fe6d30   This is my flow.

User then views the version of the flow using a back-reference to the flow id from the previous result in position 1

#> registry list-flow-versions -f &1
Using a positional back-reference for 'My Flow'
Ver   Date                         Author                     Message
---   --------------------------   ------------------------   -------------------------------------
1     Tue, Jan 23 2018 09:48 EST   anonymous                  This is the first version of my flow.

User deploys version 1 of the flow using back-references to the bucket and flow id from step 2

#> nifi pg-import -b &1 -f &1 -fv 1
Using a positional back-reference for 'My Bucket'
Using a positional back-reference for 'My Flow'
9bd157d4-0161-1000-b946-c1f9b1832efd

The reason step 4 was able to reference the results from step 2, is because the list-flow-versions command in step 3 does not produce back-references, so the results from step 2 are still available.

Adding Commands

To add a NiFi command, create a new class that extends AbstractNiFiCommand:

public class MyCommand extends AbstractNiFiCommand {

  public MyCommand() {
      super("my-command");
  }

  @Override
  protected void doExecute(NiFiClient client, Properties properties)
          throws NiFiClientException, IOException, MissingOptionException, CommandException {
      // TODO implement
  }

  @Override
  public String getDescription() {
      return "This is my new command";
  }
}

Add the new command to NiFiCommandGroup:

commands.add(new MyCommand());

To add a NiFi Registry command, perform the same steps, but extend from AbstractNiFiRegistryCommand, and add the command to NiFiRegistryCommandGroup.

Encrypt-Config Tool

The encrypt-config command line tool (invoked as ./bin/encrypt-config.sh or bin\encrypt-config.bat) reads from a nifi.properties file with plaintext sensitive configuration values, prompts for a root password or raw hexadecimal key, and encrypts each value. It replaces the plain values with the protected value in the same file, or writes to a new nifi.properties file if specified.

The default encryption algorithm utilized is AES-GCM with 256-bit keys.

Usage

To show help:

./bin/encrypt-config.sh -h

NiFi

The following are available options when targeting NiFi:

-h,--help Show usage information (this message)
-v,--verbose Sets verbose mode (default false)
-n,--niFiProperties <file> The nifi.properties file containing unprotected config values (will be overwritten unless -o is specified)
-o,--outputNiFiProperties <file> The destination nifi.properties file containing protected config values (will not modify input nifi.properties)
-l,--loginIdentityProviders <file> The login-identity-providers.xml file containing unprotected config values (will be overwritten unless -i is specified)
-i,--outputLoginIdentityProviders <file> The destination login-identity-providers.xml file containing protected config values (will not modify input login-identity-providers.xml)
-a,--authorizers <file> The authorizers.xml file containing unprotected config values (will be overwritten unless -u is specified)
-u,--outputAuthorizers <file> The destination authorizers.xml file containing protected config values (will not modify input authorizers.xml)
-f,--flowJson <file> The flow.json.gz file currently protected with old password (will be overwritten unless -g is specified)
-g,--outputFlowJson <file> The destination flow.json.gz file containing protected config values (will not modify input flow.json.gz)
-b,--bootstrapConf <file> The bootstrap.conf file to persist root key and to optionally provide any configuration for the protection scheme.
-B,--outputBootstrapConf <file> The destination bootstrap.conf file to persist root key. If specified, the input bootstrap.conf will not be modified.
-S,--protectionScheme <protectionScheme> Selects the protection scheme for encrypted properties. Valid values are: [AES_GCM, HASHICORP_VAULT_TRANSIT, HASHICORP_VAULT_KV, AWS_KMS, AWS_SECRETSMANAGER, AZURE_KEYVAULT_KEY, AZURE_KEYVAULT_SECRET, GCP_KMS] (default is AES_GCM)
-k,--key <keyhex> The raw hexadecimal key to use to encrypt the sensitive properties
-e,--oldKey <keyhex> The old raw hexadecimal key to use during key migration
-H,--oldProtectionScheme <protectionScheme> The old protection scheme to use during encryption migration (see --protectionScheme for possible values). Default is AES_GCM
-p,--password <password> The password from which to derive the key to use to encrypt the sensitive properties
-w,--oldPassword <password> The old password from which to derive the key during migration
-r,--useRawKey If provided, the secure console will prompt for the raw key value in hexadecimal form
-m,--migrate If provided, the nifi.properties and/or login-identity-providers.xml sensitive properties will be re-encrypted with the new scheme
-x,--encryptFlowJsonOnly If provided, the properties in flow.json.gz will be re-encrypted with a new key but the nifi.properties and/or login-identity-providers.xml files will not be modified
-s,--propsKey <password|keyhex> The password or key to use to encrypt the sensitive processor properties in flow.json.gz
-A,--newFlowAlgorithm <algorithm> The algorithm to use to encrypt the sensitive processor properties in flow.json.gz

NiFi Registry

The following are available options when targeting NiFi Registry using the --nifiRegistry flag:

-h,--help Show usage information (this message)
-v,--verbose Sets verbose mode (default false)
-p,--password <password> Protect the files using a password-derived key. If an argument is not provided to this flag, interactive mode will be triggered to prompt the user to enter the password.
-k,--key <keyhex> Protect the files using a raw hexadecimal key. If an argument is not provided to this flag, interactive mode will be triggered to prompt the user to enter the key.
-S,--protectionScheme <protectionScheme> Selects the protection scheme for encrypted properties. Valid values are: [AES_GCM, HASHICORP_VAULT_TRANSIT, HASHICORP_VAULT_KV, AWS_KMS, AWS_SECRETSMANAGER, AZURE_KEYVAULT_KEY, AZURE_KEYVAULT_SECRET, GCP_KMS] (default is AES_GCM)
-w,--oldPassword <password> If the input files are already protected using a password-derived key, this specifies the old password so that the files can be unprotected before re-protecting.
-e,--oldKey <keyhex> If the input files are already protected using a key, this specifies the raw hexadecimal key so that the files can be unprotected before re-protecting.
-H,--oldProtectionScheme <protectionScheme> The old protection scheme to use during encryption migration (see --protectionScheme for possible values). Default is AES_GCM.
-b,--bootstrapConf <file> The bootstrap.conf file containing no root key or an existing root key, and any other protection scheme configuration properties. If a new password or key is specified (using -p or -k) and no output bootstrap.conf file is specified, then this file will be overwritten to persist the new master key.
-B,--outputBootstrapConf <file> The destination bootstrap.conf file to persist root key. If specified, the input bootstrap.conf will not be modified.
-r,--nifiRegistryProperties <file> The nifi-registry.properties file containing unprotected config values, overwritten if no output file specified.
-R,--outputNifiRegistryProperties <file> The destination nifi-registry.properties file containing protected config values.
-a,--authorizersXml <file> The authorizers.xml file containing unprotected config values, overwritten if no output file specified.
-u,--outputAuthorizersXml <file> The destination authorizers.xml file containing protected config values.
-i,--identityProvidersXml <file> The identity-providers.xml file containing unprotected config values, overwritten if no output file specified.
-I,--outputIdentityProvidersXml <file> The destination identity-providers.xml file containing protected config values.

Protection Schemes

The protection scheme can be selected during encryption using the --protectionScheme flag. During migration, the former protection scheme is specified using the --oldProtectionScheme flag. This distinction allows a set of protected configuration files to be migrated not only to a new key, but to a completely different protection scheme.

AES_GCM

The default protection scheme, AES-G/CM simply encrypts sensitive properties and marks their protection as either aes/gcm/256 or aes/gcm/256 as appropriate. This protection is all done within NiFi itself.

HASHICORP_VAULT_TRANSIT

This protection scheme uses HashiCorp Vault Transit Secrets Engine to outsource encryption to a configured Vault server. All HashiCorp Vault configuration is stored in the bootstrap-hashicorp-vault.conf file, as referenced in the bootstrap.conf of a NiFi or NiFi Registry instance. Therefore, when using the HASHICORP_VAULT_TRANSIT protection scheme, the nifi(.registry)?.bootstrap.protection.hashicorp.vault.conf property in the bootstrap.conf specified using the -b flag must be available to the Encrypt Configuration Tool and must be configured as described in the HashiCorp Vault providers section in the NiFi Administration Guide.

HASHICORP_VAULT_KV

This protection scheme uses HashiCorp Vault Key Value Secrets Engine Version 1 to store sensitive values as Vault Secrets. All HashiCorp Vault configuration is stored in the bootstrap-hashicorp-vault.conf file, as referenced in the bootstrap.conf of a NiFi or NiFi Registry instance. Therefore, when using the HASHICORP_VAULT_KV protection scheme, the nifi(.registry)?.bootstrap.protection.hashicorp.vault.conf property in the bootstrap.conf specified using the -b flag must be available to the Encrypt Configuration Tool and must be configured as described in the HashiCorp Vault providers section in the NiFi Administration Guide.

AWS_KMS

This protection scheme uses AWS Key Management Service for encryption and decryption. AWS KMS configuration properties can be stored in the bootstrap-aws.conf file, as referenced in the bootstrap.conf of NiFi or NiFi Registry. If the configuration properties are not specified in bootstrap-aws.conf, then the provider will attempt to use the AWS default credentials provider, which checks standard environment variables and system properties. Therefore, when using the AWS_KMS protection scheme, the nifi(.registry)?.bootstrap.protection.aws.conf property in the bootstrap.conf specified using the -b flag must be available to the Encrypt Configuration Tool and must be configured as described in the AWS KMS provider section in the NiFi Administration Guide.

AWS_SECRETSMANAGER

This protection scheme uses AWS Secrets Manager Service to store sensitive values as AWS Secrets. AWS Secrets Manager configuration properties can be stored in the bootstrap-aws.conf file, as referenced in the bootstrap.conf of NiFi or NiFi Registry. If the configuration properties are not specified in bootstrap-aws.conf, then the provider will attempt to use the AWS default credentials provider, which checks standard environment variables and system properties. Therefore, when using the AWS_SECRETS_MANAGER protection scheme, the nifi(.registry)?.bootstrap.protection.aws.conf property in the bootstrap.conf specified using the -b flag must be available to the Encrypt Configuration Tool and must be configured as described in the AWS Secrets Manager provider section in the NiFi Administration Guide.

Microsoft Azure Key Vault Sensitive Property Providers

Azure Key Vault configuration properties can be stored in the bootstrap-azure.conf file, as referenced in the bootstrap.conf of NiFi or NiFi Registry.

Azure Key Vault providers will use the DefaultAzureCredential for authentication. The Azure Identity client library describes the process for credentials resolution, which leverages environment variables, system properties, and falls back to Managed Identity authentication.

When using Azure Key Vault providers, bootstrap.conf must contain the nifi.bootstrap.protection.azure.keyvault.conf property. The bootstrap.conf file location must be specified using the -b argument when running the Encrypt Config Tool.

AZURE_KEYVAULT_KEY

This protection scheme uses keys managed by Azure Key Vault Keys for encryption and decryption.

See Azure Key Vault Key Provider in the NiFi System Administrator’s Guide for required properties.

AZURE_KEYVAULT_SECRET

This protection scheme uses secrets managed by Azure Key Vault Secrets for storing and retrieving sensitive property values.

See Azure Key Vault Secret Provider in the NiFi System Administrator’s Guide for required properties.

GCP_KMS

This protection scheme uses Google Cloud Key Management Service (Google Cloud Key Management Service) for encryption and decryption. Google Cloud KMS configuration properties are to be stored in the bootstrap-gcp.conf file, as referenced in the bootstrap.conf of NiFi or NiFi Registry. Credentials must be configured as per the following documentation: Google Cloud KMS documentation. Therefore, when using the GCP_KMS protection scheme, the nifi(.registry)?.bootstrap.protection.gcp.kms.conf property in the bootstrap.conf specified using the -b flag must be available to the Encrypt Configuration Tool and must be configured as described in the Google Cloud KMS provider section in the NiFi Administration Guide.

Examples

NiFi

As an example of how the tool works, assume that you have installed the tool on a machine supporting 256-bit encryption and with the following existing values in the nifi.properties file:

# security properties #
nifi.sensitive.props.key=thisIsABadSensitiveKeyPassword
nifi.sensitive.props.algorithm=NIFI_PBKDF2_AES_GCM_256
nifi.sensitive.props.additional.keys=

nifi.security.keystore=/path/to/keystore.jks
nifi.security.keystoreType=JKS
nifi.security.keystorePasswd=thisIsABadKeystorePassword
nifi.security.keyPasswd=thisIsABadKeyPassword
nifi.security.truststore=
nifi.security.truststoreType=
nifi.security.truststorePasswd=

Enter the following arguments when using the tool:

encrypt-config.sh \
-b bootstrap.conf \
-k 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210 \
-n nifi.properties

As a result, the nifi.properties file is overwritten with protected properties and sibling encryption identifiers (aes/gcm/256, the currently supported algorithm):

# security properties #
nifi.sensitive.props.key=n2z+tTTbHuZ4V4V2||uWhdasyDXD4ZG2lMAes/vqh6u4vaz4xgL4aEbF4Y/dXevqk3ulRcOwf1vc4RDQ==
nifi.sensitive.props.key.protected=aes/gcm/256
nifi.sensitive.props.algorithm=NIFI_PBKDF2_AES_GCM_256
nifi.sensitive.props.additional.keys=

nifi.security.keystore=/path/to/keystore.jks
nifi.security.keystoreType=JKS
nifi.security.keystorePasswd=oBjT92hIGRElIGOh||MZ6uYuWNBrOA6usq/Jt3DaD2e4otNirZDytac/w/KFe0HOkrJR03vcbo
nifi.security.keystorePasswd.protected=aes/gcm/256
nifi.security.keyPasswd=ac/BaE35SL/esLiJ||+ULRvRLYdIDA2VqpE0eQXDEMjaLBMG2kbKOdOwBk/hGebDKlVg==
nifi.security.keyPasswd.protected=aes/gcm/256
nifi.security.truststore=
nifi.security.truststoreType=
nifi.security.truststorePasswd=

Additionally, the bootstrap.conf file is updated with the encryption key as follows:

# Root key in hexadecimal format for encrypted sensitive configuration values
nifi.bootstrap.sensitive.key=0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210

Sensitive configuration values are encrypted by the tool by default, however you can encrypt any additional properties, if desired. To encrypt additional properties, specify them as comma-separated values in the nifi.sensitive.props.additional.keys property.

If the nifi.properties file already has valid protected values, those property values are not modified by the tool.

When applied to login-identity-providers.xml and authorizers.xml, the property elements are updated with an encryption attribute:

Example of protected login-identity-providers.xml:

   <!-- LDAP Provider -->
   <provider>
       <identifier>ldap-provider</identifier>
       <class>org.apache.nifi.ldap.LdapProvider</class>
       <property name="Authentication Strategy">START_TLS</property>
       <property name="Manager DN">someuser</property>
       <property name="Manager Password" encryption="aes/gcm/128">q4r7WIgN0MaxdAKM||SGgdCTPGSFEcuH4RraMYEdeyVbOx93abdWTVSWvh1w+klA</property>
       <property name="TLS - Keystore"></property>
       <property name="TLS - Keystore Password" encryption="aes/gcm/128">Uah59TWX+Ru5GY5p||B44RT/LJtC08QWA5ehQf01JxIpf0qSJUzug25UwkF5a50g</property>
       <property name="TLS - Keystore Type"></property>
       ...
   </provider>

Example of protected authorizers.xml:

   <!-- LDAP User Group Provider -->
   <userGroupProvider>
       <identifier>ldap-user-group-provider</identifier>
       <class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class>
       <property name="Authentication Strategy">START_TLS</property>
       <property name="Manager DN">someuser</property>
       <property name="Manager Password" encryption="aes/gcm/128">q4r7WIgN0MaxdAKM||SGgdCTPGSFEcuH4RraMYEdeyVbOx93abdWTVSWvh1w+klA</property>
       <property name="TLS - Keystore"></property>
       <property name="TLS - Keystore Password" encryption="aes/gcm/128">Uah59TWX+Ru5GY5p||B44RT/LJtC08QWA5ehQf01JxIpf0qSJUzug25UwkF5a50g</property>
       <property name="TLS - Keystore Type"></property>
       ...
   </userGroupProvider>

NiFi Registry

As an example of how the tool works, assume that you have installed the tool on a machine supporting 256-bit encryption and with the following existing values in the nifi-registry.properties file:

# security properties #
nifi.registry.security.keystore=/path/to/keystore.jks
nifi.registry.security.keystoreType=JKS
nifi.registry.security.keystorePasswd=thisIsABadKeystorePassword
nifi.registry.security.keyPasswd=thisIsABadKeyPassword
nifi.registry.security.truststore=
nifi.registry.security.truststoreType=
nifi.registry.security.truststorePasswd=

Enter the following arguments when using the tool:

./bin/encrypt-config.sh --nifiRegistry \
-b bootstrap.conf \
-k 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210 \
-r nifi-registry.properties

As a result, the nifi-registry.properties file is overwritten with protected properties and sibling encryption identifiers (aes/gcm/256, the currently supported algorithm):

# security properties #
nifi.registry.security.keystore=/path/to/keystore.jks
nifi.registry.security.keystoreType=JKS
nifi.registry.security.keystorePasswd=oBjT92hIGRElIGOh||MZ6uYuWNBrOA6usq/Jt3DaD2e4otNirZDytac/w/KFe0HOkrJR03vcbo
nifi.registry.security.keystorePasswd.protected=aes/gcm/256
nifi.registry.security.keyPasswd=ac/BaE35SL/esLiJ||+ULRvRLYdIDA2VqpE0eQXDEMjaLBMG2kbKOdOwBk/hGebDKlVg==
nifi.registry.security.keyPasswd.protected=aes/gcm/256
nifi.registry.security.truststore=
nifi.registry.security.truststoreType=
nifi.registry.security.truststorePasswd=

When applied to identity-providers.xml or authorizers.xml, the property elements are updated with an encryption attribute. For example:

<!-- LDAP Provider -->
<provider>
   <identifier>ldap-provider</identifier>
   <class>org.apache.nifi.registry.security.ldap.LdapProvider</class>
   <property name="Authentication Strategy">START_TLS</property>
   <property name="Manager DN">someuser</property>
   <property name="Manager Password" encryption="aes/gcm/128">q4r7WIgN0MaxdAKM||SGgdCTPGSFEcuH4RraMYEdeyVbOx93abdWTVSWvh1w+klA</property>
   <property name="TLS - Keystore">/path/to/keystore.jks</property>
   <property name="TLS - Keystore Password" encryption="aes/gcm/128">Uah59TWX+Ru5GY5p||B44RT/LJtC08QWA5ehQf01JxIpf0qSJUzug25UwkF5a50g</property>
   <property name="TLS - Keystore Type">JKS</property>
   ...
</provider>

Additionally, the bootstrap.conf file is updated with the encryption key as follows:

# Root key in hexadecimal format for encrypted sensitive configuration values
nifi.registry.bootstrap.sensitive.key=0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210

If the nifi-registry.properties file already has valid protected values and you wish to protect additional values using the same root key already present in your bootstrap.conf, then run the tool without specifying a new key:

# bootstrap.conf already contains root key property
# nifi-registy.properties has been updated for nifi.registry.sensitive.props.additional.keys=...

./bin/encrypt-config.sh --nifiRegistry -b bootstrap.conf -r nifi-registry.properties

Sensitive Property Key Migration

In order to change the key used to encrypt the sensitive values, provide the new key or password using the -k or -p flags as usual, and provide the existing key or password using --old-key or --old-password respectively. This will allow the toolkit to decrypt the existing values and re-encrypt them, and update bootstrap.conf with the new key. Only one of the key or password needs to be specified for each phase (old vs. new), and any combination is sufficient:

old key → new key
old key → new password
old password → new key
old password → new password

In order to change the protection scheme (e.g., migrating from AES encryption to Vault encryption), specify the --protectionScheme and --oldProtectionScheme in the migration command.

The following is an example of the commands for protection scheme migration from AES_GCM to AWS_KMS then back. Execute these commands at the nifi directory with the nifi-toolkit directory as a sibling directory. In addition, make sure to update bootstrap-aws.conf with your AWS KMS Key ARN/ID and have your credentials and region configured.

This command encrypts nifi.properties with the AES_GCM protection scheme

./../nifi-toolkit-*-SNAPSHOT/bin/encrypt-config.sh \
-b conf/bootstrap.conf \
-n conf/nifi.properties \
-k 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210 \
-v

This command migrates nifi.properties from using AES_GCM to using AWS_KMS protection scheme

./../nifi-toolkit-*-SNAPSHOT/bin/encrypt-config.sh \
-b conf/bootstrap.conf \
-n conf/nifi.properties \
-S AWS_KMS \
-H AES_GCM \
-e 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210 \
-m \
-v

This command migrates nifi.properties back from AWS_KMS to AES_GCM protection scheme

./../nifi-toolkit-*-SNAPSHOT/bin/encrypt-config.sh \
-b conf/bootstrap.conf \
-n conf/nifi.properties \
-S AES_GCM \
-k 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210 \
-H AWS_KMS \
-m \
-v

Export All Flows

You can use the export-all-flows to perform the following tasks:

List all the buckets
For each bucket, list all flows
For each flow, list all versions
Export each version into a provided directory

Running the command requires an --outputDirectory parameter. The directory must exist and permissions correctly set.

Import All Flows

You can use the import-all-flows to perform the following tasks:

List all files, representing a flow version, from a directory created by export-all-flows
Create all the corresponding buckets
Create all the corresponding flows
Import all the corresponding flow versions

Running the command requires 2 parameters:

--input parameter represents a directory to read files from
--skipExisting optional parameter, configuring how to handle existing flow and flow version creation. If provided the flow and flow version creation will be skipped regardless of there are missing flow versions. If not provided the missing flow versions will be created.

Usage

The input source for an import-all-flows command must be created by an export-all-flows command. To avoid migration conflicts, no modification should be performed in the NiFi Registry during this activity. Buckets and flows with the same name are considered equal.

Export all flow versions:

./bin/cli.sh registry export-all-flows -u http://localhost:18080 --outputDirectory "/my_dir/flow_exports"

Import all flow versions:

./bin/cli.sh registry import-all-flows -u http://localhost:18080 --input "/my_dir/flow_exports" --skipExisting

Expected behaviour

Use case 1: reconfiguring an existing NiFi Registry

NiFi is connecting to NiFi Registry, the NiFi Registry does not change, only its configuration. All the data will be created.

Export versions:

./bin/cli.sh registry export-all-flows -u http://localhost:18080 --outputDirectory "/my_dir/flow_exports"

Stop registry
Switch provider
Start registry

Import versions

./bin/cli.sh registry import-all-flows -u http://localhost:18080 --input "/my_dir/flow_exports" --skipExisting

Use case 2: data replication

NiFi_1 is connecting to NiFi Registry_1 and NiFi_2 is connecting to NiFi Registry_2.

For disaster recovery purposes the data from NiFi Registry_1 needs to be periodically replicated to NiFi Registry_2 via a scheduled job.

The initial version of Nifi Registry_2 needs to be created by this tool.

The missing buckets, flows and versions will be created. If bucket and flow exist the missing versions will be created.

Export versions:

./bin/cli.sh registry export-all-flows -u http://nifi-registry-1:18080 --outputDirectory "/my_dir/flow_exports"

Import versions:

./bin/cli.sh registry import-all-flows -u http://nifi-registry-2:18080 --input "/my_dir/flow_exports"