Report NiFi flow data set level lineage to Apache Atlas. End-to-end lineages across NiFi environments and other systems can be reported if those are connected by different protocols and data set, such as NiFi Site-to-Site, Kafka topic or Hive tables ... etc. Atlas lineage reported by this reporting task can be useful to grasp the high level relationships between processes and data sets, in addition to NiFi provenance events providing detailed event level lineage. See 'Additional Details' for further description and limitations.
| Display Name | API Name | Default Value | Allowable Values | Description |
|---|
| Atlas URLs | atlas-urls | | | Comma separated URL of Atlas Servers (e.g. http://atlas-server-hostname:21000 or https://atlas-server-hostname:21443). For accessing Atlas behind Knox gateway, specify Knox gateway URL (e.g. https://knox-hostname:8443/gateway/{topology-name}/atlas). If not specified, 'atlas.rest.address' in Atlas Configuration File is used.
Supports Expression Language: true (will be evaluated using Environment variables only) |
| Atlas Configuration Directory | atlas-conf-dir | | | Directory path that contains 'atlas-application.properties' file. If not specified and 'Create Atlas Configuration File' is disabled, then, 'atlas-application.properties' file under root classpath is used.
This property requires exactly one directory to be provided..
Supports Expression Language: true (will be evaluated using Environment variables only) |
| Create Atlas Configuration File | atlas-conf-create | false | | If enabled, 'atlas-application.properties' file will be created in 'Atlas Configuration Directory' automatically when this Reporting Task starts. Note that the existing configuration file will be overwritten. |
| Atlas Default Metadata Namespace | atlas-default-cluster-name | | | Namespace for Atlas entities reported by this ReportingTask. If not specified, 'atlas.metadata.namespace' or 'atlas.cluster.name' (the former having priority) in Atlas Configuration File is used. Multiple mappings can be configured by user defined properties. See 'Additional Details...' for more.
Supports Expression Language: true (will be evaluated using Environment variables only) |
| Lineage Strategy | nifi-lineage-strategy | Simple Path | - Simple Path
 - Complete Path
| Specifies granularity on how NiFi data flow should be reported to Atlas. NOTE: It is strongly recommended to keep using the same strategy once this reporting task started to keep Atlas data clean. Switching strategies will not delete Atlas entities created by the old strategy. Having mixed entities created by different strategies makes Atlas lineage graph noisy. For more detailed description on each strategy and differences, refer 'NiFi Lineage Strategy' section in Additional Details. |
| Provenance Record Start Position | provenance-start-position | Beginning of Stream | - Beginning of Stream
- End of Stream
| If the Reporting Task has never been run, or if its state has been reset by a user, specifies where in the stream of Provenance Events the Reporting Task should start |
| Provenance Record Batch Size | provenance-batch-size | 1000 | | Specifies how many records to send in a single batch, at most. |
| NiFi URL for Atlas | atlas-nifi-url | | | NiFi URL is used in Atlas to represent this NiFi cluster (or standalone instance). It is recommended to use one that can be accessible remotely instead of using 'localhost'.
Supports Expression Language: true (will be evaluated using Environment variables only) |
| Atlas Authentication Method | atlas-authentication-method | Basic | - Basic
- Kerberos
| Specify how to authenticate this reporting task to Atlas server. |
| Atlas Username | atlas-username | | | User name to communicate with Atlas.
Supports Expression Language: true (will be evaluated using Environment variables only) |
| Atlas Password | atlas-password | | | Password to communicate with Atlas.
Sensitive Property: true
Supports Expression Language: true (will be evaluated using Environment variables only) |
| Kerberos Credentials Service | kerberos-credentials-service | | Controller Service API:
KerberosCredentialsService
Implementation: KeytabCredentialsService | Specifies the Kerberos Credentials Controller Service that should be used for authenticating with Kerberos |
| Kerberos Principal | nifi-kerberos-principal | | | The Kerberos principal for this NiFi instance to access Atlas API and Kafka brokers. If not set, it is expected to set a JAAS configuration file in the JVM properties defined in the bootstrap.conf file. This principal will be set into 'sasl.jaas.config' Kafka's property.
Supports Expression Language: true (will be evaluated using Environment variables only) |
| Kerberos Keytab | nifi-kerberos-keytab | | | The Kerberos keytab for this NiFi instance to access Atlas API and Kafka brokers. If not set, it is expected to set a JAAS configuration file in the JVM properties defined in the bootstrap.conf file. This principal will be set into 'sasl.jaas.config' Kafka's property.
This property requires exactly one file to be provided..
Supports Expression Language: true (will be evaluated using Environment variables only) |
| SSL Context Service | ssl-context-service | | Controller Service API:
SSLContextService
Implementations: StandardRestrictedSSLContextService
StandardSSLContextService | Specifies the SSL Context Service to use for communicating with Atlas and Kafka. |
| Kafka Bootstrap Servers | kafka-bootstrap-servers | | | Kafka Bootstrap Servers to send Atlas hook notification messages based on NiFi provenance events. E.g. 'localhost:9092' NOTE: Once this reporting task has started, restarting NiFi is required to changed this property as Atlas library holds a unmodifiable static reference to Kafka client.
Supports Expression Language: true (will be evaluated using Environment variables only) |
| Kafka Security Protocol | kafka-security-protocol | PLAINTEXT | - PLAINTEXT
- SSL
- SASL_PLAINTEXT
- SASL_SSL
| Protocol used to communicate with Kafka brokers to send Atlas hook notification messages. Corresponds to Kafka's 'security.protocol' property. |
| Kafka Kerberos Service Name | kafka-kerberos-service-name | kafka | | The service name that matches the primary name of the Kafka server configured in the broker JAAS file. This can be defined either in Kafka's JAAS config or in Kafka's config. Corresponds to Kafka's 'security.protocol' property. It is ignored unless one of the SASL options of the <Security Protocol> are selected.
Supports Expression Language: true (will be evaluated using Environment variables only) |
| Atlas Connect Timeout | atlas-connect-timeout | 60 sec | | Max wait time for connection to Atlas. |
| Atlas Read Timeout | atlas-read-timeout | 60 sec | | Max wait time for response from Atlas. |
| AWS S3 Model Version | aws-s3-model-version | v2 | - v1
- v2
| Specifies what type of AWS S3 directory entities will be created in Atlas for s3a:// transit URIs (eg. PutHDFS with S3 integration). NOTE: It is strongly recommended to keep using the same AWS S3 entity model version once this reporting task started to keep Atlas data clean. Switching versions will not delete existing Atlas entities created by the old version, nor migrate them to the new version. |
| Filesystem Path Entities Level | filesystem-paths-level | File | - File
- Directory
| Specifies how the filesystem path entities (fs_path and hdfs_path) will be logged in Atlas: File or Directory level. In case of File level, each individual file entity will be sent to Atlas as a separate entity with the full path including the filename. Directory level only logs the path of the parent directory without the filename. This setting affects processors working with files, like GetFile or PutHDFS. NOTE: Although the default value is File level for backward compatibility reasons, it is highly recommended to set it to Directory level because File level logging can generate a huge number of entities in Atlas. |
Dynamic Properties allow the user to specify both the name and value of a property.