-
Processors
- AttributeRollingWindow
- AttributesToCSV
- AttributesToJSON
- CalculateRecordStats
- CaptureChangeMySQL
- CompressContent
- ConnectWebSocket
- ConsumeAMQP
- ConsumeAzureEventHub
- ConsumeElasticsearch
- ConsumeGCPubSub
- ConsumeIMAP
- ConsumeJMS
- ConsumeKafka
- ConsumeKinesisStream
- ConsumeMQTT
- ConsumePOP3
- ConsumeSlack
- ConsumeTwitter
- ConsumeWindowsEventLog
- ControlRate
- ConvertCharacterSet
- ConvertRecord
- CopyAzureBlobStorage_v12
- CopyS3Object
- CountText
- CryptographicHashContent
- DebugFlow
- DecryptContentAge
- DecryptContentPGP
- DeduplicateRecord
- DeleteAzureBlobStorage_v12
- DeleteAzureDataLakeStorage
- DeleteByQueryElasticsearch
- DeleteDynamoDB
- DeleteFile
- DeleteGCSObject
- DeleteGridFS
- DeleteMongo
- DeleteS3Object
- DeleteSFTP
- DeleteSQS
- DetectDuplicate
- DistributeLoad
- DuplicateFlowFile
- EncodeContent
- EncryptContentAge
- EncryptContentPGP
- EnforceOrder
- EvaluateJsonPath
- EvaluateXPath
- EvaluateXQuery
- ExecuteGroovyScript
- ExecuteProcess
- ExecuteScript
- ExecuteSQL
- ExecuteSQLRecord
- ExecuteStreamCommand
- ExtractAvroMetadata
- ExtractEmailAttachments
- ExtractEmailHeaders
- ExtractGrok
- ExtractHL7Attributes
- ExtractRecordSchema
- ExtractText
- FetchAzureBlobStorage_v12
- FetchAzureDataLakeStorage
- FetchBoxFile
- FetchDistributedMapCache
- FetchDropbox
- FetchFile
- FetchFTP
- FetchGCSObject
- FetchGoogleDrive
- FetchGridFS
- FetchS3Object
- FetchSFTP
- FetchSmb
- FilterAttribute
- FlattenJson
- ForkEnrichment
- ForkRecord
- GenerateFlowFile
- GenerateRecord
- GenerateTableFetch
- GeoEnrichIP
- GeoEnrichIPRecord
- GeohashRecord
- GetAsanaObject
- GetAwsPollyJobStatus
- GetAwsTextractJobStatus
- GetAwsTranscribeJobStatus
- GetAwsTranslateJobStatus
- GetAzureEventHub
- GetAzureQueueStorage_v12
- GetDynamoDB
- GetElasticsearch
- GetFile
- GetFTP
- GetGcpVisionAnnotateFilesOperationStatus
- GetGcpVisionAnnotateImagesOperationStatus
- GetHubSpot
- GetMongo
- GetMongoRecord
- GetS3ObjectMetadata
- GetSFTP
- GetShopify
- GetSmbFile
- GetSNMP
- GetSplunk
- GetSQS
- GetWorkdayReport
- GetZendesk
- HandleHttpRequest
- HandleHttpResponse
- IdentifyMimeType
- InvokeHTTP
- InvokeScriptedProcessor
- ISPEnrichIP
- JoinEnrichment
- JoltTransformJSON
- JoltTransformRecord
- JSLTTransformJSON
- JsonQueryElasticsearch
- ListAzureBlobStorage_v12
- ListAzureDataLakeStorage
- ListBoxFile
- ListDatabaseTables
- ListDropbox
- ListenFTP
- ListenHTTP
- ListenOTLP
- ListenSlack
- ListenSyslog
- ListenTCP
- ListenTrapSNMP
- ListenUDP
- ListenUDPRecord
- ListenWebSocket
- ListFile
- ListFTP
- ListGCSBucket
- ListGoogleDrive
- ListS3
- ListSFTP
- ListSmb
- LogAttribute
- LogMessage
- LookupAttribute
- LookupRecord
- MergeContent
- MergeRecord
- ModifyBytes
- ModifyCompression
- MonitorActivity
- MoveAzureDataLakeStorage
- Notify
- PackageFlowFile
- PaginatedJsonQueryElasticsearch
- ParseEvtx
- ParseNetflowv5
- ParseSyslog
- ParseSyslog5424
- PartitionRecord
- PublishAMQP
- PublishGCPubSub
- PublishJMS
- PublishKafka
- PublishMQTT
- PublishSlack
- PutAzureBlobStorage_v12
- PutAzureCosmosDBRecord
- PutAzureDataExplorer
- PutAzureDataLakeStorage
- PutAzureEventHub
- PutAzureQueueStorage_v12
- PutBigQuery
- PutBoxFile
- PutCloudWatchMetric
- PutDatabaseRecord
- PutDistributedMapCache
- PutDropbox
- PutDynamoDB
- PutDynamoDBRecord
- PutElasticsearchJson
- PutElasticsearchRecord
- PutEmail
- PutFile
- PutFTP
- PutGCSObject
- PutGoogleDrive
- PutGridFS
- PutKinesisFirehose
- PutKinesisStream
- PutLambda
- PutMongo
- PutMongoBulkOperations
- PutMongoRecord
- PutRecord
- PutRedisHashRecord
- PutS3Object
- PutSalesforceObject
- PutSFTP
- PutSmbFile
- PutSNS
- PutSplunk
- PutSplunkHTTP
- PutSQL
- PutSQS
- PutSyslog
- PutTCP
- PutUDP
- PutWebSocket
- PutZendeskTicket
- QueryAirtableTable
- QueryAzureDataExplorer
- QueryDatabaseTable
- QueryDatabaseTableRecord
- QueryRecord
- QuerySalesforceObject
- QuerySplunkIndexingStatus
- RemoveRecordField
- RenameRecordField
- ReplaceText
- ReplaceTextWithMapping
- RetryFlowFile
- RouteHL7
- RouteOnAttribute
- RouteOnContent
- RouteText
- RunMongoAggregation
- SampleRecord
- ScanAttribute
- ScanContent
- ScriptedFilterRecord
- ScriptedPartitionRecord
- ScriptedTransformRecord
- ScriptedValidateRecord
- SearchElasticsearch
- SegmentContent
- SendTrapSNMP
- SetSNMP
- SignContentPGP
- SplitAvro
- SplitContent
- SplitExcel
- SplitJson
- SplitPCAP
- SplitRecord
- SplitText
- SplitXml
- StartAwsPollyJob
- StartAwsTextractJob
- StartAwsTranscribeJob
- StartAwsTranslateJob
- StartGcpVisionAnnotateFilesOperation
- StartGcpVisionAnnotateImagesOperation
- TagS3Object
- TailFile
- TransformXml
- UnpackContent
- UpdateAttribute
- UpdateByQueryElasticsearch
- UpdateCounter
- UpdateDatabaseTable
- UpdateRecord
- ValidateCsv
- ValidateJson
- ValidateRecord
- ValidateXml
- VerifyContentMAC
- VerifyContentPGP
- Wait
-
Controller Services
- ADLSCredentialsControllerService
- ADLSCredentialsControllerServiceLookup
- AmazonGlueSchemaRegistry
- ApicurioSchemaRegistry
- AvroReader
- AvroRecordSetWriter
- AvroSchemaRegistry
- AWSCredentialsProviderControllerService
- AzureBlobStorageFileResourceService
- AzureCosmosDBClientService
- AzureDataLakeStorageFileResourceService
- AzureEventHubRecordSink
- AzureStorageCredentialsControllerService_v12
- AzureStorageCredentialsControllerServiceLookup_v12
- CEFReader
- ConfluentEncodedSchemaReferenceReader
- ConfluentEncodedSchemaReferenceWriter
- ConfluentSchemaRegistry
- CSVReader
- CSVRecordLookupService
- CSVRecordSetWriter
- DatabaseRecordLookupService
- DatabaseRecordSink
- DatabaseTableSchemaRegistry
- DBCPConnectionPool
- DBCPConnectionPoolLookup
- DistributedMapCacheLookupService
- ElasticSearchClientServiceImpl
- ElasticSearchLookupService
- ElasticSearchStringLookupService
- EmailRecordSink
- EmbeddedHazelcastCacheManager
- ExcelReader
- ExternalHazelcastCacheManager
- FreeFormTextRecordSetWriter
- GCPCredentialsControllerService
- GCSFileResourceService
- GrokReader
- HazelcastMapCacheClient
- HikariCPConnectionPool
- HttpRecordSink
- IPLookupService
- JettyWebSocketClient
- JettyWebSocketServer
- JMSConnectionFactoryProvider
- JndiJmsConnectionFactoryProvider
- JsonConfigBasedBoxClientService
- JsonPathReader
- JsonRecordSetWriter
- JsonTreeReader
- Kafka3ConnectionService
- KerberosKeytabUserService
- KerberosPasswordUserService
- KerberosTicketCacheUserService
- LoggingRecordSink
- MapCacheClientService
- MapCacheServer
- MongoDBControllerService
- MongoDBLookupService
- PropertiesFileLookupService
- ProtobufReader
- ReaderLookup
- RecordSetWriterLookup
- RecordSinkServiceLookup
- RedisConnectionPoolService
- RedisDistributedMapCacheClientService
- RestLookupService
- S3FileResourceService
- ScriptedLookupService
- ScriptedReader
- ScriptedRecordSetWriter
- ScriptedRecordSink
- SetCacheClientService
- SetCacheServer
- SimpleCsvFileLookupService
- SimpleDatabaseLookupService
- SimpleKeyValueLookupService
- SimpleRedisDistributedMapCacheClientService
- SimpleScriptedLookupService
- SiteToSiteReportingRecordSink
- SlackRecordSink
- SmbjClientProviderService
- StandardAsanaClientProviderService
- StandardAzureCredentialsControllerService
- StandardDropboxCredentialService
- StandardFileResourceService
- StandardHashiCorpVaultClientService
- StandardHttpContextMap
- StandardJsonSchemaRegistry
- StandardKustoIngestService
- StandardKustoQueryService
- StandardOauth2AccessTokenProvider
- StandardPGPPrivateKeyService
- StandardPGPPublicKeyService
- StandardPrivateKeyService
- StandardProxyConfigurationService
- StandardRestrictedSSLContextService
- StandardS3EncryptionService
- StandardSSLContextService
- StandardWebClientServiceProvider
- Syslog5424Reader
- SyslogReader
- UDPEventRecordSink
- VolatileSchemaCache
- WindowsEventLogReader
- XMLFileLookupService
- XMLReader
- XMLRecordSetWriter
- YamlTreeReader
- ZendeskRecordSink
ConsumeAzureEventHub 2.0.0
- Bundle
- org.apache.nifi | nifi-azure-nar
- Description
- Receives messages from Microsoft Azure Event Hubs with checkpointing to ensure consistent event processing. Checkpoint tracking avoids consuming a message multiple times and enables reliable resumption of processing in the event of intermittent network failures. Checkpoint tracking requires external storage and provides the preferred approach to consuming messages from Azure Event Hubs. In clustered environment, ConsumeAzureEventHub processor instances form a consumer group and the messages are distributed among the cluster nodes (each message is processed on one cluster node only).
- Tags
- azure, cloud, eventhub, events, microsoft, streaming, streams
- Input Requirement
- FORBIDDEN
- Supports Sensitive Dynamic Properties
- false
Properties
-
Checkpoint Strategy
Specifies which strategy to use for storing and retrieving partition ownership and checkpoint information for each partition.
- Display Name
- Checkpoint Strategy
- Description
- Specifies which strategy to use for storing and retrieving partition ownership and checkpoint information for each partition.
- API Name
- checkpoint-strategy
- Default Value
- AZURE_BLOB_STORAGE
- Allowable Values
-
- Azure Blob Storage
- Component State
- Expression Language Scope
- Not Supported
- Sensitive
- false
- Required
- true
-
Batch Size
The number of messages to process within a NiFi session. This parameter affects throughput and consistency. NiFi commits its session and Event Hubs checkpoints after processing this number of messages. If NiFi session is committed, but fails to create an Event Hubs checkpoint, then it is possible that the same messages will be received again. The higher number, the higher throughput, but possibly less consistent.
- Display Name
- Batch Size
- Description
- The number of messages to process within a NiFi session. This parameter affects throughput and consistency. NiFi commits its session and Event Hubs checkpoints after processing this number of messages. If NiFi session is committed, but fails to create an Event Hubs checkpoint, then it is possible that the same messages will be received again. The higher number, the higher throughput, but possibly less consistent.
- API Name
- event-hub-batch-size
- Default Value
- 10
- Expression Language Scope
- Environment variables defined at JVM level and system properties
- Sensitive
- false
- Required
- true
-
Consumer Group
The name of the consumer group to use.
- Display Name
- Consumer Group
- Description
- The name of the consumer group to use.
- API Name
- event-hub-consumer-group
- Default Value
- $Default
- Expression Language Scope
- Environment variables defined at JVM level and system properties
- Sensitive
- false
- Required
- true
-
Initial Offset
Specify where to start receiving messages if offset is not yet stored in the checkpoint store.
- Display Name
- Initial Offset
- Description
- Specify where to start receiving messages if offset is not yet stored in the checkpoint store.
- API Name
- event-hub-initial-offset
- Default Value
- end-of-stream
- Allowable Values
-
- Start of stream
- End of stream
- Expression Language Scope
- Not Supported
- Sensitive
- false
- Required
- true
-
Message Receive Timeout
The amount of time this consumer should wait to receive the Batch Size before returning.
- Display Name
- Message Receive Timeout
- Description
- The amount of time this consumer should wait to receive the Batch Size before returning.
- API Name
- event-hub-message-receive-timeout
- Default Value
- 1 min
- Expression Language Scope
- Environment variables defined at JVM level and system properties
- Sensitive
- false
- Required
- true
-
Event Hub Name
The name of the event hub to pull messages from.
- Display Name
- Event Hub Name
- Description
- The name of the event hub to pull messages from.
- API Name
- event-hub-name
- Expression Language Scope
- Environment variables defined at JVM level and system properties
- Sensitive
- false
- Required
- true
-
Event Hub Namespace
The namespace that the Azure Event Hubs is assigned to. This is generally equal to <Event Hub Names>-ns.
- Display Name
- Event Hub Namespace
- Description
- The namespace that the Azure Event Hubs is assigned to. This is generally equal to <Event Hub Names>-ns.
- API Name
- event-hub-namespace
- Expression Language Scope
- Environment variables defined at JVM level and system properties
- Sensitive
- false
- Required
- true
-
Prefetch Count
- Display Name
- Prefetch Count
- Description
- API Name
- event-hub-prefetch-count
- Default Value
- 300
- Expression Language Scope
- Environment variables defined at JVM level and system properties
- Sensitive
- false
- Required
- true
-
Shared Access Policy Name
The name of the shared access policy. This policy must have Listen claims.
- Display Name
- Shared Access Policy Name
- Description
- The name of the shared access policy. This policy must have Listen claims.
- API Name
- event-hub-shared-access-policy-name
- Expression Language Scope
- Environment variables defined at JVM level and system properties
- Sensitive
- false
- Required
- false
-
Shared Access Policy Key
The key of the shared access policy. Either the primary or the secondary key can be used.
- Display Name
- Shared Access Policy Key
- Description
- The key of the shared access policy. Either the primary or the secondary key can be used.
- API Name
- event-hub-shared-access-policy-primary-key
- Expression Language Scope
- Not Supported
- Sensitive
- true
- Required
- false
-
Proxy Configuration Service
Specifies the Proxy Configuration Controller Service to proxy network requests. Supported proxies: HTTP + AuthN
- Display Name
- Proxy Configuration Service
- Description
- Specifies the Proxy Configuration Controller Service to proxy network requests. Supported proxies: HTTP + AuthN
- API Name
- proxy-configuration-service
- Service Interface
- org.apache.nifi.proxy.ProxyConfigurationService
- Service Implementations
- Expression Language Scope
- Not Supported
- Sensitive
- false
- Required
- false
- Dependencies
-
- Transport Type is set to any of [AmqpWebSockets]
-
Record Reader
The Record Reader to use for reading received messages. The event hub name can be referred by Expression Language '${eventhub.name}' to access a schema.
- Display Name
- Record Reader
- Description
- The Record Reader to use for reading received messages. The event hub name can be referred by Expression Language '${eventhub.name}' to access a schema.
- API Name
- record-reader
- Service Interface
- org.apache.nifi.serialization.RecordReaderFactory
- Service Implementations
- Expression Language Scope
- Not Supported
- Sensitive
- false
- Required
- false
-
Record Writer
The Record Writer to use for serializing Records to an output FlowFile. The event hub name can be referred by Expression Language '${eventhub.name}' to access a schema. If not specified, each message will create a FlowFile.
- Display Name
- Record Writer
- Description
- The Record Writer to use for serializing Records to an output FlowFile. The event hub name can be referred by Expression Language '${eventhub.name}' to access a schema. If not specified, each message will create a FlowFile.
- API Name
- record-writer
- Service Interface
- org.apache.nifi.serialization.RecordSetWriterFactory
- Service Implementations
- Expression Language Scope
- Not Supported
- Sensitive
- false
- Required
- false
-
Service Bus Endpoint
To support namespaces not in the default windows.net domain.
- Display Name
- Service Bus Endpoint
- Description
- To support namespaces not in the default windows.net domain.
- API Name
- Service Bus Endpoint
- Default Value
- .servicebus.windows.net
- Allowable Values
-
- Azure
- Azure China
- Azure Germany
- Azure US Government
- Expression Language Scope
- Not Supported
- Sensitive
- false
- Required
- true
-
Storage Account Key
The Azure Storage account key to store event hub consumer group state.
- Display Name
- Storage Account Key
- Description
- The Azure Storage account key to store event hub consumer group state.
- API Name
- storage-account-key
- Expression Language Scope
- Environment variables defined at JVM level and system properties
- Sensitive
- true
- Required
- false
- Dependencies
-
- Checkpoint Strategy is set to any of [AZURE_BLOB_STORAGE]
-
Storage Account Name
Name of the Azure Storage account to store event hub consumer group state.
- Display Name
- Storage Account Name
- Description
- Name of the Azure Storage account to store event hub consumer group state.
- API Name
- storage-account-name
- Expression Language Scope
- Environment variables defined at JVM level and system properties
- Sensitive
- false
- Required
- true
- Dependencies
-
- Checkpoint Strategy is set to any of [AZURE_BLOB_STORAGE]
-
Storage Container Name
Name of the Azure Storage container to store the event hub consumer group state. If not specified, event hub name is used.
- Display Name
- Storage Container Name
- Description
- Name of the Azure Storage container to store the event hub consumer group state. If not specified, event hub name is used.
- API Name
- storage-container-name
- Expression Language Scope
- Environment variables defined at JVM level and system properties
- Sensitive
- false
- Required
- false
- Dependencies
-
- Checkpoint Strategy is set to any of [AZURE_BLOB_STORAGE]
-
Storage SAS Token
The Azure Storage SAS token to store Event Hub consumer group state. Always starts with a ? character.
- Display Name
- Storage SAS Token
- Description
- The Azure Storage SAS token to store Event Hub consumer group state. Always starts with a ? character.
- API Name
- storage-sas-token
- Expression Language Scope
- Environment variables defined at JVM level and system properties
- Sensitive
- true
- Required
- false
- Dependencies
-
- Checkpoint Strategy is set to any of [AZURE_BLOB_STORAGE]
-
Transport Type
Advanced Message Queuing Protocol Transport Type for communication with Azure Event Hubs
- Display Name
- Transport Type
- Description
- Advanced Message Queuing Protocol Transport Type for communication with Azure Event Hubs
- API Name
- Transport Type
- Default Value
- Amqp
- Allowable Values
-
- AMQP
- AMQP_WEB_SOCKETS
- Expression Language Scope
- Not Supported
- Sensitive
- false
- Required
- true
-
Use Azure Managed Identity
Choose whether or not to use the managed identity of Azure VM/VMSS
- Display Name
- Use Azure Managed Identity
- Description
- Choose whether or not to use the managed identity of Azure VM/VMSS
- API Name
- use-managed-identity
- Default Value
- false
- Allowable Values
-
- true
- false
- Expression Language Scope
- Not Supported
- Sensitive
- false
- Required
- true
State Management
| Scopes | Description |
|---|---|
| LOCAL, CLUSTER | Local state is used to store the client id. Cluster state is used to store partition ownership and checkpoint information when component state is configured as the checkpointing strategy. |
Relationships
| Name | Description |
|---|---|
| success | FlowFiles received from Event Hub. |
Writes Attributes
| Name | Description |
|---|---|
| eventhub.enqueued.timestamp | The time (in milliseconds since epoch, UTC) at which the message was enqueued in the event hub |
| eventhub.offset | The offset into the partition at which the message was stored |
| eventhub.sequence | The sequence number associated with the message |
| eventhub.name | The name of the event hub from which the message was pulled |
| eventhub.partition | The name of the partition from which the message was pulled |
| eventhub.property.* | The application properties of this message. IE: 'application' would be 'eventhub.property.application' |