ConsumeAzureEventHub

Description:

Receives messages from Azure Event Hubs, writing the contents of the message to the content of the FlowFile.

Tags:

azure, microsoft, cloud, eventhub, events, streaming, streams

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, whether a property supports the NiFi Expression Language, and whether a property is considered "sensitive", meaning that its value will be encrypted. Before entering a value in a sensitive property, ensure that the nifi.properties file has an entry for the property nifi.sensitive.props.key.

NameDefault ValueAllowable ValuesDescription
Event Hub NamespaceThe namespace that the Azure Event Hubs is assigned to. This is generally equal to <Event Hub Names>-ns.
Supports Expression Language: true (will be evaluated using variable registry only)
Event Hub NameThe name of the event hub to pull messages from.
Supports Expression Language: true (will be evaluated using variable registry only)
Shared Access Policy NameThe name of the shared access policy. This policy must have Listen claims.
Supports Expression Language: true (will be evaluated using variable registry only)
Shared Access Policy Primary KeyThe primary key of the shared access policy
Sensitive Property: true
Use Azure Managed Identityfalse
  • true
  • false
Choose whether or not to use the managed identity of Azure VM/VMSS
Consumer Group$DefaultThe name of the consumer group to use.
Supports Expression Language: true (will be evaluated using variable registry only)
Consumer HostnameThe hostname of this event hub consumer instance. If not specified, an unique identifier is generated in 'nifi-<UUID>' format.
Supports Expression Language: true (will be evaluated using variable registry only)
Record ReaderController Service API:
RecordReaderFactory
Implementations: ScriptedReader
XMLReader
GrokReader
AvroReader
WindowsEventLogReader
JsonPathReader
ReaderLookup
CSVReader
Syslog5424Reader
SyslogReader
JsonTreeReader
ParquetReader
The Record Reader to use for reading received messages. The event hub name can be referred by Expression Language '${eventhub.name}' to access a schema.
Record WriterController Service API:
RecordSetWriterFactory
Implementations: FreeFormTextRecordSetWriter
AvroRecordSetWriter
ParquetRecordSetWriter
ScriptedRecordSetWriter
JsonRecordSetWriter
RecordSetWriterLookup
CSVRecordSetWriter
XMLRecordSetWriter
The Record Writer to use for serializing Records to an output FlowFile. The event hub name can be referred by Expression Language '${eventhub.name}' to access a schema. If not specified, each message will create a FlowFile.
Initial Offsetend-of-stream
  • Start of stream Read from the oldest message retained in the stream.
  • End of stream Ignore old retained messages even if exist, start reading new ones from now.
Specify where to start receiving messages if offset is not stored in Azure Storage.
Prefetch Count300No Description Provided.
Supports Expression Language: true (will be evaluated using variable registry only)
Batch Size10The number of messages to process within a NiFi session. This parameter affects throughput and consistency. NiFi commits its session and Event Hubs checkpoints after processing this number of messages. If NiFi session is committed, but fails to create an Event Hubs checkpoint, then it is possible that the same messages will be received again. The higher number, the higher throughput, but possibly less consistent.
Supports Expression Language: true (will be evaluated using variable registry only)
Message Receive Timeout1 minThe amount of time this consumer should wait to receive the Prefetch Count before returning.
Supports Expression Language: true (will be evaluated using variable registry only)
Storage Account NameName of the Azure Storage account to store event hub consumer group state.
Supports Expression Language: true (will be evaluated using variable registry only)
Storage Account KeyThe Azure Storage account key to store event hub consumer group state.
Sensitive Property: true
Supports Expression Language: true (will be evaluated using variable registry only)
Storage Container NameName of the Azure Storage container to store the event hub consumer group state. If not specified, event hub name is used.
Supports Expression Language: true (will be evaluated using variable registry only)

Relationships:

NameDescription
successFlowFiles received from Event Hub.
parse.failureIf a message from event hub cannot be parsed using the configured Record Reader or failed to be written by the configured Record Writer, the contents of the message will be routed to this Relationship as its own individual FlowFile.

Reads Attributes:

None specified.

Writes Attributes:

NameDescription
eventhub.enqueued.timestampThe time (in milliseconds since epoch, UTC) at which the message was enqueued in the event hub
eventhub.offsetThe offset into the partition at which the message was stored
eventhub.sequenceThe sequence number associated with the message
eventhub.nameThe name of the event hub from which the message was pulled
eventhub.partitionThe name of the partition from which the message was pulled
eventhub.property.*The application properties of this message. IE: 'application' would be 'eventhub.property.application'

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component does not allow an incoming relationship.

System Resource Considerations:

None specified.