ConsumeAzureEventHub

Description:

Receives messages from a Microsoft Azure Event Hub, writing the contents of the Azure message to the content of the FlowFile.

Tags:

azure, microsoft, cloud, eventhub, events, streaming, streams

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, whether a property supports the NiFi Expression Language, and whether a property is considered "sensitive", meaning that its value will be encrypted. Before entering a value in a sensitive property, ensure that the nifi.properties file has an entry for the property nifi.sensitive.props.key.

NameDefault ValueAllowable ValuesDescription
Event Hub NamespaceThe Azure Namespace that the Event Hub is assigned to. This is generally equal to <Event Hub Name>-ns.
Supports Expression Language: true (will be evaluated using variable registry only)
Event Hub NameThe name of the Azure Event Hub to pull messages from.
Supports Expression Language: true (will be evaluated using variable registry only)
Shared Access Policy NameThe name of the Event Hub Shared Access Policy. This Policy must have Listen permissions.
Supports Expression Language: true (will be evaluated using variable registry only)
Shared Access Policy Primary KeyThe primary key of the Event Hub Shared Access Policy.
Sensitive Property: true
Supports Expression Language: true (will be evaluated using variable registry only)
Event Hub Consumer Group$DefaultThe name of the Event Hub Consumer Group to use.
Supports Expression Language: true (will be evaluated using variable registry only)
Event Hub Consumer HostnameThe hostname of this Event Hub Consumer instance. If not specified, an unique identifier is generated in 'nifi-<UUID>' format.
Supports Expression Language: true (will be evaluated using variable registry only)
Record ReaderController Service API:
RecordReaderFactory
Implementations: Syslog5424Reader
AvroReader
JsonPathReader
ScriptedReader
XMLReader
GrokReader
JsonTreeReader
SyslogReader
CSVReader
The Record Reader to use for reading received messages. The Event Hub name can be referred by Expression Language '${eventhub.name}' to access a schema.
Record WriterController Service API:
RecordSetWriterFactory
Implementations: ScriptedRecordSetWriter
JsonRecordSetWriter
CSVRecordSetWriter
AvroRecordSetWriter
XMLRecordSetWriter
FreeFormTextRecordSetWriter
The Record Writer to use for serializing Records to an output FlowFile. The Event Hub name can be referred by Expression Language '${eventhub.name}' to access a schema. If not specified, each message will create a FlowFile.
Initial Offsetend-of-stream
  • Start of stream Read from the oldest message retained in the stream.
  • End of stream Ignore old retained messages even if exist, start reading new ones from now.
Specify where to start receiving messages if offset is not stored in Azure Storage.
Prefetch Count300No Description Provided.
Supports Expression Language: true (will be evaluated using variable registry only)
Batch Size10The number of messages to process within a NiFi session. This parameter affects throughput and consistency. NiFi commits its session and Event Hub checkpoint after processing this number of messages. If NiFi session is committed, but failed to create an Event Hub checkpoint, then it is possible that the same messages to be retrieved again. The higher number, the higher throughput, but possibly less consistent.
Supports Expression Language: true (will be evaluated using variable registry only)
Message Receive Timeout1 minThe amount of time this consumer should wait to receive the Prefetch Count before returning.
Supports Expression Language: true (will be evaluated using variable registry only)
Storage Account NameName of the Azure Storage account to store Event Hub Consumer Group state.
Supports Expression Language: true (will be evaluated using variable registry only)
Storage Account KeyThe Azure Storage account key to store Event Hub Consumer Group state.
Sensitive Property: true
Supports Expression Language: true (will be evaluated using variable registry only)
Storage Container NameName of the Azure Storage Container to store Event Hub Consumer Group state. If not specified, Event Hub name is used.
Supports Expression Language: true (will be evaluated using variable registry only)

Relationships:

NameDescription
successFlowFiles received from Event Hub.
parse.failureIf a message from Event Hub cannot be parsed using the configured Record Reader or failed to be written by the configured Record Writer, the contents of the message will be routed to this Relationship as its own individual FlowFile.

Reads Attributes:

None specified.

Writes Attributes:

NameDescription
eventhub.enqueued.timestampThe time (in milliseconds since epoch, UTC) at which the message was enqueued in the Azure Event Hub
eventhub.offsetThe offset into the partition at which the message was stored
eventhub.sequenceThe Azure Sequence number associated with the message
eventhub.nameThe name of the Event Hub from which the message was pulled
eventhub.partitionThe name of the Azure Partition from which the message was pulled

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component does not allow an incoming relationship.

System Resource Considerations:

None specified.