PublishGCPubSub

Description:

Publishes the content of the incoming flowfile to the configured Google Cloud PubSub topic. The processor supports dynamic properties. If any dynamic properties are present, they will be sent along with the message in the form of 'attributes'.

Tags:

google, google-cloud, gcp, message, pubsub, publish

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.

Display NameAPI NameDefault ValueAllowable ValuesDescription
Project IDgcp-project-idGoogle Cloud Project ID
Supports Expression Language: true (will be evaluated using variable registry only)
Proxy Configuration Serviceproxy-configuration-serviceController Service API:
ProxyConfigurationService
Implementation: StandardProxyConfigurationService
Specifies the Proxy Configuration Controller Service to proxy network requests. If set, it supersedes proxy settings configured per component. Supported proxies: HTTP + AuthN
GCP Credentials Provider ServiceGCP Credentials Provider ServiceController Service API:
GCPCredentialsService
Implementation: GCPCredentialsControllerService
The Controller Service used to obtain Google Cloud Platform credentials.
Input Batch SizeInput Batch Size100Maximum number of FlowFiles processed for each Processor invocation
Message Derivation StrategyMessage Derivation StrategyFlowFile Oriented
  • FlowFile Oriented Each incoming FlowFile is sent as a Google Cloud PubSub message
  • Record Oriented Each incoming FlowFile is parsed into NiFi records, which are each sent as a Google Cloud PubSub message
The strategy used to publish the incoming FlowFile to the Google Cloud PubSub endpoint.
Record ReaderRecord ReaderController Service API:
RecordReaderFactory
Implementations: GrokReader
JsonTreeReader
WindowsEventLogReader
ReaderLookup
ParquetReader
CSVReader
Syslog5424Reader
ExcelReader
CEFReader
XMLReader
ScriptedReader
SyslogReader
JsonPathReader
AvroReader
YamlTreeReader
The Record Reader to use for incoming FlowFiles

This Property is only considered if the [Message Derivation Strategy] Property has a value of "Record Oriented".
Record WriterRecord WriterController Service API:
RecordSetWriterFactory
Implementations: FreeFormTextRecordSetWriter
CSVRecordSetWriter
ParquetRecordSetWriter
RecordSetWriterLookup
ScriptedRecordSetWriter
XMLRecordSetWriter
JsonRecordSetWriter
AvroRecordSetWriter
The Record Writer to use in order to serialize the data before sending to GCPubSub endpoint

This Property is only considered if the [Message Derivation Strategy] Property has a value of "Record Oriented".
Maximum Message SizeMaximum Message Size1 MBThe maximum size of a Google PubSub message in bytes. Defaults to 1 MB (1048576 bytes)

This Property is only considered if the [Message Derivation Strategy] Property has a value of "FlowFile Oriented".
Topic Namegcp-pubsub-topicName of the Google Cloud PubSub Topic
Supports Expression Language: true (will be evaluated using variable registry only)
Batch Size Thresholdgcp-pubsub-publish-batch-size15Indicates the number of messages the cloud service should bundle together in a batch. If not set and left empty, only one message will be used in a batch
Batch Bytes Thresholdgcp-batch-bytes3 MBPublish request gets triggered based on this Batch Bytes Threshold property and the Batch Size Threshold property, whichever condition is met first.
Supports Expression Language: true (will be evaluated using variable registry only)
Batch Delay Thresholdgcp-pubsub-publish-batch-delay100 millisIndicates the delay threshold to use for batching. After this amount of time has elapsed (counting from the first element added), the elements will be wrapped up in a batch and sent. This value should not be set too high, usually on the order of milliseconds. Otherwise, calls might appear to never complete.
API Endpointapi-endpointpubsub.googleapis.com:443Override the gRPC endpoint in the form of [host:port]
Supports Expression Language: true (will be evaluated using variable registry only)

Dynamic Properties:

Supports Sensitive Dynamic Properties: No

Dynamic Properties allow the user to specify both the name and value of a property.

NameValueDescription
Attribute nameValue to be set to the attributeAttributes to be set for the outgoing Google Cloud PubSub message
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)

Relationships:

NameDescription
retryFlowFiles are routed to this relationship if the Google Cloud Pub/Sub operation fails but attempting the operation again may succeed.
successFlowFiles are routed to this relationship after a successful Google Cloud Pub/Sub operation.
failureFlowFiles are routed to this relationship if the Google Cloud Pub/Sub operation fails.

Reads Attributes:

None specified.

Writes Attributes:

NameDescription
gcp.pubsub.messageIdID of the pubsub message published to the configured Google Cloud PubSub topic
gcp.pubsub.count.recordsCount of pubsub messages published to the configured Google Cloud PubSub topic
gcp.pubsub.topicName of the Google Cloud PubSub topic the message was published to

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component requires an incoming relationship.

System Resource Considerations:

ResourceDescription
MEMORYThe entirety of the FlowFile's content will be read into memory to be sent as a PubSub message.

See Also:

ConsumeGCPubSub