ListenTCPRecord

Description:

Listens for incoming TCP connections and reads data from each connection using a configured record reader, and writes the records to a flow file using a configured record writer. The type of record reader selected will determine how clients are expected to send data. For example, when using a Grok reader to read logs, a client can keep an open connection and continuously stream data, but when using an JSON reader, the client cannot send an array of JSON documents and then send another array on the same connection, as the reader would be in a bad state at that point. Records will be read from the connection in blocking mode, and will timeout according to the Read Timeout specified in the processor. If the read times out, or if any other error is encountered when reading, the connection will be closed, and any records read up to that point will be handled according to the configured Read Error Strategy (Discard or Transfer). In cases where clients are keeping a connection open, the concurrent tasks for the processor should be adjusted to match the Max Number of TCP Connections allowed, so that there is a task processing each connection. The processor can be configured to use an SSL Context Service to only allow secure connections. When connected clients present certificates for mutual TLS authentication, the Distinguished Names of the client certificate's issuer and subject are added to the outgoing FlowFiles as attributes. The processor does not perform authorization based on Distinguished Name values, but since these values are attached to the outgoing FlowFiles, authorization can be implemented based on these attributes.

Tags:

listen, tcp, record, tls, ssl

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.

Display NameAPI NameDefault ValueAllowable ValuesDescription
Local Network InterfaceLocal Network InterfaceThe name of a local network interface to be used to restrict listening to a specific LAN.
Supports Expression Language: true (will be evaluated using variable registry only)
PortportThe port to listen on for communication.
Supports Expression Language: true (will be evaluated using variable registry only)
Max Size of Socket Buffermax-size-socket-buffer1 MBThe maximum size of the socket buffer that should be used. This is a suggestion to the Operating System to indicate how big the socket buffer should be. If this value is set too low, the buffer may fill up before the data can be read, and incoming data will be dropped.
Max Number of TCP Connectionsmax-number-tcp-connections2The maximum number of concurrent TCP connections to accept. In cases where clients are keeping a connection open, the concurrent tasks for the processor should be adjusted to match the Max Number of TCP Connections allowed, so that there is a task processing each connection.
Read Timeoutread-timeout10 secondsThe amount of time to wait before timing out when reading from a connection.
Record Readerrecord-readerController Service API:
RecordReaderFactory
Implementations: GrokReader
JsonTreeReader
WindowsEventLogReader
ReaderLookup
ParquetReader
CSVReader
Syslog5424Reader
ExcelReader
CEFReader
XMLReader
ScriptedReader
SyslogReader
JsonPathReader
AvroReader
YamlTreeReader
The Record Reader to use for incoming FlowFiles
Record Writerrecord-writerController Service API:
RecordSetWriterFactory
Implementations: FreeFormTextRecordSetWriter
CSVRecordSetWriter
ParquetRecordSetWriter
RecordSetWriterLookup
ScriptedRecordSetWriter
XMLRecordSetWriter
JsonRecordSetWriter
AvroRecordSetWriter
The Record Writer to use in order to serialize the data before writing to a FlowFile
Read Error Strategyreader-error-handling-strategyTransfer
  • Transfer Transfers any records already received and closes the connection.
  • Discard Discards any records already received and closes the connection.
Indicates how to deal with an error while reading the next record from a connection, when previous records have already been read from the connection.
Record Batch Sizerecord-batch-size1000The maximum number of records to write to a single FlowFile.
SSL Context Servicessl-context-serviceController Service API:
RestrictedSSLContextService
Implementation: StandardRestrictedSSLContextService
The Controller Service to use in order to obtain an SSL Context. If this property is set, messages will be received over a secure connection.
Client Authclient-authREQUIRED
  • WANT
  • REQUIRED
  • NONE
The client authentication policy to use for the SSL Context. Only used if an SSL Context Service is provided.

Relationships:

NameDescription
successMessages received successfully will be sent out this relationship.

Reads Attributes:

None specified.

Writes Attributes:

NameDescription
tcp.senderThe host that sent the data.
tcp.portThe port that the processor accepted the connection on.
record.countThe number of records written to the flow file.
mime.typeThe mime-type of the writer used to write the records to the flow file.
client.certificate.issuer.dnFor connections using mutual TLS, the Distinguished Name of the Certificate Authority that issued the client's certificate is attached to the FlowFile.
client.certificate.subject.dnFor connections using mutual TLS, the Distinguished Name of the client certificate's owner (subject) is attached to the FlowFile.

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component does not allow an incoming relationship.

System Resource Considerations:

None specified.