PutElasticsearchRecord

Description:

A record-aware Elasticsearch put processor that uses the official Elastic REST client libraries.

Additional Details...

Tags:

json, elasticsearch, elasticsearch5, elasticsearch6, elasticsearch7, elasticsearch8, put, index, record

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.

Display NameAPI NameDefault ValueAllowable ValuesDescription
Index Operationput-es-record-index-opindexThe type of the operation used to index (create, delete, index, update, upsert)
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Indexel-rest-fetch-indexThe name of the index to use.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Typeel-rest-typeThe type of this document (used by Elasticsearch for indexing and searching).
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
@timestamp Valueput-es-record-at-timestampThe value to use as the @timestamp field (required for Elasticsearch Data Streams)
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Client Serviceel-rest-client-serviceController Service API:
ElasticSearchClientService
Implementation: ElasticSearchClientServiceImpl
An Elasticsearch client service to use for running queries.
Record Readerput-es-record-readerController Service API:
RecordReaderFactory
Implementations: GrokReader
XMLReader
AvroReader
Syslog5424Reader
CEFReader
SyslogReader
JsonTreeReader
CSVReader
ScriptedReader
ParquetReader
JsonPathReader
ReaderLookup
WindowsEventLogReader
The record reader to use for reading incoming records from flowfiles.
Batch Sizeput-es-record-batch-size100The number of records to send over in a single batch.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
ID Record Pathput-es-record-id-pathA record path expression to retrieve the ID field for use with Elasticsearch. If left blank the ID will be automatically generated by Elasticsearch.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Retain ID (Record Path)put-es-record-retain-id-fieldfalse
  • true
  • false
Whether to retain the existing field used as the ID Record Path.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)

This Property is only considered if the [ID Record Path] Property has a value specified.
Index Operation Record Pathput-es-record-index-op-pathA record path expression to retrieve the Index Operation field for use with Elasticsearch. If left blank the Index Operation will be determined using the main Index Operation property.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Index Record Pathput-es-record-index-record-pathA record path expression to retrieve the index field for use with Elasticsearch. If left blank the index will be determined using the main index property.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Type Record Pathput-es-record-type-record-pathA record path expression to retrieve the type field for use with Elasticsearch. If left blank the type will be determined using the main type property.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
@timestamp Record Pathput-es-record-at-timestamp-pathA RecordPath pointing to a field in the record(s) that contains the @timestamp for the document. If left blank the @timestamp will be determined using the main @timestamp property
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Retain @timestamp (Record Path)put-es-record-retain-at-timestamp-fieldfalse
  • true
  • false
Whether to retain the existing field used as the @timestamp Record Path.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)

This Property is only considered if the [@timestamp Record Path] Property has a value specified.
Date Formatput-es-record-at-timestamp-date-formatSpecifies the format to use when writing Date fields. If not specified, the default format 'yyyy-MM-dd' is used. If specified, the value must match the Java Simple Date Format (for example, MM/dd/yyyy for a two-digit month, followed by a two-digit day, followed by a four-digit year, all separated by '/' characters, as in 01/25/2017).
Supports Expression Language: true (will be evaluated using variable registry only)
Time Formatput-es-record-at-timestamp-time-formatSpecifies the format to use when writing Time fields. If not specified, the default format 'HH:mm:ss' is used. If specified, the value must match the Java Simple Date Format (for example, HH:mm:ss for a two-digit hour in 24-hour format, followed by a two-digit minute, followed by a two-digit second, all separated by ':' characters, as in 18:04:15).
Supports Expression Language: true (will be evaluated using variable registry only)
Timestamp Formatput-es-record-at-timestamp-timestamp-formatSpecifies the format to use when writing Timestamp fields. If not specified, the default format 'yyyy-MM-dd HH:mm:ss' is used. If specified, the value must match the Java Simple Date Format (for example, MM/dd/yyyy HH:mm:ss for a two-digit month, followed by a two-digit day, followed by a four-digit year, all separated by '/' characters; and then followed by a two-digit hour in 24-hour format, followed by a two-digit minute, followed by a two-digit second, all separated by ':' characters, as in 01/25/2017 18:04:15).
Supports Expression Language: true (will be evaluated using variable registry only)
Log Error Responsesput-es-record-log-error-responsesfalse
  • true
  • false
If this is enabled, errors will be logged to the NiFi logs at the error log level. Otherwise, they will only be logged if debug logging is enabled on NiFi as a whole. The purpose of this option is to give the user the ability to debug failed operations without having to turn on debug logging.
Result Record Writerput-es-record-error-writerController Service API:
RecordSetWriterFactory
Implementations: FreeFormTextRecordSetWriter
AvroRecordSetWriter
ScriptedRecordSetWriter
CSVRecordSetWriter
XMLRecordSetWriter
ParquetRecordSetWriter
JsonRecordSetWriter
RecordSetWriterLookup
If this configuration property is set, the response from Elasticsearch will be examined for failed records and the failed records will be written to a record set with this record writer service and sent to the "errors" relationship. Successful records will be written to a record setwith this record writer service and sent to the "successful_records" relationship.
Treat "Not Found" as Errorput-es-record-not_found-is-errortrue
  • true
  • false
If true, "not_found" Elasticsearch Document associated Records will be routed to the "successful_records" relationship, otherwise to the "errors" relationship.

This Property is only considered if the [Result Record Writer] Property has a value specified.

Dynamic Properties:

Supports Sensitive Dynamic Properties: No

Dynamic Properties allow the user to specify both the name and value of a property.

NameValueDescription
The name of a URL query parameter to addThe value of the URL query parameterAdds the specified property name/value as a query parameter in the Elasticsearch URL used for processing. These parameters will override any matching parameters in the _bulk request body
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)

Relationships:

NameDescription
retryAll flowfiles that fail due to server/cluster availability go to this relationship.
successAll flowfiles that succeed in being transferred into Elasticsearch go here. Documents received by the Elasticsearch _bulk API may still result in errors on the Elasticsearch side. The Elasticsearch response will need to be examined to determine whether any Document(s)/Record(s) resulted in errors.
successful_recordsIf a "Result Record Writer" is set, any Record(s) corresponding to Elasticsearch document(s) that did not result in an "error" (within Elasticsearch) will be routed here.
failureAll flowfiles that fail for reasons unrelated to server availability go to this relationship.
errorsIf a "Result Record Writer" is set, any Record(s) corresponding to Elasticsearch document(s) that resulted in an "error" (within Elasticsearch) will be routed here.

Reads Attributes:

None specified.

Writes Attributes:

NameDescription
elasticsearch.put.errorThe error message provided by Elasticsearch if there is an error indexing the documents.
elasticsearch.put.error.countThe number of records that generated errors in the Elasticsearch _bulk API.
elasticsearch.put.success.countThe number of records that were successfully processed by the Elasticsearch _bulk API.

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component requires an incoming relationship.

System Resource Considerations:

ResourceDescription
MEMORYThe Batch of Records will be stored in memory until the bulk operation is performed.