PutElasticsearchHttpRecord

Description:

Writes the records from a FlowFile into to Elasticsearch, using the specified parameters such as the index to insert into and the type of the document, as well as the operation type (index, upsert, delete, etc.). Note: The Bulk API is used to send the records. This means that the entire contents of the incoming flow file are read into memory, and each record is transformed into a JSON document which is added to a single HTTP request body. For very large flow files (files with a large number of records, e.g.), this could cause memory usage issues.

Tags:

elasticsearch, insert, update, upsert, delete, write, put, http, record

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, whether a property supports the NiFi Expression Language, and whether a property is considered "sensitive", meaning that its value will be encrypted. Before entering a value in a sensitive property, ensure that the nifi.properties file has an entry for the property nifi.sensitive.props.key.

NameDefault ValueAllowable ValuesDescription
Elasticsearch URLElasticsearch URL which will be connected to, including scheme (http, e.g.), host, and port. The default port for the REST API is 9200.
Supports Expression Language: true (will be evaluated using variable registry only)
SSL Context ServiceController Service API:
SSLContextService
Implementations: StandardRestrictedSSLContextService
StandardSSLContextService
The SSL Context Service used to provide client certificate information for TLS/SSL connections. This service only applies if the Elasticsearch endpoint(s) have been secured with TLS/SSL.
UsernameUsername to access the Elasticsearch cluster
Supports Expression Language: true (will be evaluated using variable registry only)
PasswordPassword to access the Elasticsearch cluster
Sensitive Property: true
Supports Expression Language: true (will be evaluated using variable registry only)
Connection Timeout5 secsMax wait time for the connection to the Elasticsearch REST API.
Supports Expression Language: true (will be evaluated using variable registry only)
Response Timeout15 secsMax wait time for a response from the Elasticsearch REST API.
Supports Expression Language: true (will be evaluated using variable registry only)
Proxy Configuration ServiceController Service API:
ProxyConfigurationService
Implementation: StandardProxyConfigurationService
Specifies the Proxy Configuration Controller Service to proxy network requests. If set, it supersedes proxy settings configured per component. Supported proxies: SOCKS, HTTP + AuthN
Proxy HostThe fully qualified hostname or IP address of the proxy server
Supports Expression Language: true (will be evaluated using variable registry only)
Proxy PortThe port of the proxy server
Supports Expression Language: true (will be evaluated using variable registry only)
Proxy UsernameProxy Username
Supports Expression Language: true (will be evaluated using variable registry only)
Proxy PasswordProxy Password
Sensitive Property: true
Supports Expression Language: true (will be evaluated using variable registry only)
Record ReaderController Service API:
RecordReaderFactory
Implementations: ScriptedReader
AvroReader
SyslogReader
JsonPathReader
Syslog5424Reader
CSVReader
XMLReader
JsonTreeReader
GrokReader
Specifies the Controller Service to use for parsing incoming data and determining the data's schema.
Identifier Record PathA RecordPath pointing to a field in the record(s) that contains the identifier for the document. If the Index Operation is "index", this property may be left empty or evaluate to an empty value, in which case the document's identifier will be auto-generated by Elasticsearch. For all other Index Operations, the field's value must be non-empty.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
IndexThe name of the index to insert into
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
TypeThe type of this document (used by Elasticsearch for indexing and searching)
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Character SetUTF-8Specifies the character set of the document data.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Index OperationindexThe type of the operation used to index (index, update, upsert, delete)
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Suppress Null Valuesnever-suppress
  • Never Suppress Fields that are missing (present in the schema but not in the record), or that have a value of null, will be written out as a null value
  • Always Suppress Fields that are missing (present in the schema but not in the record), or that have a value of null, will not be written out
  • Suppress Missing Values When a field has a value of null, it will be written out. However, if a field is defined in the schema and not present in the record, the field will not be written out.
Specifies how the writer should handle a null field
Date FormatSpecifies the format to use when reading/writing Date fields. If not specified, the default format 'yyyy-MM-dd' is used. If specified, the value must match the Java Simple Date Format (for example, MM/dd/yyyy for a two-digit month, followed by a two-digit day, followed by a four-digit year, all separated by '/' characters, as in 01/01/2017).
Supports Expression Language: true (will be evaluated using variable registry only)
Time FormatSpecifies the format to use when reading/writing Time fields. If not specified, the default format 'HH:mm:ss' is used. If specified, the value must match the Java Simple Date Format (for example, HH:mm:ss for a two-digit hour in 24-hour format, followed by a two-digit minute, followed by a two-digit second, all separated by ':' characters, as in 18:04:15).
Supports Expression Language: true (will be evaluated using variable registry only)
Timestamp FormatSpecifies the format to use when reading/writing Timestamp fields. If not specified, the default format 'yyyy-MM-dd HH:mm:ss' is used. If specified, the value must match the Java Simple Date Format (for example, MM/dd/yyyy HH:mm:ss for a two-digit month, followed by a two-digit day, followed by a four-digit year, all separated by '/' characters; and then followed by a two-digit hour in 24-hour format, followed by a two-digit minute, followed by a two-digit second, all separated by ':' characters, as in 01/01/2017 18:04:15).
Supports Expression Language: true (will be evaluated using variable registry only)

Dynamic Properties:

Dynamic Properties allow the user to specify both the name and value of a property.

NameValueDescription
A URL query parameterThe value to set it toAdds the specified property name/value as a query parameter in the Elasticsearch URL used for processing
Supports Expression Language: true (will be evaluated using variable registry only)

Relationships:

NameDescription
retryA FlowFile is routed to this relationship if the database cannot be updated but attempting the operation again may succeed
successAll FlowFiles that are written to Elasticsearch are routed to this relationship
failureAll FlowFiles that cannot be written to Elasticsearch are routed to this relationship

Reads Attributes:

None specified.

Writes Attributes:

NameDescription
record.countThe number of records in an outgoing FlowFile. This is only populated on the 'success' relationship.
failure.countThe number of records found by Elasticsearch to have errors. This is only populated on the 'failure' relationship.

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component requires an incoming relationship.

System Resource Considerations:

None specified.