This processor is deprecated and may be removed in future releases.
Please consider using one the following alternatives: SearchElasticsearch
Scrolls through an Elasticsearch query using the specified connection properties. This processor is intended to be run on the primary node, and is designed for scrolling through huge result sets, as in the case of a reindex. The state must be cleared before another query can be run. Each page of results is returned, wrapped in a JSON object like so: { "hits" : [ <doc1>, <doc2>, <docn> ] }. Note that the full body of each page of documents will be read into memory before being written to a Flow File for transfer.
elasticsearch, query, scroll, read, get, http
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
| Display Name | API Name | Default Value | Allowable Values | Description |
|---|---|---|---|---|
| Elasticsearch URL | elasticsearch-http-url | Elasticsearch URL which will be connected to, including scheme (http, e.g.), host, and port. The default port for the REST API is 9200. Supports Expression Language: true (will be evaluated using variable registry only) | ||
| SSL Context Service | SSL Context Service | Controller Service API: SSLContextService Implementations: StandardSSLContextService StandardRestrictedSSLContextService | The SSL Context Service used to provide client certificate information for TLS/SSL connections. This service only applies if the Elasticsearch endpoint(s) have been secured with TLS/SSL. | |
| Character Set | Character Set | UTF-8 | Specifies the character set of the document data. Supports Expression Language: true (will be evaluated using variable registry only) | |
| Username | Username | Username to access the Elasticsearch cluster Supports Expression Language: true (will be evaluated using variable registry only) | ||
| Password | Password | Password to access the Elasticsearch cluster Sensitive Property: true Supports Expression Language: true (will be evaluated using variable registry only) | ||
| Connection Timeout | elasticsearch-http-connect-timeout | 5 secs | Max wait time for the connection to the Elasticsearch REST API. Supports Expression Language: true (will be evaluated using variable registry only) | |
| Response Timeout | elasticsearch-http-response-timeout | 15 secs | Max wait time for a response from the Elasticsearch REST API. Supports Expression Language: true (will be evaluated using variable registry only) | |
| Proxy Configuration Service | proxy-configuration-service | Controller Service API: ProxyConfigurationService Implementation: StandardProxyConfigurationService | Specifies the Proxy Configuration Controller Service to proxy network requests. If set, it supersedes proxy settings configured per component. Supported proxies: SOCKS, HTTP + AuthN In case of SOCKS, it is not guaranteed that the selected SOCKS Version will be used by the processor. | |
| Proxy Host | elasticsearch-http-proxy-host | The fully qualified hostname or IP address of the proxy server Supports Expression Language: true (will be evaluated using variable registry only) | ||
| Proxy Port | elasticsearch-http-proxy-port | The port of the proxy server Supports Expression Language: true (will be evaluated using variable registry only) | ||
| Proxy Username | proxy-username | Proxy Username Supports Expression Language: true (will be evaluated using variable registry only) | ||
| Proxy Password | proxy-password | Proxy Password Sensitive Property: true Supports Expression Language: true (will be evaluated using variable registry only) | ||
| Query | scroll-es-query | The Lucene-style query to run against ElasticSearch (e.g., genre:blues AND -artist:muddy) Supports Expression Language: true (will be evaluated using variable registry only) | ||
| Scroll Duration | scroll-es-scroll | 1m | The scroll duration is how long each search context is kept in memory. Supports Expression Language: true (will be evaluated using variable registry only) | |
| Page Size | scroll-es-size | 20 | Determines how many documents to return per page during scrolling. Supports Expression Language: true (will be evaluated using variable registry only) | |
| Index | scroll-es-index | The name of the index to read from. If the property is set to _all, the query will match across all indexes. Supports Expression Language: true (will be evaluated using variable registry only) | ||
| Type | scroll-es-type | The type of document (if unset, the query will be against all types in the _index). This should be unset or '_doc' for Elasticsearch 7.0+. Supports Expression Language: true (will be evaluated using variable registry only) | ||
| Fields | scroll-es-fields | A comma-separated list of fields to retrieve from the document. If the Fields property is left blank, then the entire document's source will be retrieved. Supports Expression Language: true (will be evaluated using variable registry only) | ||
| Sort | scroll-es-sort | A sort parameter (e.g., timestamp:asc). If the Sort property is left blank, then the results will be retrieved in document order. Supports Expression Language: true (will be evaluated using variable registry only) |
Supports Sensitive Dynamic Properties: No
Dynamic Properties allow the user to specify both the name and value of a property.
| Name | Value | Description |
|---|---|---|
| A URL query parameter | The value to set it to | Adds the specified property name/value as a query parameter in the Elasticsearch URL used for processing Supports Expression Language: true (will be evaluated using variable registry only) |
| Name | Description |
|---|---|
| success | All FlowFiles that are read from Elasticsearch are routed to this relationship. |
| failure | All FlowFiles that cannot be read from Elasticsearch are routed to this relationship. Note that only incoming flow files will be routed to failure. |
| Name | Description |
|---|---|
| es.index | The Elasticsearch index containing the document |
| es.type | The Elasticsearch document type |
| Scope | Description |
|---|---|
| LOCAL | After each successful scroll page, the latest scroll_id is persisted in scrollId as input for the next scroll call. Once the entire query is complete, finishedQuery state will be set to true, and the processor will not execute unless this is cleared. |