This Property is only considered if the [Initial Value] Property has a value specified.Initial Value Date Time Zone | es-rest-range-time-zone | | | If the "Range Query Field" is a Date field, convert the "Initial Value" to UTC with this time zone. Valid values are ISO 8601 UTC offsets, such as "+01:00" or "-08:00", and IANA time zone IDs, such as "Europe/London".
This Property is only considered if the [Initial Value] Property has a value specified. |
Additional Filters | es-rest-additional-filters | | | One or more query filters in JSON syntax, not Lucene syntax. Ex: [{"match":{"somefield":"somevalue"}}, {"match":{"anotherfield":"anothervalue"}}]. These filters wil be used as part of a Bool query's filter. |
Size | es-rest-size | | | The maximum number of documents to retrieve in the query. If the query is paginated, this "size" applies to each page of the query, not the "size" of the entire result set. Supports Expression Language: true (will be evaluated using Environment variables only) |
Sort | es-rest-query-sort | | | Sort results by one or more fields, in JSON syntax. Ex: [{"price" : {"order" : "asc", "mode" : "avg"}}, {"post_date" : {"format": "strict_date_optional_time_nanos"}}] Supports Expression Language: true (will be evaluated using Environment variables only) |
Aggregations | es-rest-query-aggs | | | One or more query aggregations (or "aggs"), in JSON syntax. Ex: {"items": {"terms": {"field": "product", "size": 10}}} Supports Expression Language: true (will be evaluated using Environment variables only) |
Fields | es-rest-query-fields | | | Fields of indexed documents to be retrieved, in JSON syntax. Ex: ["user.id", "http.response.*", {"field": "@timestamp", "format": "epoch_millis"}] Supports Expression Language: true (will be evaluated using Environment variables only) |
Script Fields | es-rest-query-script-fields | | | Fields to created using script evaluation at query runtime, in JSON syntax. Ex: {"test1": {"script": {"lang": "painless", "source": "doc['price'].value * 2"}}, "test2": {"script": {"lang": "painless", "source": "doc['price'].value * params.factor", "params": {"factor": 2.0}}}} Supports Expression Language: true (will be evaluated using Environment variables only) |
Query Attribute | el-query-attribute | | | If set, the executed query will be set on each result flowfile in the specified attribute. Supports Expression Language: true (will be evaluated using Environment variables only) |
Index | el-rest-fetch-index | | | The name of the index to use. Supports Expression Language: true (will be evaluated using Environment variables only) |
Type | el-rest-type | | | The type of this document (used by Elasticsearch for indexing and searching). Supports Expression Language: true (will be evaluated using Environment variables only) |
Client Service | el-rest-client-service | | Controller Service API: ElasticSearchClientService Implementation: ElasticSearchClientServiceImpl | An Elasticsearch client service to use for running queries. |
Search Results Split | el-rest-split-up-hits | PER_RESPONSE | - PER_HIT
![Flowfile per hit. Flowfile per hit.](../../../../../html/images/iconInfo.png) - PER_RESPONSE
![Flowfile per response. Flowfile per response.](../../../../../html/images/iconInfo.png) - PER_QUERY
![Combine results from all query responses (one flowfile per entire paginated result set of hits). Note that aggregations cannot be paged, they are generated across the entire result set and returned as part of the first page. Results are output with one JSON object per line (allowing hits to be combined from multiple pages without loading all results into memory). Combine results from all query responses (one flowfile per entire paginated result set of hits). Note that aggregations cannot be paged, they are generated across the entire result set and returned as part of the first page. Results are output with one JSON object per line (allowing hits to be combined from multiple pages without loading all results into memory).](../../../../../html/images/iconInfo.png)
| Output a flowfile containing all hits or one flowfile for each individual hit or one flowfile containing all hits from all paged responses. |
Search Results Format | el-rest-format-hits | FULL | - FULL
![Contains full Elasticsearch Hit, including Document Source and Metadata. Contains full Elasticsearch Hit, including Document Source and Metadata.](../../../../../html/images/iconInfo.png) - SOURCE_ONLY
![Document Source only (where present). Document Source only (where present).](../../../../../html/images/iconInfo.png) - METADATA_ONLY
![Hit Metadata only. Hit Metadata only.](../../../../../html/images/iconInfo.png)
| Format of Hits output. |
Aggregation Results Split | el-rest-split-up-aggregations | PER_RESPONSE | - PER_HIT
![Flowfile per hit. Flowfile per hit.](../../../../../html/images/iconInfo.png) - PER_RESPONSE
![Flowfile per response. Flowfile per response.](../../../../../html/images/iconInfo.png)
| Output a flowfile containing all aggregations or one flowfile for each individual aggregation. |
Aggregation Results Format | el-rest-format-aggregations | FULL | - FULL
![Contains full Elasticsearch Aggregation, including Buckets and Metadata. Contains full Elasticsearch Aggregation, including Buckets and Metadata.](../../../../../html/images/iconInfo.png) - BUCKETS_ONLY
![Bucket Content only. Bucket Content only.](../../../../../html/images/iconInfo.png) - METADATA_ONLY
![Aggregation Metadata only. Aggregation Metadata only.](../../../../../html/images/iconInfo.png)
| Format of Aggregation output. |
Output No Hits | el-rest-output-no-hits | false | | Output a "hits" flowfile even if no hits found for query. If true, an empty "hits" flowfile will be output even if "aggregations" are output. |
Pagination Type | el-rest-pagination-type | SCROLL | - SCROLL
![Use Elasticsearch "_scroll" API to page results. Does not accept additional query parameters. Use Elasticsearch "_scroll" API to page results. Does not accept additional query parameters.](../../../../../html/images/iconInfo.png) - SEARCH_AFTER
![Use Elasticsearch "search_after" _search API to page sorted results. Use Elasticsearch "search_after" _search API to page sorted results.](../../../../../html/images/iconInfo.png) - POINT_IN_TIME
![Use Elasticsearch (7.10+ with XPack) "point in time" _search API to page sorted results. Not available for use with AWS OpenSearch. Use Elasticsearch (7.10+ with XPack) "point in time" _search API to page sorted results. Not available for use with AWS OpenSearch.](../../../../../html/images/iconInfo.png)
| Pagination method to use. Not all types are available for all Elasticsearch versions, check the Elasticsearch docs to confirm which are applicable and recommended for your service. |
Pagination Keep Alive | el-rest-pagination-keep-alive | 10 mins | | Pagination "keep_alive" period. Period Elasticsearch will keep the scroll/pit cursor alive in between requests (this is not the time expected for all pages to be returned, but the maximum allowed time for requests between page retrievals). |
Dynamic Properties:
Supports Sensitive Dynamic Properties: No
Dynamic Properties allow the user to specify both the name and value of a property.
Name | Value | Description |
---|
The name of a URL query parameter to add | The value of the URL query parameter | Adds the specified property name/value as a query parameter in the Elasticsearch URL used for processing. These parameters will override any matching parameters in the query request body. For SCROLL type queries, these parameters are only used in the initial (first page) query as the Elasticsearch Scroll API does not support the same query parameters for subsequent pages of data. Supports Expression Language: true (will be evaluated using Environment variables only) |
Relationships:
Name | Description |
---|
aggregations | Aggregations are routed to this relationship. |
hits | Search hits are routed to this relationship. |
Reads Attributes:
None specified.Writes Attributes:
Name | Description |
---|
mime.type | application/json |
page.number | The number of the page (request), starting from 1, in which the results were returned that are in the output flowfile |
hit.count | The number of hits that are in the output flowfile |
elasticsearch.query.error | The error message provided by Elasticsearch if there is an error querying the index. |
State management:
Scope | Description |
---|
CLUSTER | The pagination state (scrollId, searchAfter, pitId, hitCount, pageCount, pageExpirationTimestamp, trackingRangeValue) is retained in between invocations of this processor until the Scroll/PiT has expired (when the current time is later than the last query execution plus the Pagination Keep Alive interval). |
Restricted:
This component is not restricted.Input requirement:
This component does not allow an incoming relationship.System Resource Considerations:
Resource | Description |
---|
MEMORY | Care should be taken on the size of each page because each response from Elasticsearch will be loaded into memory all at once and converted into the resulting flowfiles. |
See Also:
SearchElasticsearch, PaginatedJsonQueryElasticsearch