JsonQueryElasticsearch

Description:

A processor that allows the user to run a query (with aggregations) written with the Elasticsearch JSON DSL. It does not automatically paginate queries for the user. If an incoming relationship is added to this processor, it will use the flowfile's content for the query. Care should be taken on the size of the query because the entire response from Elasticsearch will be loaded into memory all at once and converted into the resulting flowfiles.

Additional Details...

Tags:

elasticsearch, elasticsearch5, elasticsearch6, elasticsearch7, elasticsearch8, query, read, get, json

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.

Display NameAPI NameDefault ValueAllowable ValuesDescription
Query Definition Styleel-rest-query-definition-styleFULL_QUERY
  • FULL_QUERY Provide the full Query.
  • BUILD_QUERY Build the Query from separate JSON objects.
How the JSON Query will be defined for use by the processor.
Queryel-rest-queryA query in JSON syntax, not Lucene syntax. Ex: {"query":{"match":{"somefield":"somevalue"}}}. If this parameter is not set, the query will be read from the flowfile content. If the query (property and flowfile content) is empty, a default empty JSON Object will be used, which will result in a "match_all" query in Elasticsearch.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)

This Property is only considered if the [Query Definition Style] Property has a value of "FULL_QUERY".
Query Clauseel-rest-query-clauseA "query" clause in JSON syntax, not Lucene syntax. Ex: {"match":{"somefield":"somevalue"}}. If the query is empty, a default JSON Object will be used, which will result in a "match_all" query in Elasticsearch.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)

This Property is only considered if the [Query Definition Style] Property has a value of "BUILD_QUERY".
Sizees-rest-sizeThe maximum number of documents to retrieve in the query. If the query is paginated, this "size" applies to each page of the query, not the "size" of the entire result set.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)

This Property is only considered if the [Query Definition Style] Property has a value of "BUILD_QUERY".
Sortes-rest-query-sortSort results by one or more fields, in JSON syntax. Ex: [{"price" : {"order" : "asc", "mode" : "avg"}}, {"post_date" : {"format": "strict_date_optional_time_nanos"}}]
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)

This Property is only considered if the [Query Definition Style] Property has a value of "BUILD_QUERY".
Aggregationses-rest-query-aggsOne or more query aggregations (or "aggs"), in JSON syntax. Ex: {"items": {"terms": {"field": "product", "size": 10}}}
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)

This Property is only considered if the [Query Definition Style] Property has a value of "BUILD_QUERY".
Fieldses-rest-query-fieldsFields of indexed documents to be retrieved, in JSON syntax. Ex: ["user.id", "http.response.*", {"field": "@timestamp", "format": "epoch_millis"}]
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)

This Property is only considered if the [Query Definition Style] Property has a value of "BUILD_QUERY".
Script Fieldses-rest-query-script-fieldsFields to created using script evaluation at query runtime, in JSON syntax. Ex: {"test1": {"script": {"lang": "painless", "source": "doc['price'].value * 2"}}, "test2": {"script": {"lang": "painless", "source": "doc['price'].value * params.factor", "params": {"factor": 2.0}}}}
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)

This Property is only considered if the [Query Definition Style] Property has a value of "BUILD_QUERY".
Query Attributeel-query-attributeIf set, the executed query will be set on each result flowfile in the specified attribute.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Indexel-rest-fetch-indexThe name of the index to use.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Typeel-rest-typeThe type of this document (used by Elasticsearch for indexing and searching).
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Client Serviceel-rest-client-serviceController Service API:
ElasticSearchClientService
Implementation: ElasticSearchClientServiceImpl
An Elasticsearch client service to use for running queries.
Search Results Splitel-rest-split-up-hitsPER_RESPONSE
  • PER_HIT Flowfile per hit.
  • PER_RESPONSE Flowfile per response.
Output a flowfile containing all hits or one flowfile for each individual hit.
Search Results Formatel-rest-format-hitsFULL
  • FULL Contains full Elasticsearch Hit, including Document Source and Metadata.
  • SOURCE_ONLY Document Source only (where present).
  • METADATA_ONLY Hit Metadata only.
Format of Hits output.
Aggregation Results Splitel-rest-split-up-aggregationsPER_RESPONSE
  • PER_HIT Flowfile per hit.
  • PER_RESPONSE Flowfile per response.
Output a flowfile containing all aggregations or one flowfile for each individual aggregation.
Aggregation Results Formatel-rest-format-aggregationsFULL
  • FULL Contains full Elasticsearch Aggregation, including Buckets and Metadata.
  • BUCKETS_ONLY Bucket Content only.
  • METADATA_ONLY Aggregation Metadata only.
Format of Aggregation output.
Output No Hitsel-rest-output-no-hitsfalse
  • true
  • false
Output a "hits" flowfile even if no hits found for query. If true, an empty "hits" flowfile will be output even if "aggregations" are output.

Dynamic Properties:

Supports Sensitive Dynamic Properties: No

Dynamic Properties allow the user to specify both the name and value of a property.

NameValueDescription
The name of a URL query parameter to addThe value of the URL query parameterAdds the specified property name/value as a query parameter in the Elasticsearch URL used for processing. These parameters will override any matching parameters in the query request body
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)

Relationships:

NameDescription
aggregationsAggregations are routed to this relationship.
failureAll flowfiles that fail for reasons unrelated to server availability go to this relationship.
originalAll original flowfiles that don't cause an error to occur go to this relationship.
hitsSearch hits are routed to this relationship.

Reads Attributes:

None specified.

Writes Attributes:

NameDescription
mime.typeapplication/json
aggregation.nameThe name of the aggregation whose results are in the output flowfile
aggregation.numberThe number of the aggregation whose results are in the output flowfile
hit.countThe number of hits that are in the output flowfile
elasticsearch.query.errorThe error message provided by Elasticsearch if there is an error querying the index.

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component allows an incoming relationship.

System Resource Considerations:

None specified.

See Also:

PaginatedJsonQueryElasticsearch