DruidTranquilityController

Description:

Asynchronously sends flowfiles to Druid Indexing Task using Tranquility API. If aggregation and roll-up of data is required, an Aggregator JSON descriptor needs to be provided.Details on how describe aggregation using JSON can be found at: http://druid.io/docs/latest/querying/aggregations.html

Tags:

Druid, Timeseries, OLAP, ingest

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.

NameDefault ValueAllowable ValuesDescription
Druid Data SourceA data source is the Druid equivalent of a database table.
Supports Expression Language: true (will be evaluated using variable registry only)
Zookeeper Connection StringA comma-separated list of host:port pairs, each corresponding to a ZooKeeper server. Ex: localhost:2181
Supports Expression Language: true (will be evaluated using variable registry only)
Zookeeper Retry Base Sleep Time1000When a connection to Zookeeper needs to be retried, this property specifies the amount of time (in milliseconds) to wait at first before retrying.
Supports Expression Language: true (will be evaluated using variable registry only)
Zookeeper Retry Max Retries20When a connection to Zookeeper needs to be retried, this property specifies how many times to attempt reconnection.
Supports Expression Language: true (will be evaluated using variable registry only)
Zookeeper Retry Sleep Time30000When a connection to Zookeeper needs to be retried, this property specifies the amount of time to sleep (in milliseconds) between retries.
Supports Expression Language: true (will be evaluated using variable registry only)
Index Service Pathdruid/overlordDruid Index Service path as defined via the Druid Overlord druid.service property.
Supports Expression Language: true (will be evaluated using variable registry only)
Discovery Path/druid/discoveryDruid Discovery Path as configured in Druid Common druid.discovery.curator.path property
Supports Expression Language: true (will be evaluated using variable registry only)
Cluster Partitions1The number of partitions in the Druid cluster.
Supports Expression Language: true (will be evaluated using variable registry only)
Cluster Replication1The replication factor for the Druid cluster.
Supports Expression Language: true (will be evaluated using variable registry only)
Dimension FieldsA comma separated list of field names that will be stored as dimensions on ingest.
Supports Expression Language: true (will be evaluated using variable registry only)
Aggregator JSONTranquility-compliant JSON string that defines what aggregators to apply on ingest.Example: [{ "type" : "count", "name" : "count",},{ "name" : "value_sum", "type" : "doubleSum", "fieldName" : "value"},{ "fieldName" : "value", "name" : "value_min", "type" : "doubleMin"},{ "type" : "doubleMax", "name" : "value_max", "fieldName" : "value"}]
Supports Expression Language: true (will be evaluated using variable registry only)
Segment GranularityTEN_MINUTE
  • NONE
  • SECOND
  • MINUTE
  • TEN_MINUTE
  • HOUR
  • DAY
  • MONTH
  • YEAR
Time unit by which to group and aggregate/rollup events. The value must be at least as large as the value of Query Granularity.
Query GranularityMINUTE
  • NONE
  • SECOND
  • MINUTE
  • FIFTEEN_MINUTE
  • THIRTY_MINUTE
  • HOUR
  • DAY
  • MONTH
  • YEAR
Time unit by which to group and aggregate/rollup events. The value must be less than or equal to the value of Segment Granularity.
Late Event Grace PeriodPT10M
  • 1 minute 1 minute
  • 10 minutes 10 minutes
  • 60 minutes 1 hour
Grace period to allow late arriving events for real time ingest.
Timestamp fieldtimestampThe name of the field that will be used as the timestamp. Should be in ISO8601 format.
Supports Expression Language: true (will be evaluated using variable registry only)
Batch Size2000Maximum number of messages to send at once.
Supports Expression Language: true (will be evaluated using variable registry only)
Max Pending Batches5Maximum number of batches that may be in flight before service blocks and waits for one to finish.
Supports Expression Language: true (will be evaluated using variable registry only)
Linger (milliseconds)1000Wait this long for batches to collect more messages (up to Batch Size) before sending them. Set to zero to disable waiting. Set to -1 to always wait for complete batches before sending.
Supports Expression Language: true (will be evaluated using variable registry only)

State management:

This component does not store state.

Restricted:

This component is not restricted.

System Resource Considerations:

None specified.