ControlRate 2.0.0

Bundle
org.apache.nifi | nifi-standard-nar
Description
Controls the rate at which data is transferred to follow-on processors. If you configure a very small Time Duration, then the accuracy of the throttle gets worse. You can improve this accuracy by decreasing the Yield Duration, at the expense of more Tasks given to the processor.
Tags
rate, rate control, throttle, throughput
Input Requirement
REQUIRED
Supports Sensitive Dynamic Properties
false
  • Additional Details for ControlRate 2.0.0

    ControlRate

    This processor throttles throughput of FlowFiles based on a configured rate. The rate can be specified as either a direct data rate (bytes per time period), or by counting FlowFiles or a specific attribute value. In all cases, the time period for measurement is specified in the Time Duration property.

    The processor operates in one of four available modes. The mode is determined by the Rate Control Criteria property.

    Mode Description
    Data Rate The FlowFile content size is accumulated for all FlowFiles passing through this processor. FlowFiles are throttled to ensure a maximum overall data rate (bytes per time period) is not exceeded. The Maximum Rate property specifies the maximum bytes allowed per Time Duration.
    FlowFile Count FlowFiles are counted regardless of content size. No more than the specified number of FlowFiles pass through this processor in the given Time Duration. The Maximum Rate property specifies the maximum number of FlowFiles allowed per Time Duration.
    Attribute Value The value of an attribute is accumulated to determine overall rate. The Rate Controlled Attribute property specifies the attribute whose value will be accumulated. The value of the specified attribute is expected to be an integer. This mode is independent of overall FlowFile size and count.
    Data Rate or FlowFile Count This mode provides a combination of Data Rate and FlowFile Count. Both rates are accumulated and FlowFiles are throttled if either rate is exceeded. Both Maximum Data Rate and Maximum FlowFile Rate properties must be specified to determine content size and FlowFile count per Time Duration.

    If the Grouping Attribute property is specified, all rates are accumulated separately for unique values of the specified attribute. For example, assume Grouping Attribute property is specified and its value is “city”. All FlowFiles containing a “city” attribute with value “Albuquerque” will have an accumulated rate calculated. A separate rate will be calculated for all FlowFiles containing a “city” attribute with a value “Boston”. In other words, separate rate calculations will be accumulated for all unique values of the Grouping Attribute.

Properties
Relationships
Name Description
failure FlowFiles will be routed to this relationship if they are missing a necessary Rate Controlled Attribute or the attribute is not in the expected format
success FlowFiles are transferred to this relationship under normal conditions
Use Cases
  • Limit the rate at which data is sent to a downstream system with little to no bursts
    Description
    Limit the rate at which data is sent to a downstream system with little to no bursts
    Keywords
    throttle, limit, slow down, data rate
    Configuration
    Set the "Rate Control Criteria" to `data rate`.
Set the "Time Duration" property to `1 sec`.
Configure the "Maximum Rate" property to specify how much data should be allowed through each second.

For example, to allow through 8 MB per second, set "Maximum Rate" to `8 MB`.
  • Limit the rate at which FlowFiles are sent to a downstream system with little to no bursts
    Description
    Limit the rate at which FlowFiles are sent to a downstream system with little to no bursts
    Keywords
    throttle, limit, slow down, flowfile rate
    Configuration
    Set the "Rate Control Criteria" to `flowfile count`.
Set the "Time Duration" property to `1 sec`.
Configure the "Maximum Rate" property to specify how many FlowFiles should be allowed through each second.

For example, to allow through 100 FlowFiles per second, set "Maximum Rate" to `100`.
  • Reject requests that exceed a specific rate with little to no bursts
    Description
    Reject requests that exceed a specific rate with little to no bursts
    Keywords
    throttle, limit, slow down, request rate
    Configuration
    Set the "Rate Control Criteria" to `flowfile count`.
Set the "Time Duration" property to `1 sec`.
Set the "Rate Exceeded Strategy" property to `Route to 'rate exceeded'`.
Configure the "Maximum Rate" property to specify how many requests should be allowed through each second.

For example, to allow through 100 requests per second, set "Maximum Rate" to `100`.
If more than 100 requests come in during any one second, the additional requests will be routed to `rate exceeded` instead of `success`.
  • Reject requests that exceed a specific rate, allowing for bursts
    Description
    Reject requests that exceed a specific rate, allowing for bursts
    Keywords
    throttle, limit, slow down, request rate
    Configuration
    Set the "Rate Control Criteria" to `flowfile count`.
Set the "Time Duration" property to `1 min`.
Set the "Rate Exceeded Strategy" property to `Route to 'rate exceeded'`.
Configure the "Maximum Rate" property to specify how many requests should be allowed through each minute.

For example, to allow through 100 requests per second, set "Maximum Rate" to `6000`.
This will allow through 6,000 FlowFiles per minute, which averages to 100 FlowFiles per second. However, those 6,000 FlowFiles may come all within the first couple of
seconds, or they may come in over a period of 60 seconds. As a result, this gives us an average rate of 100 FlowFiles per second but allows for bursts of data.
If more than 6,000 requests come in during any one minute, the additional requests will be routed to `rate exceeded` instead of `success`.