MergeRecord

Description:

This Processor merges together multiple record-oriented FlowFiles into a single FlowFile that contains all of the Records of the input FlowFiles. This Processor works by creating 'bins' and then adding FlowFiles to these bins until they are full. Once a bin is full, all of the FlowFiles will be combined into a single output FlowFile, and that FlowFile will be routed to the 'merged' Relationship. A bin will consist of potentially many 'like FlowFiles'. In order for two FlowFiles to be considered 'like FlowFiles', they must have the same Schema (as identified by the Record Reader) and, if the <Correlation Attribute Name> property is set, the same value for the specified attribute. See Processor Usage and Additional Details for more information. NOTE: this processor should NOT be configured with Cron Driven for the Scheduling Strategy.

Additional Details...

Tags:

merge, record, content, correlation, stream, event

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.

Relationships:

Reads Attributes:

Writes Attributes:

State management:

Restricted:

Input requirement:

Example Use Cases:

Use Case:

Combine together many arbitrary Records in order to create a single, larger file

Configuration:

Configure the "Record Reader" to specify a Record Reader that is appropriate for the incoming data type.

Configure the "Record Writer" to specify a Record Writer that is appropriate for the desired output data type.

Set "Merge Strategy" to Bin-Packing Algorithm.

Set the "Minimum Bin Size" to desired file size of the merged output file. For example, a value of 1 MB will result in not merging data until at least

1 MB of data is available (unless the Max Bin Age is reached first). If there is no desired minimum file size, leave the default value of 0 B.

Set the "Minimum Number of Records" property to the minimum number of Records that should be included in the merged output file. For example, setting the value

to 10000 ensures that the output file will have at least 10,000 Records in it (unless the Max Bin Age is reached first).

Set the "Max Bin Age" to specify the maximum amount of time to hold data before merging. This can be thought of as a "timeout" at which time the Processor will

merge whatever data it is, even if the "Minimum Bin Size" and "Minimum Number of Records" has not been reached. It is always recommended to set the value.

A reasonable default might be 10 mins if there is no other latency requirement.

Connect the 'merged' Relationship to the next component in the flow. Auto-terminate the 'original' Relationship.

Example Use Cases Involving Other Components:

Use Case:

Combine together many Records that have the same value for a particular field in the data, in order to create a single, larger file

Keywords:

Components involved:

Configuration:

Configure the "Record Reader" to specify a Record Reader that is appropriate for the incoming data type.

Configure the "Record Writer" to specify a Record Writer that is appropriate for the desired output data type.

Add a single additional property. The name of the property should describe the field on which the data is being merged together.

The property's value should be a RecordPath that specifies which output FlowFile the Record belongs to.

For example, to merge together data that has the same value for the "productSku" field, add a property named productSku with a value of /productSku.

Connect the "success" Relationship to MergeRecord.

Auto-terminate the "original" Relationship.

Configuration:

Configure the "Record Reader" to specify a Record Reader that is appropriate for the incoming data type.

Configure the "Record Writer" to specify a Record Writer that is appropriate for the desired output data type.

Set "Merge Strategy" to Bin-Packing Algorithm.

Set the "Minimum Bin Size" to desired file size of the merged output file. For example, a value of 1 MB will result in not merging data until at least

1 MB of data is available (unless the Max Bin Age is reached first). If there is no desired minimum file size, leave the default value of 0 B.

Set the "Minimum Number of Records" property to the minimum number of Records that should be included in the merged output file. For example, setting the value

to 10000 ensures that the output file will have at least 10,000 Records in it (unless the Max Bin Age is reached first).

Set the "Maximum Number of Records" property to a value at least as large as the "Minimum Number of Records." If there is no need to limit the maximum number of

records per file, this number can be set to a value that will never be reached such as 1000000000.

Set the "Max Bin Age" to specify the maximum amount of time to hold data before merging. This can be thought of as a "timeout" at which time the Processor will

merge whatever data it is, even if the "Minimum Bin Size" and "Minimum Number of Records" has not been reached. It is always recommended to set the value.

A reasonable default might be 10 mins if there is no other latency requirement.

Set the value of the "Correlation Attribute Name" property to the name of the property that you added in the PartitionRecord Processor. For example, if merging data

based on the "productSku" field, the property in PartitionRecord was named productSku so the value of the "Correlation Attribute Name" property should

be productSku.

Set the "Maximum Number of Bins" property to a value that is at least as large as the different number of values that will be present for the Correlation Attribute.

For example, if you expect 1,000 different SKUs, set this value to at least 1001. It is not advisable, though, to set the value above 10,000.

Connect the 'merged' Relationship to the next component in the flow.

Auto-terminate the 'original' Relationship.

System Resource Considerations:

See Also:

MergeContent, SplitRecord, PartitionRecord