RecordReaderFactory
Implementations: CSVReader
JsonPathReader
AvroReader
CEFReader
Syslog5424Reader
JsonTreeReader
WindowsEventLogReader
XMLReader
SyslogReader
JASN1Reader
ReaderLookup
ParquetReader
GrokReader
ScriptedReader
YamlTreeReader
ExcelReader
Supports Sensitive Dynamic Properties: No
Dynamic Properties allow the user to specify both the name and value of a property.
Name | Value | Description |
---|---|---|
The name given to the dynamic property is the name of the attribute that will be used to denote the value of the associated RecordPath. | A RecordPath that points to a field in the Record. | Each dynamic property represents a RecordPath that will be evaluated against each record in an incoming FlowFile. When the value of the RecordPath is determined for a Record, an attribute is added to the outgoing FlowFile. The name of the attribute is the same as the name of this property. The value of the attribute is the same as the value of the field in the Record that the RecordPath points to. Note that no attribute will be added if the value returned for the RecordPath is null or is not a scalar value (i.e., the value is an Array, Map, or Record). Supports Expression Language: true (will be evaluated using flow file attributes and Environment variables) |
Name | Description |
---|---|
success | FlowFiles that are successfully partitioned will be routed to this relationship |
failure | If a FlowFile cannot be partitioned from the configured input format to the configured output format, the unchanged FlowFile will be routed to this relationship |
original | Once all records in an incoming FlowFile have been partitioned, the original FlowFile is routed to this relationship. |
Name | Description |
---|---|
record.count | The number of records in an outgoing FlowFile |
mime.type | The MIME Type that the configured Record Writer indicates is appropriate |
fragment.identifier | All partitioned FlowFiles produced from the same parent FlowFile will have the same randomly generated UUID added for this attribute |
fragment.index | A one-up number that indicates the ordering of the partitioned FlowFiles that were created from a single parent FlowFile |
fragment.count | The number of partitioned FlowFiles generated from the parent FlowFile |
segment.original.filename | The filename of the parent FlowFile |
<dynamic property name> | For each dynamic property that is added, an attribute may be added to the FlowFile. See the description for Dynamic Properties for more information. |
Separate records into separate FlowFiles so that all of the records in a FlowFile have the same value for a given field or set of fields.
Choose a RecordReader that is appropriate based on the format of the incoming data.
Choose a RecordWriter that writes the data in the desired output format.
Add a single additional property. The name of the property should describe the type of data that is being used to partition the data. The property's value should be a RecordPath that specifies which output FlowFile the Record belongs to.
For example, if we want to separate records based on their transactionType
field, we could add a new property named transactionType
. The value of the property might be /transaction/type
. An input FlowFile will then be separated into as few FlowFiles as possible such that each output FlowFile has the same value for the transactionType
field.
Separate records based on whether or not they adhere to a specific criteria
Choose a RecordReader that is appropriate based on the format of the incoming data.
Choose a RecordWriter that writes the data in the desired output format.
Add a single additional property. The name of the property should describe the criteria. The property's value should be a RecordPath that returns true
if the Record meets the criteria or false
otherwise.
For example, if we want to separate records based on whether or not they have a transaction total of more than $1,000 we could add a new property named largeTransaction
with a value of /transaction/total > 1000
. This will create two FlowFiles. In the first, all records will have a total over 1000
. In the second, all records will have a transaction less than or equal to 1000. Each FlowFile will have an attribute named largeTransaction
with a value of true
or false
.