ScriptedPartitionRecord

Description:

Receives Record-oriented data (i.e., data that can be read by the configured Record Reader) and evaluates the user provided script against each record in the incoming flow file. Each record is then grouped with other records sharing the same partition and a FlowFile is created for each groups of records. Two records shares the same partition if the evaluation of the script results the same return value for both. Those will be considered as part of the same partition.

Additional Details...

Tags:

record, partition, script, groovy, jython, python, segment, split, group, organize

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.

Display NameAPI NameDefault ValueAllowable ValuesDescription
Record ReaderRecord ReaderController Service API:
RecordReaderFactory
Implementations: CEFReader
Syslog5424Reader
ScriptedReader
SyslogReader
CSVReader
GrokReader
JsonTreeReader
XMLReader
AvroReader
ReaderLookup
WindowsEventLogReader
ParquetReader
JsonPathReader
The Record Reader to use parsing the incoming FlowFile into Records
Record WriterRecord WriterController Service API:
RecordSetWriterFactory
Implementations: RecordSetWriterLookup
XMLRecordSetWriter
FreeFormTextRecordSetWriter
CSVRecordSetWriter
ScriptedRecordSetWriter
AvroRecordSetWriter
JsonRecordSetWriter
ParquetRecordSetWriter
The Record Writer to use for serializing Records after they have been transformed
Script LanguageScript EngineGroovy
  • Clojure
  • ECMAScript
  • Groovy
  • lua
  • python
  • ruby
The Language to use for the script
Script BodyScript BodyBody of script to execute. Only one of Script File or Script Body may be used
Script FileScript FilePath to script file to execute. Only one of Script File or Script Body may be used

This property requires exactly one file to be provided..

Supports Expression Language: true (will be evaluated using variable registry only)
Module DirectoryModule DirectoryComma-separated list of paths to files and/or directories which contain modules required by the script.

This property expects a comma-separated list of resources. Each of the resources may be of any of the following types: directory, file.

Supports Expression Language: true (will be evaluated using variable registry only)

Relationships:

NameDescription
successFlowFiles that are successfully partitioned will be routed to this relationship
failureIf a FlowFile cannot be partitioned from the configured input format to the configured output format, the unchanged FlowFile will be routed to this relationship
originalOnce all records in an incoming FlowFile have been partitioned, the original FlowFile is routed to this relationship.

Reads Attributes:

None specified.

Writes Attributes:

NameDescription
partitionThe partition of the outgoing flow file. If the script indicates that the partition has a null value, the attribute will be set to the literal string "<null partition>" (without quotes). Otherwise, the attribute is set to the String representation of whatever value is returned by the script.
mime.typeSets the mime.type attribute to the MIME Type specified by the Record Writer
record.countThe number of records within the flow file.
record.error.messageThis attribute provides on failure the error message encountered by the Reader or Writer.
fragment.indexA one-up number that indicates the ordering of the partitioned FlowFiles that were created from a single parent FlowFile
fragment.countThe number of partitioned FlowFiles generated from the parent FlowFile

State management:

This component does not store state.

Restricted:

Required PermissionExplanation
execute codeProvides operator the ability to execute arbitrary code assuming all permissions that NiFi has.

System Resource Considerations:

None specified.

See Also:

ScriptedTransformRecord, ScriptedValidateRecord, ScriptedFilterRecord