CSVRecordSetWriter

Description:

Writes the contents of a RecordSet as CSV data. The first line written will be the column names (unless the 'Include Header Line' property is false). All subsequent lines will be the values corresponding to the record fields.

Tags:

csv, result, set, recordset, record, writer, serializer, row, tsv, tab, separated, delimited

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.

NameDefault ValueAllowable ValuesDescription
Schema Write Strategyschema-name
  • Set 'schema.name' Attribute The FlowFile will be given an attribute named 'schema.name' and this attribute will indicate the name of the schema in the Schema Registry. Note that ifthe schema for a record is not obtained from a Schema Registry, then no attribute will be added.
  • Set 'avro.schema' Attribute The FlowFile will be given an attribute named 'avro.schema' and this attribute will contain the Avro Schema that describes the records in the FlowFile. The contents of the FlowFile need not be Avro, but the text of the schema will be used.
  • HWX Schema Reference Attributes The FlowFile will be given a set of 3 attributes to describe the schema: 'schema.identifier', 'schema.version', and 'schema.protocol.version'. Note that if the schema for a record does not contain the necessary identifier and version, an Exception will be thrown when attempting to write the data.
  • HWX Content-Encoded Schema Reference The content of the FlowFile will contain a reference to a schema in the Schema Registry service. The reference is encoded as a single byte indicating the 'protocol version', followed by 8 bytes indicating the schema identifier, and finally 4 bytes indicating the schema version, as per the Hortonworks Schema Registry serializers and deserializers, as found at https://github.com/hortonworks/registry. This will be prepended to each FlowFile. Note that if the schema for a record does not contain the necessary identifier and version, an Exception will be thrown when attempting to write the data.
  • Confluent Schema Registry Reference The content of the FlowFile will contain a reference to a schema in the Schema Registry service. The reference is encoded as a single 'Magic Byte' followed by 4 bytes representing the identifier of the schema, as outlined at http://docs.confluent.io/current/schema-registry/docs/serializer-formatter.html. This will be prepended to each FlowFile. Note that if the schema for a record does not contain the necessary identifier and version, an Exception will be thrown when attempting to write the data. This is based on the encoding used by version 3.2.x of the Confluent Schema Registry.
  • Do Not Write Schema Do not add any schema-related information to the FlowFile.
Specifies how the schema for a Record should be added to the data.
Schema Access Strategyinherit-record-schema
  • Use 'Schema Name' Property The name of the Schema to use is specified by the 'Schema Name' Property. The value of this property is used to lookup the Schema in the configured Schema Registry service.
  • Inherit Record Schema The schema used to write records will be the same schema that was given to the Record when the Record was created.
  • Use 'Schema Text' Property The text of the Schema itself is specified by the 'Schema Text' Property. The value of this property must be a valid Avro Schema. If Expression Language is used, the value of the 'Schema Text' property must be valid after substituting the expressions.
Specifies how to obtain the schema that is to be used for interpreting the data.
Schema RegistryController Service API:
SchemaRegistry
Implementations: ConfluentSchemaRegistry
HortonworksSchemaRegistry
AvroSchemaRegistry
Specifies the Controller Service to use for the Schema Registry
Schema Name${schema.name}Specifies the name of the schema to lookup in the Schema Registry property
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Schema VersionSpecifies the version of the schema to lookup in the Schema Registry. If not specified then the latest version of the schema will be retrieved.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Schema BranchSpecifies the name of the branch to use when looking up the schema in the Schema Registry property. If the chosen Schema Registry does not support branching, this value will be ignored.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Schema Text${avro.schema}The text of an Avro-formatted Schema
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Date FormatSpecifies the format to use when reading/writing Date fields. If not specified, Date fields will be assumed to be number of milliseconds since epoch (Midnight, Jan 1, 1970 GMT). If specified, the value must match the Java Simple Date Format (for example, MM/dd/yyyy for a two-digit month, followed by a two-digit day, followed by a four-digit year, all separated by '/' characters, as in 01/01/2017).
Time FormatSpecifies the format to use when reading/writing Time fields. If not specified, Time fields will be assumed to be number of milliseconds since epoch (Midnight, Jan 1, 1970 GMT). If specified, the value must match the Java Simple Date Format (for example, HH:mm:ss for a two-digit hour in 24-hour format, followed by a two-digit minute, followed by a two-digit second, all separated by ':' characters, as in 18:04:15).
Timestamp FormatSpecifies the format to use when reading/writing Timestamp fields. If not specified, Timestamp fields will be assumed to be number of milliseconds since epoch (Midnight, Jan 1, 1970 GMT). If specified, the value must match the Java Simple Date Format (for example, MM/dd/yyyy HH:mm:ss for a two-digit month, followed by a two-digit day, followed by a four-digit year, all separated by '/' characters; and then followed by a two-digit hour in 24-hour format, followed by a two-digit minute, followed by a two-digit second, all separated by ':' characters, as in 01/01/2017 18:04:15).
CSV Formatcustom
  • Custom Format The format of the CSV is configured by using the properties of this Controller Service, such as Value Separator
  • RFC 4180 CSV data follows the RFC 4180 Specification defined at https://tools.ietf.org/html/rfc4180
  • Microsoft Excel CSV data follows the format used by Microsoft Excel
  • Tab-Delimited CSV data is Tab-Delimited instead of Comma Delimited
  • MySQL Format CSV data follows the format used by MySQL
  • Informix Unload The format used by Informix when issuing the UNLOAD TO file_name command
  • Informix Unload Escape Disabled The format used by Informix when issuing the UNLOAD TO file_name command with escaping disabled
Specifies which "format" the CSV data is in, or specifies if custom formatting should be used.
Value Separator,The character that is used to separate values/fields in a CSV Record
Include Header Linetrue
  • true
  • false
Specifies whether or not the CSV column names should be written out as the first line.
Quote Character"The character that is used to quote values so that escape characters do not have to be used
Escape Character\The character that is used to escape characters that would otherwise have a specific meaning to the CSV Parser.
Comment MarkerThe character that is used to denote the start of a comment. Any line that begins with this comment will be ignored.
Null StringSpecifies a String that, if present as a value in the CSV, should be considered a null field instead of using the literal value.
Trim Fieldstrue
  • true
  • false
Whether or not white space should be removed from the beginning and end of fields
Quote ModeMINIMAL
  • Quote All Values All values will be quoted using the configured quote character.
  • Quote Minimal Values will be quoted only if they are contain special characters such as newline characters or field separators.
  • Quote Non-Numeric Values Values will be quoted unless the value is a number.
  • Do Not Quote Values Values will not be quoted. Instead, all special characters will be escaped using the configured escape character.
Specifies how fields should be quoted when they are written
Record Separator\nSpecifies the characters to use in order to separate CSV Records
Include Trailing Delimiterfalse
  • true
  • false
If true, a trailing delimiter will be added to each CSV Record that is written. If false, the trailing delimiter will be omitted.
Character SetUTF-8The Character Encoding that is used to encode/decode the CSV file

State management:

This component does not store state.

Restricted:

This component is not restricted.

System Resource Considerations:

None specified.