CSVReader

Comment Marker

The character that is used to denote the start of a comment. Any line that begins with this comment will be ignored.

Display Name

Comment Marker

Description

The character that is used to denote the start of a comment. Any line that begins with this comment will be ignored.

API Name

Comment Marker

Expression Language Scope

Environment variables and FlowFile Attributes

Sensitive

false

Required

false

Dependencies

CSV Format is set to any of [custom]

CSV Format

Specifies which "format" the CSV data is in, or specifies if custom formatting should be used.

Display Name

CSV Format

Description

Specifies which "format" the CSV data is in, or specifies if custom formatting should be used.

API Name

CSV Format

Default Value

custom

Allowable Values

Custom Format
RFC 4180
Microsoft Excel
Tab-Delimited
MySQL Format
Informix Unload
Informix Unload Escape Disabled

Expression Language Scope

Not Supported

Sensitive

false

Required

true

CSV Parser

Specifies which parser to use to read CSV records. NOTE: Different parsers may support different subsets of functionality and may also exhibit different levels of performance.

Display Name

CSV Parser

Description

Specifies which parser to use to read CSV records. NOTE: Different parsers may support different subsets of functionality and may also exhibit different levels of performance.

API Name

csv-reader-csv-parser

Default Value

commons-csv

Allowable Values

Apache Commons CSV
Jackson CSV
FastCSV

Expression Language Scope

Not Supported

Sensitive

false

Required

true

Allow Duplicate Header Names

Whether duplicate header names are allowed. Header names are case-sensitive, for example "name" and "Name" are treated as separate fields. Handling of duplicate header names is CSV Parser specific (where applicable): * Apache Commons CSV - duplicate headers will result in column data "shifting" right with new fields created for "unknown_field_index_X" where "X" is the CSV column index number * Jackson CSV - duplicate headers will be de-duplicated with the field value being that of the right-most duplicate CSV column * FastCSV - duplicate headers will be de-duplicated with the field value being that of the left-most duplicate CSV column

Display Name

Allow Duplicate Header Names

Description

Whether duplicate header names are allowed. Header names are case-sensitive, for example "name" and "Name" are treated as separate fields. Handling of duplicate header names is CSV Parser specific (where applicable): * Apache Commons CSV - duplicate headers will result in column data "shifting" right with new fields created for "unknown_field_index_X" where "X" is the CSV column index number * Jackson CSV - duplicate headers will be de-duplicated with the field value being that of the right-most duplicate CSV column * FastCSV - duplicate headers will be de-duplicated with the field value being that of the left-most duplicate CSV column

API Name

csvutils-allow-duplicate-header-names

Default Value

true

Allowable Values

true
false

Expression Language Scope

Not Supported

Sensitive

false

Required

false

Dependencies

CSV Format is set to any of [custom]

Character Set

The Character Encoding that is used to encode/decode the CSV file

Display Name: Character Set
Description: The Character Encoding that is used to encode/decode the CSV file
API Name: csvutils-character-set
Default Value: UTF-8
Expression Language Scope: Not Supported
Sensitive: false
Required: true

Date Format

Specifies the format to use when reading/writing Date fields. If not specified, Date fields will be assumed to be number of milliseconds since epoch (Midnight, Jan 1, 1970 GMT). If specified, the value must match the Java java.time.format.DateTimeFormatter format (for example, MM/dd/yyyy for a two-digit month, followed by a two-digit day, followed by a four-digit year, all separated by '/' characters, as in 01/01/2017).

Display Name: Date Format
Description: Specifies the format to use when reading/writing Date fields. If not specified, Date fields will be assumed to be number of milliseconds since epoch (Midnight, Jan 1, 1970 GMT). If specified, the value must match the Java java.time.format.DateTimeFormatter format (for example, MM/dd/yyyy for a two-digit month, followed by a two-digit day, followed by a four-digit year, all separated by '/' characters, as in 01/01/2017).
API Name: Date Format
Expression Language Scope: Not Supported
Sensitive: false
Required: false

Escape Character

The character that is used to escape characters that would otherwise have a specific meaning to the CSV Parser. If the property has been specified via Expression Language but the expression gets evaluated to an invalid Escape Character at runtime, then it will be skipped and the default Escape Character will be used. Setting it to an empty string means no escape character should be used.

Display Name

Escape Character

Description

The character that is used to escape characters that would otherwise have a specific meaning to the CSV Parser. If the property has been specified via Expression Language but the expression gets evaluated to an invalid Escape Character at runtime, then it will be skipped and the default Escape Character will be used. Setting it to an empty string means no escape character should be used.

API Name

Escape Character

Default Value

\

Expression Language Scope

Environment variables and FlowFile Attributes

Sensitive

false

Required

true

Dependencies

CSV Format is set to any of [custom]

Ignore CSV Header Column Names

If the first line of a CSV is a header, and the configured schema does not match the fields named in the header line, this controls how the Reader will interpret the fields. If this property is true, then the field names mapped to each column are driven only by the configured schema and any fields not in the schema will be ignored. If this property is false, then the field names found in the CSV Header will be used as the names of the fields.

Display Name

Ignore CSV Header Column Names

Description

If the first line of a CSV is a header, and the configured schema does not match the fields named in the header line, this controls how the Reader will interpret the fields. If this property is true, then the field names mapped to each column are driven only by the configured schema and any fields not in the schema will be ignored. If this property is false, then the field names found in the CSV Header will be used as the names of the fields.

API Name

ignore-csv-header

Default Value

false

Allowable Values

true
false

Expression Language Scope

Not Supported

Sensitive

false

Required

false

Null String

Specifies a String that, if present as a value in the CSV, should be considered a null field instead of using the literal value.

Display Name

Null String

Description

Specifies a String that, if present as a value in the CSV, should be considered a null field instead of using the literal value.

API Name

Null String

Expression Language Scope

Not Supported

Sensitive

false

Required

false

Dependencies

CSV Format is set to any of [custom]

Quote Character

The character that is used to quote values so that escape characters do not have to be used. If the property has been specified via Expression Language but the expression gets evaluated to an invalid Quote Character at runtime, then it will be skipped and the default Quote Character will be used.

Display Name

Quote Character

Description

The character that is used to quote values so that escape characters do not have to be used. If the property has been specified via Expression Language but the expression gets evaluated to an invalid Quote Character at runtime, then it will be skipped and the default Quote Character will be used.

API Name

Quote Character

Default Value

"

Expression Language Scope

Environment variables and FlowFile Attributes

Sensitive

false

Required

true

Dependencies

CSV Format is set to any of [custom]

Record Separator

Specifies the characters to use in order to separate CSV Records

Display Name

Record Separator

Description

Specifies the characters to use in order to separate CSV Records

API Name

Record Separator

Default Value

\n

Expression Language Scope

Not Supported

Sensitive

false

Required

true

Dependencies

CSV Format is set to any of [custom]

Schema Access Strategy

Specifies how to obtain the schema that is to be used for interpreting the data.

Display Name

Schema Access Strategy

Description

Specifies how to obtain the schema that is to be used for interpreting the data.

API Name

schema-access-strategy

Default Value

infer-schema

Allowable Values

Use 'Schema Name' Property
Use 'Schema Text' Property
Schema Reference Reader
Use String Fields From Header
Infer Schema

Expression Language Scope

Not Supported

Sensitive

false

Required

true

Schema Branch

Specifies the name of the branch to use when looking up the schema in the Schema Registry property. If the chosen Schema Registry does not support branching, this value will be ignored.

Display Name

Schema Branch

Description

Specifies the name of the branch to use when looking up the schema in the Schema Registry property. If the chosen Schema Registry does not support branching, this value will be ignored.

API Name

schema-branch

Expression Language Scope

Environment variables and FlowFile Attributes

Sensitive

false

Required

false

Dependencies

Schema Access Strategy is set to any of [schema-name]

Schema Name

Specifies the name of the schema to lookup in the Schema Registry property

Display Name

Schema Name

Description

Specifies the name of the schema to lookup in the Schema Registry property

API Name

schema-name

Default Value

${schema.name}

Expression Language Scope

Environment variables and FlowFile Attributes

Sensitive

false

Required

false

Dependencies

Schema Access Strategy is set to any of [schema-name]

Schema Reference Reader

Service implementation responsible for reading FlowFile attributes or content to determine the Schema Reference Identifier

Display Name

Schema Reference Reader

Description

Service implementation responsible for reading FlowFile attributes or content to determine the Schema Reference Identifier

API Name

schema-reference-reader

Service Interface

org.apache.nifi.schemaregistry.services.SchemaReferenceReader

Service Implementations

org.apache.nifi.confluent.schemaregistry.ConfluentEncodedSchemaReferenceReader

Expression Language Scope

Not Supported

Sensitive

false

Required

true

Dependencies

Schema Access Strategy is set to any of [schema-reference-reader]

Schema Registry

Specifies the Controller Service to use for the Schema Registry

Display Name

Schema Registry

Description

Specifies the Controller Service to use for the Schema Registry

API Name

schema-registry

Service Interface

org.apache.nifi.schemaregistry.services.SchemaRegistry

Service Implementations

org.apache.nifi.aws.schemaregistry.AmazonGlueSchemaRegistry

org.apache.nifi.apicurio.schemaregistry.ApicurioSchemaRegistry

org.apache.nifi.schemaregistry.services.AvroSchemaRegistry

org.apache.nifi.confluent.schemaregistry.ConfluentSchemaRegistry

org.apache.nifi.db.schemaregistry.DatabaseTableSchemaRegistry

Expression Language Scope

Not Supported

Sensitive

false

Required

false

Dependencies

Schema Access Strategy is set to any of [schema-reference-reader, schema-name]

Schema Text

The text of an Avro-formatted Schema

Display Name

Schema Text

Description

The text of an Avro-formatted Schema

API Name

schema-text

Default Value

${avro.schema}

Expression Language Scope

Environment variables and FlowFile Attributes

Sensitive

false

Required

false

Dependencies

Schema Access Strategy is set to any of [schema-text-property]

Schema Version

Specifies the version of the schema to lookup in the Schema Registry. If not specified then the latest version of the schema will be retrieved.

Display Name

Schema Version

Description

Specifies the version of the schema to lookup in the Schema Registry. If not specified then the latest version of the schema will be retrieved.

API Name

schema-version

Expression Language Scope

Environment variables and FlowFile Attributes

Sensitive

false

Required

false

Dependencies

Schema Access Strategy is set to any of [schema-name]

Treat First Line as Header

Specifies whether or not the first line of CSV should be considered a Header or should be considered a record. If the Schema Access Strategy indicates that the columns must be defined in the header, then this property will be ignored, since the header must always be present and won't be processed as a Record. Otherwise, if 'true', then the first line of CSV data will not be processed as a record and if 'false',then the first line will be interpreted as a record.

Display Name

Treat First Line as Header

Description

Specifies whether or not the first line of CSV should be considered a Header or should be considered a record. If the Schema Access Strategy indicates that the columns must be defined in the header, then this property will be ignored, since the header must always be present and won't be processed as a Record. Otherwise, if 'true', then the first line of CSV data will not be processed as a record and if 'false',then the first line will be interpreted as a record.

API Name

Skip Header Line

Default Value

false

Allowable Values

true
false

Expression Language Scope

Not Supported

Sensitive

false

Required

true

Time Format

Specifies the format to use when reading/writing Time fields. If not specified, Time fields will be assumed to be number of milliseconds since epoch (Midnight, Jan 1, 1970 GMT). If specified, the value must match the Java java.time.format.DateTimeFormatter format (for example, HH:mm:ss for a two-digit hour in 24-hour format, followed by a two-digit minute, followed by a two-digit second, all separated by ':' characters, as in 18:04:15).

Display Name: Time Format
Description: Specifies the format to use when reading/writing Time fields. If not specified, Time fields will be assumed to be number of milliseconds since epoch (Midnight, Jan 1, 1970 GMT). If specified, the value must match the Java java.time.format.DateTimeFormatter format (for example, HH:mm:ss for a two-digit hour in 24-hour format, followed by a two-digit minute, followed by a two-digit second, all separated by ':' characters, as in 18:04:15).
API Name: Time Format
Expression Language Scope: Not Supported
Sensitive: false
Required: false

Timestamp Format

Specifies the format to use when reading/writing Timestamp fields. If not specified, Timestamp fields will be assumed to be number of milliseconds since epoch (Midnight, Jan 1, 1970 GMT). If specified, the value must match the Java java.time.format.DateTimeFormatter format (for example, MM/dd/yyyy HH:mm:ss for a two-digit month, followed by a two-digit day, followed by a four-digit year, all separated by '/' characters; and then followed by a two-digit hour in 24-hour format, followed by a two-digit minute, followed by a two-digit second, all separated by ':' characters, as in 01/01/2017 18:04:15).

Display Name: Timestamp Format
Description: Specifies the format to use when reading/writing Timestamp fields. If not specified, Timestamp fields will be assumed to be number of milliseconds since epoch (Midnight, Jan 1, 1970 GMT). If specified, the value must match the Java java.time.format.DateTimeFormatter format (for example, MM/dd/yyyy HH:mm:ss for a two-digit month, followed by a two-digit day, followed by a four-digit year, all separated by '/' characters; and then followed by a two-digit hour in 24-hour format, followed by a two-digit minute, followed by a two-digit second, all separated by ':' characters, as in 01/01/2017 18:04:15).
API Name: Timestamp Format
Expression Language Scope: Not Supported
Sensitive: false
Required: false

Trim double quote

Whether or not to trim starting and ending double quotes. For example: with trim string '"test"' would be parsed to 'test', without trim would be parsed to '"test"'.If set to 'false' it means full compliance with RFC-4180. Default value is true, with trim.

Display Name

Trim double quote

Description

Whether or not to trim starting and ending double quotes. For example: with trim string '"test"' would be parsed to 'test', without trim would be parsed to '"test"'.If set to 'false' it means full compliance with RFC-4180. Default value is true, with trim.

API Name

Trim double quote

Default Value

true

Allowable Values

true
false

Expression Language Scope

Not Supported

Sensitive

false

Required

true

Dependencies

CSV Format is set to any of [rfc-4180]

Trim Fields

Whether or not white space should be removed from the beginning and end of fields

Display Name

Trim Fields

Description

Whether or not white space should be removed from the beginning and end of fields

API Name

Trim Fields

Default Value

true

Allowable Values

true
false

Expression Language Scope

Not Supported

Sensitive

false

Required

true

Dependencies

CSV Format is set to any of [custom]

Value Separator

The character that is used to separate values/fields in a CSV Record. If the property has been specified via Expression Language but the expression gets evaluated to an invalid Value Separator at runtime, then it will be skipped and the default Value Separator will be used.

Display Name

Value Separator

Description

The character that is used to separate values/fields in a CSV Record. If the property has been specified via Expression Language but the expression gets evaluated to an invalid Value Separator at runtime, then it will be skipped and the default Value Separator will be used.

API Name

Value Separator

Default Value

,

Expression Language Scope

Environment variables and FlowFile Attributes

Sensitive

false

Required

true

Dependencies

CSV Format is set to any of [custom]

CSVReader 2.0.0