YamlTreeReader

Description:

Parses YAML into individual Record objects. While the reader expects each record to be well-formed YAML, the content of a FlowFile may consist of many records, each as a well-formed YAML array or YAML object. If an array is encountered, each element in that array will be treated as a separate record. If the schema that is configured contains a field that is not present in the YAML, a null value will be used. If the YAML contains a field that is not present in the schema, that field will be skipped. Please note this controller service does not support resolving the use of YAML aliases. Any alias present will be treated as a string. See the Usage of the Controller Service for more information and examples.

Additional Details...

Tags:

yaml, tree, record, reader, parser

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.

Display NameAPI NameDefault ValueAllowable ValuesDescription
Schema Access Strategyschema-access-strategyInfer Schema
  • Infer Schema The Schema of the data will be inferred automatically when the data is read. See component Usage and Additional Details for information about how the schema is inferred.
  • Use 'Schema Name' Property The name of the Schema to use is specified by the 'Schema Name' Property. The value of this property is used to lookup the Schema in the configured Schema Registry service.
  • Use 'Schema Text' Property The text of the Schema itself is specified by the 'Schema Text' Property. The value of this property must be a valid Avro Schema. If Expression Language is used, the value of the 'Schema Text' property must be valid after substituting the expressions.
  • HWX Schema Reference Attributes The FlowFile contains 3 Attributes that will be used to lookup a Schema from the configured Schema Registry: 'schema.identifier', 'schema.version', and 'schema.protocol.version'
  • HWX Content-Encoded Schema Reference The content of the FlowFile contains a reference to a schema in the Schema Registry service. The reference is encoded as a single byte indicating the 'protocol version', followed by 8 bytes indicating the schema identifier, and finally 4 bytes indicating the schema version, as per the Hortonworks Schema Registry serializers and deserializers, found at https://github.com/hortonworks/registry
  • Confluent Content-Encoded Schema Reference The content of the FlowFile contains a reference to a schema in the Schema Registry service. The reference is encoded as a single 'Magic Byte' followed by 4 bytes representing the identifier of the schema, as outlined at http://docs.confluent.io/current/schema-registry/docs/serializer-formatter.html. This is based on version 3.2.x of the Confluent Schema Registry.
Specifies how to obtain the schema that is to be used for interpreting the data.
Schema Registryschema-registryController Service API:
SchemaRegistry
Implementations: DatabaseTableSchemaRegistry
HortonworksSchemaRegistry
ConfluentSchemaRegistry
AvroSchemaRegistry
AmazonGlueSchemaRegistry		Specifies the Controller Service to use for the Schema Registry

This Property is only considered if the [Schema Access Strategy] Property is set to one of the following values: [Confluent Content-Encoded Schema Reference], [Use 'Schema Name' Property], [HWX Schema Reference Attributes], [HWX Content-Encoded Schema Reference]
Schema Nameschema-name${schema.name}Specifies the name of the schema to lookup in the Schema Registry property
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)

This Property is only considered if the [Schema Access Strategy] Property has a value of "Use 'Schema Name' Property".
Schema Versionschema-versionSpecifies the version of the schema to lookup in the Schema Registry. If not specified then the latest version of the schema will be retrieved.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)

This Property is only considered if the [Schema Access Strategy] Property has a value of "Use 'Schema Name' Property".
Schema Branchschema-branchSpecifies the name of the branch to use when looking up the schema in the Schema Registry property. If the chosen Schema Registry does not support branching, this value will be ignored.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)

This Property is only considered if the [Schema Access Strategy] Property has a value of "Use 'Schema Name' Property".
Schema Textschema-text${avro.schema}The text of an Avro-formatted Schema
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)

This Property is only considered if the [Schema Access Strategy] Property has a value of "Use 'Schema Text' Property".
Schema Inference Cacheschema-inference-cacheController Service API:
RecordSchemaCacheService
Implementation: VolatileSchemaCache		Specifies a Schema Cache to use when inferring the schema. If not populated, the schema will be inferred each time. However, if a cache is specified, the cache will first be consulted and if the applicable schema can be found, it will be used instead of inferring the schema.

This Property is only considered if the [Schema Access Strategy] Property has a value of "Infer Schema".
Starting Field Strategystarting-field-strategyRoot Node
  • Root Node Begins processing from the root node.
  • Nested Field Skips forward to the given nested JSON field (array or object) to begin processing.
Start processing from the root node or from a specified nested node.
Starting Field Namestarting-field-nameSkips forward to the given nested JSON field (array or object) to begin processing.

This Property is only considered if the [Starting Field Strategy] Property has a value of "Nested Field".
Schema Application Strategyschema-application-strategySelected Part
  • Whole JSON Applies the schema for the whole JSON.
  • Selected Part Applies the schema for the selected part starting from the "Starting Field Name".
Specifies whether the schema is defined for the whole JSON or for the selected part starting from "Starting Field Name".

This Property is only considered if all of the following conditions are met:
  • The [Schema Access Strategy] Property is set to one of the following values: [Confluent Content-Encoded Schema Reference], [Use 'Schema Name' Property], [Use 'Schema Text' Property], [HWX Schema Reference Attributes], [HWX Content-Encoded Schema Reference]
  • The [Starting Field Strategy] Property has a value of "Nested Field".
Max String LengthMax String Length20 MBThe maximum allowed length of a string value when parsing the JSON document
Allow CommentsAllow Commentsfalse
  • true
  • false
Whether to allow comments when parsing the JSON document
Date FormatDate FormatSpecifies the format to use when reading/writing Date fields. If not specified, Date fields will be assumed to be number of milliseconds since epoch (Midnight, Jan 1, 1970 GMT). If specified, the value must match the Java Simple Date Format (for example, MM/dd/yyyy for a two-digit month, followed by a two-digit day, followed by a four-digit year, all separated by '/' characters, as in 01/01/2017).
Time FormatTime FormatSpecifies the format to use when reading/writing Time fields. If not specified, Time fields will be assumed to be number of milliseconds since epoch (Midnight, Jan 1, 1970 GMT). If specified, the value must match the Java Simple Date Format (for example, HH:mm:ss for a two-digit hour in 24-hour format, followed by a two-digit minute, followed by a two-digit second, all separated by ':' characters, as in 18:04:15).
Timestamp FormatTimestamp FormatSpecifies the format to use when reading/writing Timestamp fields. If not specified, Timestamp fields will be assumed to be number of milliseconds since epoch (Midnight, Jan 1, 1970 GMT). If specified, the value must match the Java Simple Date Format (for example, MM/dd/yyyy HH:mm:ss for a two-digit month, followed by a two-digit day, followed by a four-digit year, all separated by '/' characters; and then followed by a two-digit hour in 24-hour format, followed by a two-digit minute, followed by a two-digit second, all separated by ':' characters, as in 01/01/2017 18:04:15).

State management:

This component does not store state.

Restricted:

This component is not restricted.

System Resource Considerations:

None specified.

See Also:

JsonPathReader