LookupRecord 2.0.0

Bundle
org.apache.nifi | nifi-standard-nar
Description
Extracts one or more fields from a Record and looks up a value for those fields in a LookupService. If a result is returned by the LookupService, that result is optionally added to the Record. In this case, the processor functions as an Enrichment processor. Regardless, the Record is then routed to either the 'matched' relationship or 'unmatched' relationship (if the 'Routing Strategy' property is configured to do so), indicating whether or not a result was returned by the LookupService, allowing the processor to also function as a Routing processor. The "coordinates" to use for looking up a value in the Lookup Service are defined by adding a user-defined property. Each property that is added will have an entry added to a Map, where the name of the property becomes the Map Key and the value returned by the RecordPath becomes the value for that key. If multiple values are returned by the RecordPath, then the Record will be routed to the 'unmatched' relationship (or 'success', depending on the 'Routing Strategy' property's configuration). If one or more fields match the Result RecordPath, all fields that match will be updated. If there is no match in the configured LookupService, then no fields will be updated. I.e., it will not overwrite an existing value in the Record with a null value. Please note, however, that if the results returned by the LookupService are not accounted for in your schema (specifically, the schema that is configured for your Record Writer) then the fields will not be written out to the FlowFile.
Tags
avro, convert, csv, database, db, enrichment, filter, json, logs, lookup, record, route
Input Requirement
REQUIRED
Supports Sensitive Dynamic Properties
false
  • Additional Details for LookupRecord 2.0.0

    LookupRecord

    LookupRecord makes use of the NiFi RecordPath Domain-Specific Language (DSL) to allow the user to indicate which field(s), depending on the Record Update Strategy, in the Record should be updated. The Record will be updated using the value returned by the provided Lookup Service.

    Record Update Strategy - Use Property

    In this case, the user should add, to the Processor’s configuration, as much User-defined Properties as required by the Lookup Service to form the lookup coordinates. The name of the properties should match the names expected by the Lookup Service.

    The field evaluated using the path configured in the “Result RecordPath” property will be the field updated with the value returned by the Lookup Service.

    Let’s assume a Simple Key Value Lookup Service containing the following key/value pairs:

    FR => France CA => Canada

    Let’s assume the following JSON with three records as input:

    [
  {
    "country": null,
    "code": "FR"
  },
  {
    "country": null,
    "code": "CA"
  },
  {
    "country": null,
    "code": "JP"
  }
]

    The processor is configured with “Use Property” as “Record Update Strategy”, the “Result RecordPath” is configured with “/country” and a user-defined property is added with the name “key” (as required by this Lookup Service) and the value “/code”.

    When triggered, the processor will look for the value associated to the “/code” path and will use the value as the “key” of the Lookup Service. The value returned by the Lookup Service will be used to update the value corresponding to " /country". With the above examples, it will produce:

    [
  {
    "country": "France",
    "code": "FR"
  },
  {
    "country": "Canada",
    "code": "CA"
  },
  {
    "country": null,
    "code": "JP"
  }
]

    Record Update Strategy - Replace Existing Values

    With this strategy, the “Result RecordPath” property will be ignored and the configured Lookup Service must be a single simple key lookup service. For each user-defined property, the value contained in the field corresponding to the record path will be used as the key in the Lookup Service and will be replaced by the value returned by the Lookup Service. It is possible to configure multiple dynamic properties to update multiple fields in one execution. This strategy only supports simple types replacements (strings, integers, etc).

    Since this strategy allows in-place replacement, it is possible to use Record Paths for fields contained in arrays.

    Let’s assume a Simple Key Value Lookup Service containing the following key/value pairs:

    FR => France CA => Canada fr => French en => English

    Let’s assume the following JSON with two records as input:

    [
  {
    "locales": [
      {
        "region": "FR",
        "language": "fr"
      },
      {
        "region": "US",
        "language": "en"
      }
    ]
  },
  {
    "locales": [
      {
        "region": "CA",
        "language": "fr"
      },
      {
        "region": "JP",
        "language": "ja"
      }
    ]
  }
]

    The processor is configured with “Replace Existing Values” as “Record Update Strategy”, two user-defined properties are added: “region” => “/locales[*]/region” and “language => “/locales[*]/language”..

    When triggered, the processor will loop over the user-defined properties. First, it’ll search for the fields corresponding to “/locales[*]/region”, for each value from the record, the value will be used as the key with the Lookup Service and the value will be replaced by the result returned by the Lookup Service. Example: the first region is “FR” and this key is associated to the value “France” in the Lookup Service, so the value “FR” is replaced by " France” in the record. With the above examples, it will produce:

    [
  {
    "locales": [
      {
        "region": "France",
        "language": "French"
      },
      {
        "region": "US",
        "language": "English"
      }
    ]
  },
  {
    "locales": [
      {
        "region": "Canada",
        "language": "French"
      },
      {
        "region": "JP",
        "language": "ja"
      }
    ]
  }
]
Properties
Dynamic Properties
Relationships
Name Description
failure If a FlowFile cannot be enriched, the unchanged FlowFile will be routed to this relationship
success All records will be sent to this Relationship if configured to do so, unless a failure occurs
Writes Attributes
Name Description
mime.type Sets the mime.type attribute to the MIME Type specified by the Record Writer
record.count The number of records in the FlowFile
See Also