ListGCSBucket 2.0.0

Bundle
org.apache.nifi | nifi-gcp-nar
Description
Retrieves a listing of objects from a GCS bucket. For each object that is listed, creates a FlowFile that represents the object so that it can be fetched in conjunction with FetchGCSObject. This Processor is designed to run on Primary Node only in a cluster. If the primary node changes, the new Primary Node will pick up where the previous node left off without duplicating all of the data.
Tags
gcs, google, google cloud, list, storage
Input Requirement
FORBIDDEN
Supports Sensitive Dynamic Properties
false
  • Additional Details for ListGCSBucket 2.0.0

    ListGCSBucket

    Streaming Versus Batch Processing

    ListGCSBucket performs a listing of all GCS Objects that it encounters in the configured GCS bucket. There are two common, broadly defined use cases.

    Streaming Use Case

    By default, the Processor will create a separate FlowFile for each object in the bucket and add attributes for filename, bucket, etc. A common use case is to connect ListGCSBucket to the FetchGCSObject processor. These two processors used in conjunction with one another provide the ability to easily monitor a bucket and fetch the contents of any new object as it lands in GCS in an efficient streaming fashion.

    Batch Use Case

    Another common use case is the desire to process all newly arriving objects in a given bucket, and to then perform some action only when all objects have completed their processing. The above approach of streaming the data makes this difficult, because NiFi is inherently a streaming platform in that there is no “job” that has a beginning and an end. Data is simply picked up as it becomes available.

    To solve this, the ListGCSBucket Processor can optionally be configured with a Record Writer. When a Record Writer is configured, a single FlowFile will be created that will contain a Record for each object in the bucket, instead of a separate FlowFile per object. See the documentation for ListFile for an example of how to build a dataflow that allows for processing all the objects before proceeding with any other step.

    One important difference between the data produced by ListFile and ListGCSBucket, though, is the structure of the Records that are emitted. The Records emitted by ListFile have a different schema than those emitted by ListGCSBucket. ListGCSBucket emits records that follow the following schema (in Avro format):

    {
  "type": "record",
  "name": "nifiRecord",
  "namespace": "org.apache.nifi",
  "fields": [
    {
      "name": "bucket",
      "type": "string"
    },
    {
      "name": "name",
      "type": "string"
    },
    {
      "name": "size",
      "type": [
        "null",
        "long"
      ]
    },
    {
      "name": "cacheControl",
      "type": [
        "null",
        "string"
      ]
    },
    {
      "name": "componentCount",
      "type": [
        "null",
        "int"
      ]
    },
    {
      "name": "contentDisposition",
      "type": [
        "null",
        "long"
      ]
    },
    {
      "name": "contentEncoding",
      "type": [
        "null",
        "string"
      ]
    },
    {
      "name": "contentLanguage",
      "type": [
        "null",
        "string"
      ]
    },
    {
      "name": "crc32c",
      "type": [
        "null",
        "string"
      ]
    },
    {
      "name": "createTime",
      "type": [
        "null",
        {
          "type": "long",
          "logicalType": "timestamp-millis"
        }
      ]
    },
    {
      "name": "updateTime",
      "type": [
        "null",
        {
          "type": "long",
          "logicalType": "timestamp-millis"
        }
      ]
    },
    {
      "name": "encryptionAlgorithm",
      "type": [
        "null",
        "string"
      ]
    },
    {
      "name": "encryptionKeySha256",
      "type": [
        "null",
        "string"
      ]
    },
    {
      "name": "etag",
      "type": [
        "null",
        "string"
      ]
    },
    {
      "name": "generatedId",
      "type": [
        "null",
        "string"
      ]
    },
    {
      "name": "generation",
      "type": [
        "null",
        "long"
      ]
    },
    {
      "name": "md5",
      "type": [
        "null",
        "string"
      ]
    },
    {
      "name": "mediaLink",
      "type": [
        "null",
        "string"
      ]
    },
    {
      "name": "metageneration",
      "type": [
        "null",
        "long"
      ]
    },
    {
      "name": "owner",
      "type": [
        "null",
        "string"
      ]
    },
    {
      "name": "ownerType",
      "type": [
        "null",
        "string"
      ]
    },
    {
      "name": "uri",
      "type": [
        "null",
        "string"
      ]
    }
  ]
}
Properties
State Management
Scopes Description
CLUSTER After performing a listing of keys, the timestamp of the newest key is stored, along with the keys that share that same timestamp. This allows the Processor to list only keys that have been added or modified after this date the next time that the Processor is run. State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected, the new node can pick up where the previous node left off, without duplicating the data.
Relationships
Name Description
success FlowFiles are routed to this relationship after a successful Google Cloud Storage operation.
Writes Attributes
Name Description
filename The name of the file
gcs.bucket Bucket of the object.
gcs.key Name of the object.
gcs.size Size of the object.
gcs.cache.control Data cache control of the object.
gcs.component.count The number of components which make up the object.
gcs.content.disposition The data content disposition of the object.
gcs.content.encoding The content encoding of the object.
gcs.content.language The content language of the object.
mime.type The MIME/Content-Type of the object
gcs.crc32c The CRC32C checksum of object's data, encoded in base64 in big-endian order.
gcs.create.time The creation time of the object (milliseconds)
gcs.update.time The last modification time of the object (milliseconds)
gcs.encryption.algorithm The algorithm used to encrypt the object.
gcs.encryption.sha256 The SHA256 hash of the key used to encrypt the object
gcs.etag The HTTP 1.1 Entity tag for the object.
gcs.generated.id The service-generated for the object
gcs.generation The data generation of the object.
gcs.md5 The MD5 hash of the object's data encoded in base64.
gcs.media.link The media download link to the object.
gcs.metageneration The metageneration of the object.
gcs.owner The owner (uploader) of the object.
gcs.owner.type The ACL entity type of the uploader of the object.
gcs.uri The URI of the object as a string.
See Also