GetS3ObjectMetadata

Description:

Check for the existence of a file in S3 without attempting to download it. This processor can be used as a router for work flows that need to check on a file in S3 before proceeding with data processing

Tags:

Amazon, S3, AWS, Archive, Exists

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.

Display NameAPI NameDefault ValueAllowable ValuesDescription
Metadata TargetMetadata TargetAttributes
  • Attributes When selected, the metadata will be written to FlowFile attributes with the prefix "s3." following the convention used in other processors. For example: the standard S3 attribute Content-Type will be written as s3.Content-Type when using the default value. User-defined metadatawill be included in the attributes added to the FlowFile
  • FlowFile Body Write the metadata to FlowFile content as JSON data.
This determines where the metadata will be written when found.
Metadata Attribute Include PatternMetadata Attribute Include Pattern.*A regular expression pattern to use for determining which object metadata entries are included as FlowFile attributes. This pattern is only applied to the 'found' relationship and will not be used to filter the error attributes in the 'failure' relationship.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)

This Property is only considered if the [Metadata Target] Property has a value of "Attributes".
BucketBucketThe S3 Bucket to interact with
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Object KeyObject Key${filename}The S3 Object Key to use. This is analogous to a filename for traditional file systems.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
AWS Credentials Provider ServiceAWS Credentials Provider serviceController Service API:
AWSCredentialsProviderService
Implementation: AWSCredentialsProviderControllerService
The Controller Service that is used to obtain AWS credentials provider
RegionRegionUS West (Oregon)
  • AWS GovCloud (US) AWS Region Code : us-gov-west-1
  • AWS GovCloud (US-East) AWS Region Code : us-gov-east-1
  • US East (N. Virginia) AWS Region Code : us-east-1
  • US East (Ohio) AWS Region Code : us-east-2
  • US West (N. California) AWS Region Code : us-west-1
  • US West (Oregon) AWS Region Code : us-west-2
  • EU (Ireland) AWS Region Code : eu-west-1
  • EU (London) AWS Region Code : eu-west-2
  • EU (Paris) AWS Region Code : eu-west-3
  • EU (Frankfurt) AWS Region Code : eu-central-1
  • EU (Zurich) AWS Region Code : eu-central-2
  • EU (Stockholm) AWS Region Code : eu-north-1
  • EU (Milan) AWS Region Code : eu-south-1
  • EU (Spain) AWS Region Code : eu-south-2
  • Asia Pacific (Hong Kong) AWS Region Code : ap-east-1
  • Asia Pacific (Mumbai) AWS Region Code : ap-south-1
  • Asia Pacific (Hyderabad) AWS Region Code : ap-south-2
  • Asia Pacific (Singapore) AWS Region Code : ap-southeast-1
  • Asia Pacific (Sydney) AWS Region Code : ap-southeast-2
  • Asia Pacific (Jakarta) AWS Region Code : ap-southeast-3
  • Asia Pacific (Melbourne) AWS Region Code : ap-southeast-4
  • Asia Pacific (Tokyo) AWS Region Code : ap-northeast-1
  • Asia Pacific (Seoul) AWS Region Code : ap-northeast-2
  • Asia Pacific (Osaka) AWS Region Code : ap-northeast-3
  • South America (Sao Paulo) AWS Region Code : sa-east-1
  • China (Beijing) AWS Region Code : cn-north-1
  • China (Ningxia) AWS Region Code : cn-northwest-1
  • Canada (Central) AWS Region Code : ca-central-1
  • Canada West (Calgary) AWS Region Code : ca-west-1
  • Middle East (UAE) AWS Region Code : me-central-1
  • Middle East (Bahrain) AWS Region Code : me-south-1
  • Africa (Cape Town) AWS Region Code : af-south-1
  • US ISO East AWS Region Code : us-iso-east-1
  • US ISOB East (Ohio) AWS Region Code : us-isob-east-1
  • US ISO West AWS Region Code : us-iso-west-1
  • Israel (Tel Aviv) AWS Region Code : il-central-1
  • Use 's3.region' Attribute Uses 's3.region' FlowFile attribute as region.
The AWS Region to connect to.
Communications TimeoutCommunications Timeout30 secsThe amount of time to wait in order to establish a connection to AWS or receive data from AWS before timing out.
FullControl User ListFullControl User List${s3.permissions.full.users}A comma-separated list of Amazon User ID's or E-mail addresses that specifies who should have Full Control for an object
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Read Permission User ListRead Permission User List${s3.permissions.read.users}A comma-separated list of Amazon User ID's or E-mail addresses that specifies who should have Read Access for an object
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Read ACL User ListRead ACL User List${s3.permissions.readacl.users}A comma-separated list of Amazon User ID's or E-mail addresses that specifies who should have permissions to read the Access Control List for an object
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
OwnerOwner${s3.owner}The Amazon ID to use for the object's owner
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
SSL Context ServiceSSL Context ServiceController Service API:
SSLContextService
Implementations: StandardSSLContextService
StandardRestrictedSSLContextService
Specifies an optional SSL Context Service that, if provided, will be used to create connections
Endpoint Override URLEndpoint Override URLEndpoint URL to use instead of the AWS default including scheme, host, port, and path. The AWS libraries select an endpoint URL based on the AWS region, but this property overrides the selected endpoint URL, allowing use with other S3-compatible endpoints.
Supports Expression Language: true (will be evaluated using variable registry only)
Signer OverrideSigner OverrideDefault Signature
  • Default Signature
  • Signature Version 4
  • Signature Version 2
  • Custom Signature
The AWS S3 library uses Signature Version 4 by default but this property allows you to specify the Version 2 signer to support older S3-compatible services or even to plug in your own custom signer implementation.
Custom Signer Class Namecustom-signer-class-nameFully qualified class name of the custom signer class. The signer must implement com.amazonaws.auth.Signer interface.
Supports Expression Language: true (will be evaluated using variable registry only)

This Property is only considered if the [Signer Override] Property has a value of "Custom Signature".
Custom Signer Module Locationcustom-signer-module-locationComma-separated list of paths to files and/or directories which contain the custom signer's JAR file and its dependencies (if any).

This property expects a comma-separated list of resources. Each of the resources may be of any of the following types: file, directory.

Supports Expression Language: true (will be evaluated using variable registry only)

This Property is only considered if the [Signer Override] Property has a value of "Custom Signature".
Proxy Configuration Serviceproxy-configuration-serviceController Service API:
ProxyConfigurationService
Implementation: StandardProxyConfigurationService
Specifies the Proxy Configuration Controller Service to proxy network requests. If set, it supersedes proxy settings configured per component. Supported proxies: HTTP + AuthN

Relationships:

NameDescription
not foundNo object was found in the bucket the supplied key
failureFlowFiles are routed to failure relationship
foundAn object was found in the bucket at the supplied key

Reads Attributes:

None specified.

Writes Attributes:

None specified.

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component requires an incoming relationship.

System Resource Considerations:

None specified.

See Also:

PutS3Object, DeleteS3Object, ListS3, TagS3Object, DeleteS3Object, FetchS3Object