PutS3Object 2.0.0

Bundle
org.apache.nifi | nifi-aws-nar
Description
Writes the contents of a FlowFile as an S3 Object to an Amazon S3 Bucket.
Tags
AWS, Amazon, Archive, Put, S3
Input Requirement
REQUIRED
Supports Sensitive Dynamic Properties
false
  • Additional Details for PutS3Object 2.0.0

    PutS3Object

    Multi-part Upload Details

    The upload uses either the PutS3Object method or the PutS3MultipartUpload method. The PutS3Object method sends the file in a single synchronous call, but it has a 5GB size limit. Larger files are sent using the PutS3MultipartUpload method. This multipart process saves state after each step so that a large upload can be resumed with minimal loss if the processor or cluster is stopped and restarted. A multipart upload consists of three steps:

    1. Initiate upload
    2. Upload the parts
    3. Complete the upload

    For multipart uploads, the processor saves state locally tracking the upload ID and parts uploaded, which must both be provided to complete the upload. The AWS libraries select an endpoint URL based on the AWS region, but this can be overridden with the ‘Endpoint Override URL’ property for use with other S3-compatible endpoints. The S3 API specifies that the maximum file size for a PutS3Object upload is 5GB. It also requires that parts in a multipart upload must be at least 5MB in size, except for the last part. These limits establish the bounds for the Multipart Upload Threshold and Part Size properties.

    Configuration Details

    Object Key

    The Object Key property value should not start with “/”.

    Credentials File

    The Credentials File property allows the user to specify the path to a file containing the AWS access key and secret key. The contents of the file should be in the following format:

        [default]
    accessKey=<access key>
    secretKey=<security key>

    Make sure the credentials file is readable by the NiFi service user.

    When using the Credential File property, ensure that there are no values for the Access Key and Secret Key properties. The Value column should read “No value set” for both. Note: Do not check “Set empty string” for either as the empty string is considered a set value.

Properties
Dynamic Properties
Relationships
Name Description
failure If the Processor is unable to process a given FlowFile, it will be routed to this Relationship.
success FlowFiles are routed to this Relationship after they have been successfully processed.
Reads Attributes
Name Description
filename Uses the FlowFile's filename as the filename for the S3 object
Writes Attributes
Name Description
s3.url The URL that can be used to access the S3 object
s3.bucket The S3 bucket where the Object was put in S3
s3.key The S3 key within where the Object was put in S3
s3.contenttype The S3 content type of the S3 Object that put in S3
s3.version The version of the S3 Object that was put to S3
s3.exception The class name of the exception thrown during processor execution
s3.additionalDetails The S3 supplied detail from the failed operation
s3.statusCode The HTTP error code (if available) from the failed operation
s3.errorCode The S3 moniker of the failed operation
s3.errorMessage The S3 exception message from the failed operation
s3.etag The ETag of the S3 Object
s3.contentdisposition The content disposition of the S3 Object that put in S3
s3.cachecontrol The cache-control header of the S3 Object
s3.uploadId The uploadId used to upload the Object to S3
s3.expiration A human-readable form of the expiration date of the S3 object, if one is set
s3.sseAlgorithm The server side encryption algorithm of the object
s3.usermetadata A human-readable form of the User Metadata of the S3 object, if any was set
s3.encryptionStrategy The name of the encryption strategy, if any was set
See Also