PutDynamoDBRecord

Description:

Inserts items into DynamoDB based on record-oriented data. The record fields are mapped into DynamoDB item fields, including partition and sort keys if set. Depending on the number of records the processor might execute the insert in multiple chunks in order to overcome DynamoDB's limitation on batch writing. This might result partially processed FlowFiles in which case the FlowFile will be transferred to the "unprocessed" relationship with the necessary attribute to retry later without duplicating the already executed inserts.

Additional Details...

Tags:

Amazon, DynamoDB, AWS, Put, Insert, Record

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.

Display NameAPI NameDefault ValueAllowable ValuesDescription
Record Readerrecord-readerController Service API:
RecordReaderFactory
Implementations: JsonTreeReader
GrokReader
AvroReader
SyslogReader
ExcelReader
XMLReader
Syslog5424Reader
CSVReader
CEFReader
JsonPathReader
ReaderLookup
ScriptedReader
ParquetReader
WindowsEventLogReader		Specifies the Controller Service to use for parsing incoming data and determining the data's schema.
AWS Credentials Provider ServiceAWS Credentials Provider serviceController Service API:
AWSCredentialsProviderService
Implementation: AWSCredentialsProviderControllerService		The Controller Service that is used to obtain AWS credentials provider
RegionRegionUS West (Oregon)
  • AWS GovCloud (US) AWS Region Code : us-gov-west-1
  • AWS GovCloud (US-East) AWS Region Code : us-gov-east-1
  • US East (N. Virginia) AWS Region Code : us-east-1
  • US East (Ohio) AWS Region Code : us-east-2
  • US West (N. California) AWS Region Code : us-west-1
  • US West (Oregon) AWS Region Code : us-west-2
  • EU (Ireland) AWS Region Code : eu-west-1
  • EU (London) AWS Region Code : eu-west-2
  • EU (Paris) AWS Region Code : eu-west-3
  • EU (Frankfurt) AWS Region Code : eu-central-1
  • EU (Zurich) AWS Region Code : eu-central-2
  • EU (Stockholm) AWS Region Code : eu-north-1
  • EU (Milan) AWS Region Code : eu-south-1
  • EU (Spain) AWS Region Code : eu-south-2
  • Asia Pacific (Hong Kong) AWS Region Code : ap-east-1
  • Asia Pacific (Mumbai) AWS Region Code : ap-south-1
  • Asia Pacific (Hyderabad) AWS Region Code : ap-south-2
  • Asia Pacific (Singapore) AWS Region Code : ap-southeast-1
  • Asia Pacific (Sydney) AWS Region Code : ap-southeast-2
  • Asia Pacific (Jakarta) AWS Region Code : ap-southeast-3
  • Asia Pacific (Melbourne) AWS Region Code : ap-southeast-4
  • Asia Pacific (Tokyo) AWS Region Code : ap-northeast-1
  • Asia Pacific (Seoul) AWS Region Code : ap-northeast-2
  • Asia Pacific (Osaka) AWS Region Code : ap-northeast-3
  • South America (Sao Paulo) AWS Region Code : sa-east-1
  • China (Beijing) AWS Region Code : cn-north-1
  • China (Ningxia) AWS Region Code : cn-northwest-1
  • Canada (Central) AWS Region Code : ca-central-1
  • Middle East (UAE) AWS Region Code : me-central-1
  • Middle East (Bahrain) AWS Region Code : me-south-1
  • Africa (Cape Town) AWS Region Code : af-south-1
  • US ISO East AWS Region Code : us-iso-east-1
  • US ISOB East (Ohio) AWS Region Code : us-isob-east-1
  • US ISO West AWS Region Code : us-iso-west-1
The AWS Region to connect to.
Table NameTable NameThe DynamoDB table name
Supports Expression Language: true (will be evaluated using variable registry only)
Partition Key Strategypartition-key-strategyPartition By Field
  • Partition By Field Uses the value of the Record field identified by the "Partition Key Field" property as partition key value.
  • Partition By Attribute Uses an incoming FlowFile attribute identified by "Partition Key Attribute" as the value of the partition key. The incoming Records must not contain field with the same name defined by the "Partition Key Field".
  • Generated UUID Uses a generated UUID as value for the partition key. The incoming Records must not contain field with the same name defined by the "Partition Key Field".
Defines the strategy the processor uses to assign partition key value to the inserted Items.
Partition Key Fieldpartition-key-fieldDefines the name of the partition key field in the DynamoDB table. Partition key is also known as hash key. Depending on the "Partition Key Strategy" the field value might come from the incoming Record or a generated one.
Supports Expression Language: true (will be evaluated using variable registry only)
Partition Key Attributepartition-key-attributeSpecifies the FlowFile attribute that will be used as the value of the partition key when using "Partition by attribute" partition key strategy.
Supports Expression Language: true (will be evaluated using variable registry only)

This Property is only considered if the [Partition Key Strategy] Property has a value of "Partition By Attribute".
Sort Key Strategysort-key-strategyNone
  • None The processor will not assign sort key to the inserted Items.
  • Sort By Field Uses the value of the Record field identified by the "Sort Key Field" property as sort key value.
  • Generate Sequence The processor will assign a number for every item based on the original record's position in the incoming FlowFile. This will be used as sort key value.
Defines the strategy the processor uses to assign sort key to the inserted Items.
Sort Key Fieldsort-key-fieldDefines the name of the sort key field in the DynamoDB table. Sort key is also known as range key.
Supports Expression Language: true (will be evaluated using variable registry only)

This Property is only considered if the [Sort Key Strategy] Property is set to one of the following values: [Sort By Field], [Generate Sequence]
Communications TimeoutCommunications Timeout30 secsThe amount of time to wait in order to establish a connection to AWS or receive data from AWS before timing out.
Proxy Configuration Serviceproxy-configuration-serviceController Service API:
ProxyConfigurationService
Implementation: StandardProxyConfigurationService		Specifies the Proxy Configuration Controller Service to proxy network requests. If set, it supersedes proxy settings configured per component.
SSL Context ServiceSSL Context ServiceController Service API:
SSLContextService
Implementations: StandardRestrictedSSLContextService
StandardSSLContextService		Specifies an optional SSL Context Service that, if provided, will be used to create connections

Relationships:

NameDescription
unprocessedFlowFiles are routed to unprocessed relationship when DynamoDB is not able to process all the items in the request. Typical reasons are insufficient table throughput capacity and exceeding the maximum bytes per request. Unprocessed FlowFiles can be retried with a new request.
successFlowFiles are routed to success relationship
failureFlowFiles are routed to failure relationship

Reads Attributes:

NameDescription
dynamodb.chunks.processedNumber of chunks successfully inserted into DynamoDB. If not set, it is considered as 0

Writes Attributes:

NameDescription
dynamodb.chunks.processedNumber of chunks successfully inserted into DynamoDB. If not set, it is considered as 0
dynamodb.key.error.unprocessedDynamoDB unprocessed keys
dynmodb.range.key.value.errorDynamoDB range key error
dynamodb.key.error.not.foundDynamoDB key not found
dynamodb.error.exception.messageDynamoDB exception message
dynamodb.error.codeDynamoDB error code
dynamodb.error.messageDynamoDB error message
dynamodb.error.typeDynamoDB error type
dynamodb.error.serviceDynamoDB error service
dynamodb.error.retryableDynamoDB error is retryable
dynamodb.error.request.idDynamoDB error request id
dynamodb.error.status.codeDynamoDB error status code
dynamodb.item.io.errorIO exception message on creating item

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component requires an incoming relationship.

System Resource Considerations:

ResourceDescription
MEMORYAn instance of this component can cause high usage of this system resource. Multiple instances or high concurrency settings may result a degradation of performance.
NETWORKAn instance of this component can cause high usage of this system resource. Multiple instances or high concurrency settings may result a degradation of performance.

See Also:

DeleteDynamoDB, GetDynamoDB, PutDynamoDB