QuerySalesforceObject 2.0.0

Bundle
org.apache.nifi | nifi-salesforce-nar
Description
Retrieves records from a Salesforce sObject. Users can add arbitrary filter conditions by setting the 'Custom WHERE Condition' property. The processor can also run a custom query, although record processing is not supported in that case. Supports incremental retrieval: users can define a field in the 'Age Field' property that will be used to determine when the record was created. When this property is set the processor will retrieve new records. Incremental loading and record-based processing are only supported in property-based queries. It's also possible to define an initial cutoff value for the age, filtering out all older records even for the first run. In case of 'Property Based Query' this processor should run on the Primary Node only. FlowFile attribute 'record.count' indicates how many records were retrieved and written to the output. The processor can accept an optional input FlowFile and reference the FlowFile attributes in the query. When 'Include Deleted Records' is true, the processor will include deleted records (soft-deletes) in the results by using the 'queryAll' API. The 'IsDeleted' field will be automatically included in the results when querying deleted records.
Tags
query, salesforce, sobject, soql
Input Requirement
ALLOWED
Supports Sensitive Dynamic Properties
false
  • Additional Details for QuerySalesforceObject 2.0.0

    QuerySalesforceObject

    Description

    Objects in Salesforce are database tables, their rows are known as records, and their columns are called fields. The QuerySalesforceObject processor queries Salesforce objects and retrieves their records. The processor constructs the query from processor properties or executes a custom SOQL (Salesforce Object Query Language) query and retrieves the result record dataset using the Salesforce REST API. The ‘Query Type’ processor property allows the query to be built in two ways. The ‘Property Based Query’ option allows to define a ‘SELECT from ’ type query, with the fields defined in the ‘Field Names’ property and the Salesforce object defined in the ‘sObject Name’ property, whereas the ‘Custom Query’ option allows you to supply an arbitrary SOQL query. By using ‘Custom Query’, the processor can accept an optional input flowfile and reference the flowfile attributes in the query. However, incremental loading and record-based processing are only supported in ‘Property Based Queries’.

    OAuth2 Access Token Provider Service

    The OAuth2 Access Token Provider Service handles Salesforce REST API authorization. In order to use OAuth2 authorization, create a new StandardOauth2AccessTokenProvider service and configure it as follows.

    • Authorization Server URL: It is the concatenation of the Salesforce URL and the token request service URL ( /services/oauth2/token).
    • Grant Type: User Password.
    • Username: The email address registered in the Salesforce account.
    • Password: For the Password a Security token must be requested. Go to Profile -> Settings and under the Reset My Security Token option, request one, which will be sent to the registered email address. The password is made up of the Salesforce account password and the Security token concatenated together without a space.
    • Client ID: Create a new Connected App within Salesforce. Go to Setup -> On the left search panel find App Manager -> Create New Connected App. Once it’s done, the Consumer Key goes to the Client ID property.
    • Client Secret: Available on the Connected App page under Consumer Secret.

    Age properties

    The age properties are important to avoid processing duplicate records. Age filtering provides a sliding window that starts with the processor’s prior run time and ends with the current run time minus the age delay. Only records that are within the sliding window are queried and processed. On the processor, the Age Field property must be a datetime field of the queried object, this will be subject to the condition that it is greater than the processor’s previous but less than the current run time (e.g. LastModifiedDate). The first run, for example, will query records whose LastModifiedDate field is earlier than the current run time. The second will look for records with LastModifiedDate fields that are later than the previous run time but earlier than the current run time.

    The processor uses the Initial Age Filter as a specific timestamp that sets the beginning of the sliding window from which processing builds the initial query. The format must adhere to the Salesforce SOQL standards (see Salesforce documentation). The Age Delay moves the time of the records to be processed earlier than the current run time if necessary.

Properties
State Management
Scopes Description
CLUSTER When 'Age Field' is set, after performing a query the time of execution is stored. Subsequent queries will be augmented with an additional condition so that only records that are newer than the stored execution time (adjusted with the optional value of 'Age Delay') will be retrieved. State is stored across the cluster so that this Processor can be run on Primary Node only and if a new Primary Node is selected, the new node can pick up where the previous node left off, without duplicating the data.
Relationships
Name Description
success For FlowFiles created as a result of a successful query.
failure The input flowfile gets sent to this relationship when the query fails.
original The input flowfile gets sent to this relationship when the query succeeds.
Writes Attributes
Name Description
mime.type Sets the mime.type attribute to the MIME Type specified by the Record Writer.
record.count Sets the number of records in the FlowFile.
total.record.count Sets the total number of records in the FlowFile.
See Also