-
Processors
- AttributeRollingWindow
- AttributesToCSV
- AttributesToJSON
- CalculateRecordStats
- CaptureChangeMySQL
- CompressContent
- ConnectWebSocket
- ConsumeAMQP
- ConsumeAzureEventHub
- ConsumeElasticsearch
- ConsumeGCPubSub
- ConsumeIMAP
- ConsumeJMS
- ConsumeKafka
- ConsumeKinesisStream
- ConsumeMQTT
- ConsumePOP3
- ConsumeSlack
- ConsumeTwitter
- ConsumeWindowsEventLog
- ControlRate
- ConvertCharacterSet
- ConvertRecord
- CopyAzureBlobStorage_v12
- CopyS3Object
- CountText
- CryptographicHashContent
- DebugFlow
- DecryptContentAge
- DecryptContentPGP
- DeduplicateRecord
- DeleteAzureBlobStorage_v12
- DeleteAzureDataLakeStorage
- DeleteByQueryElasticsearch
- DeleteDynamoDB
- DeleteFile
- DeleteGCSObject
- DeleteGridFS
- DeleteMongo
- DeleteS3Object
- DeleteSFTP
- DeleteSQS
- DetectDuplicate
- DistributeLoad
- DuplicateFlowFile
- EncodeContent
- EncryptContentAge
- EncryptContentPGP
- EnforceOrder
- EvaluateJsonPath
- EvaluateXPath
- EvaluateXQuery
- ExecuteGroovyScript
- ExecuteProcess
- ExecuteScript
- ExecuteSQL
- ExecuteSQLRecord
- ExecuteStreamCommand
- ExtractAvroMetadata
- ExtractEmailAttachments
- ExtractEmailHeaders
- ExtractGrok
- ExtractHL7Attributes
- ExtractRecordSchema
- ExtractText
- FetchAzureBlobStorage_v12
- FetchAzureDataLakeStorage
- FetchBoxFile
- FetchDistributedMapCache
- FetchDropbox
- FetchFile
- FetchFTP
- FetchGCSObject
- FetchGoogleDrive
- FetchGridFS
- FetchS3Object
- FetchSFTP
- FetchSmb
- FilterAttribute
- FlattenJson
- ForkEnrichment
- ForkRecord
- GenerateFlowFile
- GenerateRecord
- GenerateTableFetch
- GeoEnrichIP
- GeoEnrichIPRecord
- GeohashRecord
- GetAsanaObject
- GetAwsPollyJobStatus
- GetAwsTextractJobStatus
- GetAwsTranscribeJobStatus
- GetAwsTranslateJobStatus
- GetAzureEventHub
- GetAzureQueueStorage_v12
- GetDynamoDB
- GetElasticsearch
- GetFile
- GetFTP
- GetGcpVisionAnnotateFilesOperationStatus
- GetGcpVisionAnnotateImagesOperationStatus
- GetHubSpot
- GetMongo
- GetMongoRecord
- GetS3ObjectMetadata
- GetSFTP
- GetShopify
- GetSmbFile
- GetSNMP
- GetSplunk
- GetSQS
- GetWorkdayReport
- GetZendesk
- HandleHttpRequest
- HandleHttpResponse
- IdentifyMimeType
- InvokeHTTP
- InvokeScriptedProcessor
- ISPEnrichIP
- JoinEnrichment
- JoltTransformJSON
- JoltTransformRecord
- JSLTTransformJSON
- JsonQueryElasticsearch
- ListAzureBlobStorage_v12
- ListAzureDataLakeStorage
- ListBoxFile
- ListDatabaseTables
- ListDropbox
- ListenFTP
- ListenHTTP
- ListenOTLP
- ListenSlack
- ListenSyslog
- ListenTCP
- ListenTrapSNMP
- ListenUDP
- ListenUDPRecord
- ListenWebSocket
- ListFile
- ListFTP
- ListGCSBucket
- ListGoogleDrive
- ListS3
- ListSFTP
- ListSmb
- LogAttribute
- LogMessage
- LookupAttribute
- LookupRecord
- MergeContent
- MergeRecord
- ModifyBytes
- ModifyCompression
- MonitorActivity
- MoveAzureDataLakeStorage
- Notify
- PackageFlowFile
- PaginatedJsonQueryElasticsearch
- ParseEvtx
- ParseNetflowv5
- ParseSyslog
- ParseSyslog5424
- PartitionRecord
- PublishAMQP
- PublishGCPubSub
- PublishJMS
- PublishKafka
- PublishMQTT
- PublishSlack
- PutAzureBlobStorage_v12
- PutAzureCosmosDBRecord
- PutAzureDataExplorer
- PutAzureDataLakeStorage
- PutAzureEventHub
- PutAzureQueueStorage_v12
- PutBigQuery
- PutBoxFile
- PutCloudWatchMetric
- PutDatabaseRecord
- PutDistributedMapCache
- PutDropbox
- PutDynamoDB
- PutDynamoDBRecord
- PutElasticsearchJson
- PutElasticsearchRecord
- PutEmail
- PutFile
- PutFTP
- PutGCSObject
- PutGoogleDrive
- PutGridFS
- PutKinesisFirehose
- PutKinesisStream
- PutLambda
- PutMongo
- PutMongoBulkOperations
- PutMongoRecord
- PutRecord
- PutRedisHashRecord
- PutS3Object
- PutSalesforceObject
- PutSFTP
- PutSmbFile
- PutSNS
- PutSplunk
- PutSplunkHTTP
- PutSQL
- PutSQS
- PutSyslog
- PutTCP
- PutUDP
- PutWebSocket
- PutZendeskTicket
- QueryAirtableTable
- QueryAzureDataExplorer
- QueryDatabaseTable
- QueryDatabaseTableRecord
- QueryRecord
- QuerySalesforceObject
- QuerySplunkIndexingStatus
- RemoveRecordField
- RenameRecordField
- ReplaceText
- ReplaceTextWithMapping
- RetryFlowFile
- RouteHL7
- RouteOnAttribute
- RouteOnContent
- RouteText
- RunMongoAggregation
- SampleRecord
- ScanAttribute
- ScanContent
- ScriptedFilterRecord
- ScriptedPartitionRecord
- ScriptedTransformRecord
- ScriptedValidateRecord
- SearchElasticsearch
- SegmentContent
- SendTrapSNMP
- SetSNMP
- SignContentPGP
- SplitAvro
- SplitContent
- SplitExcel
- SplitJson
- SplitPCAP
- SplitRecord
- SplitText
- SplitXml
- StartAwsPollyJob
- StartAwsTextractJob
- StartAwsTranscribeJob
- StartAwsTranslateJob
- StartGcpVisionAnnotateFilesOperation
- StartGcpVisionAnnotateImagesOperation
- TagS3Object
- TailFile
- TransformXml
- UnpackContent
- UpdateAttribute
- UpdateByQueryElasticsearch
- UpdateCounter
- UpdateDatabaseTable
- UpdateRecord
- ValidateCsv
- ValidateJson
- ValidateRecord
- ValidateXml
- VerifyContentMAC
- VerifyContentPGP
- Wait
-
Controller Services
- ADLSCredentialsControllerService
- ADLSCredentialsControllerServiceLookup
- AmazonGlueSchemaRegistry
- ApicurioSchemaRegistry
- AvroReader
- AvroRecordSetWriter
- AvroSchemaRegistry
- AWSCredentialsProviderControllerService
- AzureBlobStorageFileResourceService
- AzureCosmosDBClientService
- AzureDataLakeStorageFileResourceService
- AzureEventHubRecordSink
- AzureStorageCredentialsControllerService_v12
- AzureStorageCredentialsControllerServiceLookup_v12
- CEFReader
- ConfluentEncodedSchemaReferenceReader
- ConfluentEncodedSchemaReferenceWriter
- ConfluentSchemaRegistry
- CSVReader
- CSVRecordLookupService
- CSVRecordSetWriter
- DatabaseRecordLookupService
- DatabaseRecordSink
- DatabaseTableSchemaRegistry
- DBCPConnectionPool
- DBCPConnectionPoolLookup
- DistributedMapCacheLookupService
- ElasticSearchClientServiceImpl
- ElasticSearchLookupService
- ElasticSearchStringLookupService
- EmailRecordSink
- EmbeddedHazelcastCacheManager
- ExcelReader
- ExternalHazelcastCacheManager
- FreeFormTextRecordSetWriter
- GCPCredentialsControllerService
- GCSFileResourceService
- GrokReader
- HazelcastMapCacheClient
- HikariCPConnectionPool
- HttpRecordSink
- IPLookupService
- JettyWebSocketClient
- JettyWebSocketServer
- JMSConnectionFactoryProvider
- JndiJmsConnectionFactoryProvider
- JsonConfigBasedBoxClientService
- JsonPathReader
- JsonRecordSetWriter
- JsonTreeReader
- Kafka3ConnectionService
- KerberosKeytabUserService
- KerberosPasswordUserService
- KerberosTicketCacheUserService
- LoggingRecordSink
- MapCacheClientService
- MapCacheServer
- MongoDBControllerService
- MongoDBLookupService
- PropertiesFileLookupService
- ProtobufReader
- ReaderLookup
- RecordSetWriterLookup
- RecordSinkServiceLookup
- RedisConnectionPoolService
- RedisDistributedMapCacheClientService
- RestLookupService
- S3FileResourceService
- ScriptedLookupService
- ScriptedReader
- ScriptedRecordSetWriter
- ScriptedRecordSink
- SetCacheClientService
- SetCacheServer
- SimpleCsvFileLookupService
- SimpleDatabaseLookupService
- SimpleKeyValueLookupService
- SimpleRedisDistributedMapCacheClientService
- SimpleScriptedLookupService
- SiteToSiteReportingRecordSink
- SlackRecordSink
- SmbjClientProviderService
- StandardAsanaClientProviderService
- StandardAzureCredentialsControllerService
- StandardDropboxCredentialService
- StandardFileResourceService
- StandardHashiCorpVaultClientService
- StandardHttpContextMap
- StandardJsonSchemaRegistry
- StandardKustoIngestService
- StandardKustoQueryService
- StandardOauth2AccessTokenProvider
- StandardPGPPrivateKeyService
- StandardPGPPublicKeyService
- StandardPrivateKeyService
- StandardProxyConfigurationService
- StandardRestrictedSSLContextService
- StandardS3EncryptionService
- StandardSSLContextService
- StandardWebClientServiceProvider
- Syslog5424Reader
- SyslogReader
- UDPEventRecordSink
- VolatileSchemaCache
- WindowsEventLogReader
- XMLFileLookupService
- XMLReader
- XMLRecordSetWriter
- YamlTreeReader
- ZendeskRecordSink
RestLookupService 2.0.0
- Bundle
- org.apache.nifi | nifi-lookup-services-nar
- Description
- Use a REST service to look up values.
- Tags
- http, json, lookup, rest, xml
- Input Requirement
- Supports Sensitive Dynamic Properties
- true
-
Additional Details for RestLookupService 2.0.0
RestLookupService
General
This lookup service has the following optional lookup coordinate keys:
- request.method; defaults to ‘get’, valid values:
- delete
- get
- post
- put
- request.body; contains a string representing JSON, XML, etc. to be sent with any of those methods except for “get”.
- mime.type; specifies media type of the request body, required when ‘body’ is passed.
- *; any other keys can be configured to pass variables to resolve target URLs. See ‘Dynamic URLs’ section below.
The record reader is used to consume the response of the REST service call and turn it into one or more records. The record path property is provided to allow for a lookup path to either a nested record or a single point deep in the REST response. Note: a valid schema must be built that encapsulates the REST response accurately in order for this service to work.
Headers
Headers are supported using dynamic properties. Just add a dynamic property and the name will be the header name and the value will be the value for the header. Expression language powered by input from the variable registry is supported.
Dynamic URLs
The URL property supports expression language through the lookup key/value pairs configured on the component using this lookup service (e.g. LookupRecord processor). The configuration specified by the user will be passed through to the expression language engine for evaluation. Note: flowfile attributes will be disregarded here for this property.
Ex. URL: http://example.com/service/${user.name}/friend/${friend.id}, combined with example record paths at LookupRecord processor:
- user.name => “/example/username”
- friend.id => “/example/first_friend”
Would dynamically produce an endpoint of http://example.com/service/john.smith/friend/12345
Using Environment Properties with URLs
In addition to the lookup key/value pairs, environment properties / system variables can be referred from expression languages configured at the URL property.
Ex. URL: http://${apiServerHostname}:${apiServerPort}/service/${user.name}/friend/${friend.id}, combined with the previous example record paths, and environment properties:
- apiServerHostname => “test.example.com”
- apiServerPort => “8080”
Would dynamically produce an endpoint of http://test.example.com:8080/service/john.smith/friend/12345
- request.method; defaults to ‘get’, valid values:
-
Proxy Configuration Service
Specifies the Proxy Configuration Controller Service to proxy network requests. Supported proxies: SOCKS, HTTP + AuthN In case of SOCKS, it is not guaranteed that the selected SOCKS Version will be used by the processor.
- Display Name
- Proxy Configuration Service
- Description
- Specifies the Proxy Configuration Controller Service to proxy network requests. Supported proxies: SOCKS, HTTP + AuthN In case of SOCKS, it is not guaranteed that the selected SOCKS Version will be used by the processor.
- API Name
- proxy-configuration-service
- Service Interface
- org.apache.nifi.proxy.ProxyConfigurationService
- Service Implementations
- Expression Language Scope
- Not Supported
- Sensitive
- false
- Required
- false
-
Authentication Strategy
Authentication strategy to use with REST service.
- Display Name
- Authentication Strategy
- Description
- Authentication strategy to use with REST service.
- API Name
- rest-lookup-authentication-strategy
- Default Value
- NONE
- Allowable Values
-
- None
- Basic
- OAuth2
- Expression Language Scope
- Not Supported
- Sensitive
- false
- Required
- true
-
Basic Authentication Password
The password to be used by the client to authenticate against the Remote URL.
- Display Name
- Basic Authentication Password
- Description
- The password to be used by the client to authenticate against the Remote URL.
- API Name
- rest-lookup-basic-auth-password
- Expression Language Scope
- Environment variables defined at JVM level and system properties
- Sensitive
- true
- Required
- false
- Dependencies
-
- Authentication Strategy is set to any of [BASIC]
-
Basic Authentication Username
The username to be used by the client to authenticate against the Remote URL. Cannot include control characters (0-31), ':', or DEL (127).
- Display Name
- Basic Authentication Username
- Description
- The username to be used by the client to authenticate against the Remote URL. Cannot include control characters (0-31), ':', or DEL (127).
- API Name
- rest-lookup-basic-auth-username
- Expression Language Scope
- Environment variables defined at JVM level and system properties
- Sensitive
- false
- Required
- false
- Dependencies
-
- Authentication Strategy is set to any of [BASIC]
-
Connection Timeout
Max wait time for connection to remote service.
- Display Name
- Connection Timeout
- Description
- Max wait time for connection to remote service.
- API Name
- rest-lookup-connection-timeout
- Default Value
- 5 secs
- Expression Language Scope
- Not Supported
- Sensitive
- false
- Required
- true
-
Use Digest Authentication
Whether to communicate with the website using Digest Authentication. 'Basic Authentication Username' and 'Basic Authentication Password' are used for authentication.
- Display Name
- Use Digest Authentication
- Description
- Whether to communicate with the website using Digest Authentication. 'Basic Authentication Username' and 'Basic Authentication Password' are used for authentication.
- API Name
- rest-lookup-digest-auth
- Default Value
- false
- Allowable Values
-
- true
- false
- Expression Language Scope
- Not Supported
- Sensitive
- false
- Required
- false
- Dependencies
-
- Authentication Strategy is set to any of [BASIC]
-
OAuth2 Access Token Provider
Enables managed retrieval of OAuth2 Bearer Token applied to HTTP requests using the Authorization Header.
- Display Name
- OAuth2 Access Token Provider
- Description
- Enables managed retrieval of OAuth2 Bearer Token applied to HTTP requests using the Authorization Header.
- API Name
- rest-lookup-oauth2-access-token-provider
- Service Interface
- org.apache.nifi.oauth2.OAuth2AccessTokenProvider
- Service Implementations
- Expression Language Scope
- Not Supported
- Sensitive
- false
- Required
- true
- Dependencies
-
- Authentication Strategy is set to any of [OAUTH2]
-
Read Timeout
Max wait time for response from remote service.
- Display Name
- Read Timeout
- Description
- Max wait time for response from remote service.
- API Name
- rest-lookup-read-timeout
- Default Value
- 15 secs
- Expression Language Scope
- Not Supported
- Sensitive
- false
- Required
- true
-
Record Path
An optional record path that can be used to define where in a record to get the real data to merge into the record set to be enriched. See documentation for examples of when this might be useful.
- Display Name
- Record Path
- Description
- An optional record path that can be used to define where in a record to get the real data to merge into the record set to be enriched. See documentation for examples of when this might be useful.
- API Name
- rest-lookup-record-path
- Expression Language Scope
- Environment variables defined at JVM level and system properties
- Sensitive
- false
- Required
- false
-
Record Reader
The record reader to use for loading the payload and handling it as a record set.
- Display Name
- Record Reader
- Description
- The record reader to use for loading the payload and handling it as a record set.
- API Name
- rest-lookup-record-reader
- Service Interface
- org.apache.nifi.serialization.RecordReaderFactory
- Service Implementations
- Expression Language Scope
- Not Supported
- Sensitive
- false
- Required
- true
-
Response Handling Strategy
Whether to return all responses or throw errors for unsuccessful HTTP status codes.
- Display Name
- Response Handling Strategy
- Description
- Whether to return all responses or throw errors for unsuccessful HTTP status codes.
- API Name
- rest-lookup-response-handling-strategy
- Default Value
- RETURNED
- Allowable Values
-
- Returned
- Evaluated
- Expression Language Scope
- Not Supported
- Sensitive
- false
- Required
- true
-
SSL Context Service
The SSL Context Service used to provide client certificate information for TLS/SSL connections.
- Display Name
- SSL Context Service
- Description
- The SSL Context Service used to provide client certificate information for TLS/SSL connections.
- API Name
- rest-lookup-ssl-context-service
- Service Interface
- org.apache.nifi.ssl.SSLContextService
- Service Implementations
- Expression Language Scope
- Not Supported
- Sensitive
- false
- Required
- false
-
URL
The URL for the REST endpoint. Expression language is evaluated against the lookup key/value pairs, not flowfile attributes.
- Display Name
- URL
- Description
- The URL for the REST endpoint. Expression language is evaluated against the lookup key/value pairs, not flowfile attributes.
- API Name
- rest-lookup-url
- Expression Language Scope
- Environment variables and FlowFile Attributes
- Sensitive
- false
- Required
- true
-
*
All dynamic properties are added as HTTP headers with the name as the header name and the value as the header value.
- Name
- *
- Description
- All dynamic properties are added as HTTP headers with the name as the header name and the value as the header value.
- Value
- *
- Expression Language Scope
- FLOWFILE_ATTRIBUTES