GetSFTP

Description:

Fetches files from an SFTP Server and creates FlowFiles from them

Tags:

sftp, get, retrieve, files, fetch, remote, ingest, source, input

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.

Display NameAPI NameDefault ValueAllowable ValuesDescription
HostnameHostnameThe fully qualified hostname or IP address of the remote system
Supports Expression Language: true (will be evaluated using Environment variables only)
PortPort22The port that the remote system is listening on for file transfers
Supports Expression Language: true (will be evaluated using Environment variables only)
UsernameUsernameUsername
Supports Expression Language: true (will be evaluated using Environment variables only)
PasswordPasswordPassword for the user account
Sensitive Property: true
Supports Expression Language: true (will be evaluated using Environment variables only)
Private Key PathPrivate Key PathThe fully qualified path to the Private Key file

This property requires exactly one file to be provided..

Supports Expression Language: true (will be evaluated using Environment variables only)
Private Key PassphrasePrivate Key PassphrasePassword for the private key
Sensitive Property: true
Supports Expression Language: true (will be evaluated using Environment variables only)
Remote PathRemote PathThe path on the remote system from which to pull or push files
Supports Expression Language: true (will be evaluated using Environment variables only)
File Filter RegexFile Filter RegexProvides a Java Regular Expression for filtering Filenames; if a filter is supplied, only files whose names match that Regular Expression will be fetched
Path Filter RegexPath Filter RegexWhen Search Recursively is true, then only subdirectories whose path matches the given Regular Expression will be scanned
Polling IntervalPolling Interval60 secDetermines how long to wait between fetching the listing for new files
Search RecursivelySearch Recursivelyfalse
  • true
  • false
If true, will pull files from arbitrarily nested subdirectories; otherwise, will not traverse subdirectories
Follow symlinkfollow-symlinkfalse
  • true
  • false
If true, will pull even symbolic files and also nested symbolic subdirectories; otherwise, will not read symbolic files and will not traverse symbolic link subdirectories
Ignore Dotted FilesIgnore Dotted Filestrue
  • true
  • false
If true, files whose names begin with a dot (".") will be ignored
Delete OriginalDelete Originaltrue
  • true
  • false
Determines whether or not the file is deleted from the remote system after it has been successfully transferred
Connection TimeoutConnection Timeout30 secAmount of time to wait before timing out while creating a connection
Data TimeoutData Timeout30 secWhen transferring a file between the local and remote system, this value specifies how long is allowed to elapse without any data being transferred between systems
Host Key FileHost Key FileIf supplied, the given file will be used as the Host Key; otherwise, if 'Strict Host Key Checking' property is applied (set to true) then uses the 'known_hosts' and 'known_hosts2' files from ~/.ssh directory else no host key file will be used

This property requires exactly one file to be provided..
Max SelectsMax Selects100The maximum number of files to pull in a single connection
Remote Poll Batch SizeRemote Poll Batch Size5000The value specifies how many file paths to find in a given directory on the remote system when doing a file listing. This value in general should not need to be modified but when polling against a remote system with a tremendous number of files this value can be critical. Setting this value too high can result very poor performance and setting it too low can cause the flow to be slower than normal.
Strict Host Key CheckingStrict Host Key Checkingfalse
  • true
  • false
Indicates whether or not strict enforcement of hosts keys should be applied
Send Keep Alive On TimeoutSend Keep Alive On Timeouttrue
  • true
  • false
Send a Keep Alive message every 5 seconds up to 5 times for an overall timeout of 25 seconds.
Use CompressionUse Compressionfalse
  • true
  • false
Indicates whether or not ZLIB compression should be used when transferring files
Use Natural OrderingUse Natural Orderingfalse
  • true
  • false
If true, will pull files in the order in which they are naturally listed; otherwise, the order in which the files will be pulled is not defined
Proxy Configuration Serviceproxy-configuration-serviceController Service API:
ProxyConfigurationService
Implementation: StandardProxyConfigurationService
Specifies the Proxy Configuration Controller Service to proxy network requests. If set, it supersedes proxy settings configured per component. Supported proxies: HTTP + AuthN, SOCKS + AuthN
Proxy TypeProxy TypeDIRECT
  • DIRECT
  • HTTP
  • SOCKS
Proxy type used for file transfers
Proxy HostProxy HostThe fully qualified hostname or IP address of the proxy server
Supports Expression Language: true (will be evaluated using Environment variables only)
Proxy PortProxy PortThe port of the proxy server
Supports Expression Language: true (will be evaluated using Environment variables only)
Http Proxy UsernameHttp Proxy UsernameHttp Proxy Username
Supports Expression Language: true (will be evaluated using Environment variables only)
Http Proxy PasswordHttp Proxy PasswordHttp Proxy Password
Sensitive Property: true
Supports Expression Language: true (will be evaluated using Environment variables only)
Ciphers AllowedCiphers AllowedA comma-separated list of Ciphers allowed for SFTP connections. Leave unset to allow all. Available options are: 3des-cbc, 3des-ctr, aes128-cbc, aes128-ctr, aes128-gcm@openssh.com, aes192-cbc, aes192-ctr, aes256-cbc, aes256-ctr, aes256-gcm@openssh.com, arcfour, arcfour128, arcfour256, blowfish-cbc, blowfish-ctr, cast128-cbc, cast128-ctr, chacha20-poly1305@openssh.com, idea-cbc, idea-ctr, serpent128-cbc, serpent128-ctr, serpent192-cbc, serpent192-ctr, serpent256-cbc, serpent256-ctr, twofish-cbc, twofish128-cbc, twofish128-ctr, twofish192-cbc, twofish192-ctr, twofish256-cbc, twofish256-ctr
Supports Expression Language: true (will be evaluated using Environment variables only)
Key Algorithms AllowedKey Algorithms AllowedA comma-separated list of Key Algorithms allowed for SFTP connections. Leave unset to allow all. Available options are: ecdsa-sha2-nistp256, ecdsa-sha2-nistp256-cert-v01@openssh.com, ecdsa-sha2-nistp384, ecdsa-sha2-nistp384-cert-v01@openssh.com, ecdsa-sha2-nistp521, ecdsa-sha2-nistp521-cert-v01@openssh.com, rsa-sha2-256, rsa-sha2-512, ssh-dss, ssh-dss-cert-v01@openssh.com, ssh-ed25519, ssh-ed25519-cert-v01@openssh.com, ssh-rsa, ssh-rsa-cert-v01@openssh.com
Supports Expression Language: true (will be evaluated using Environment variables only)
Key Exchange Algorithms AllowedKey Exchange Algorithms AllowedA comma-separated list of Key Exchange Algorithms allowed for SFTP connections. Leave unset to allow all. Available options are: curve25519-sha256, curve25519-sha256@libssh.org, diffie-hellman-group-exchange-sha1, diffie-hellman-group-exchange-sha256, diffie-hellman-group1-sha1, diffie-hellman-group14-sha1, diffie-hellman-group14-sha256, diffie-hellman-group14-sha256@ssh.com, diffie-hellman-group15-sha256, diffie-hellman-group15-sha256@ssh.com, diffie-hellman-group15-sha384@ssh.com, diffie-hellman-group15-sha512, diffie-hellman-group16-sha256, diffie-hellman-group16-sha384@ssh.com, diffie-hellman-group16-sha512, diffie-hellman-group16-sha512@ssh.com, diffie-hellman-group17-sha512, diffie-hellman-group18-sha512, diffie-hellman-group18-sha512@ssh.com, ecdh-sha2-nistp256, ecdh-sha2-nistp384, ecdh-sha2-nistp521, ext-info-c
Supports Expression Language: true (will be evaluated using Environment variables only)
Message Authentication Codes AllowedMessage Authentication Codes AllowedA comma-separated list of Message Authentication Codes allowed for SFTP connections. Leave unset to allow all. Available options are: hmac-md5, hmac-md5-96, hmac-md5-96-etm@openssh.com, hmac-md5-etm@openssh.com, hmac-ripemd160, hmac-ripemd160-96, hmac-ripemd160-etm@openssh.com, hmac-ripemd160@openssh.com, hmac-sha1, hmac-sha1-96, hmac-sha1-96@openssh.com, hmac-sha1-etm@openssh.com, hmac-sha2-256, hmac-sha2-256-etm@openssh.com, hmac-sha2-512, hmac-sha2-512-etm@openssh.com
Supports Expression Language: true (will be evaluated using Environment variables only)

Relationships:

NameDescription
successAll FlowFiles that are received are routed to success

Reads Attributes:

None specified.

Writes Attributes:

NameDescription
filenameThe filename is set to the name of the file on the remote server
pathThe path is set to the path of the file's directory on the remote server. For example, if the <Remote Path> property is set to /tmp, files picked up from /tmp will have the path attribute set to /tmp. If the <Search Recursively> property is set to true and a file is picked up from /tmp/abc/1/2/3, then the path attribute will be set to /tmp/abc/1/2/3
file.lastModifiedTimeThe date and time that the source file was last modified
file.ownerThe numeric owner id of the source file
file.groupThe numeric group id of the source file
file.permissionsThe read/write/execute permissions of the source file
absolute.pathThe full/absolute path from where a file was picked up. The current 'path' attribute is still populated, but may be a relative path

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component does not allow an incoming relationship.

System Resource Considerations:

None specified.

See Also:

PutSFTP