LivySessionController

Description:

Manages pool of Spark sessions over HTTP

Tags:

Livy, REST, Spark, http

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.

NameDefault ValueAllowable ValuesDescription
Livy HostThe hostname (or IP address) of the Livy server.
Supports Expression Language: true (will be evaluated using variable registry only)
Livy Port8998The port number for the Livy server.
Supports Expression Language: true (will be evaluated using variable registry only)
Session Pool Size2Number of sessions to keep open
Supports Expression Language: true (will be evaluated using variable registry only)
Session Typespark
  • spark
  • pyspark
  • pyspark3
  • sparkr
The type of Spark session to start (spark, pyspark, pyspark3, sparkr, e.g.)
Session Manager Status Interval2 secThe amount of time to wait between requesting session information updates.
Supports Expression Language: true (will be evaluated using variable registry only)
SSL Context ServiceController Service API:
SSLContextService
Implementations: StandardRestrictedSSLContextService
StandardSSLContextService
The SSL Context Service used to provide client certificate information for TLS/SSL (https) connections.
Connection Timeout5 secsMax wait time for connection to remote service.
Session JARsJARs to be used in the Spark session.
Supports Expression Language: true (will be evaluated using variable registry only)
Session FilesFiles to be used in the Spark session.
Supports Expression Language: true (will be evaluated using variable registry only)
Kerberos Credentials ServiceController Service API:
KerberosCredentialsService
Implementation: KeytabCredentialsService
Specifies the Kerberos Credentials Controller Service that should be used for authenticating with Kerberos

State management:

This component does not store state.

Restricted:

This component is not restricted.

System Resource Considerations:

None specified.