ExecuteSparkInteractive

Description:

Execute Spark Code over a Livy-managed HTTP session to a live Spark context. Supports cached RDD sharing.

Tags:

spark, livy, http, execute

Properties:

In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.

NameDefault ValueAllowable ValuesDescription
Livy Controller ServiceController Service API:
LivySessionService
Implementation: LivySessionController
The controller service to use for Livy-managed session(s).
CodeThe code to execute in the session. This property can be empty, a constant value, or built from attributes using Expression Language. If this property is specified, it will be used regardless of the content of incoming flowfiles. If this property is empty, the content of the incoming flow file is expected to contain valid code to be issued by the processor to the session. Note that Expression Language is not evaluated for flow file contents.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Character SetUTF-8The character set encoding for the incoming flow file.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Status Check Interval1 secThe amount of time to wait between checking the status of an operation.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)

Relationships:

NameDescription
successFlowFiles that are successfully processed are sent to this relationship
waitFlowFiles that are waiting on an available Spark session will be sent to this relationship
failureFlowFiles are routed to this relationship when they cannot be parsed

Reads Attributes:

None specified.

Writes Attributes:

None specified.

State management:

This component does not store state.

Restricted:

This component is not restricted.

Input requirement:

This component requires an incoming relationship.

System Resource Considerations:

None specified.