ConsumeTwitter

ConsumeTwitter 2.0.0

Bundle: org.apache.nifi | nifi-social-media-nar
Description: Streams tweets from Twitter's streaming API v2. The stream provides a sample stream or a search stream based on previously uploaded rules. This processor also provides a pass through for certain fields of the tweet to be returned as part of the response. See https://developer.twitter.com/en/docs/twitter-api/data-dictionary/introduction for more information regarding the Tweet object model.
Tags: json, social media, status, tweets, twitter
Input Requirement: FORBIDDEN
Supports Sensitive Dynamic Properties: false

Properties

Backfill Minutes
The number of minutes (up to 5 minutes) of streaming data to be requested after a disconnect. Only available for project with academic research access. See https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/recovery-and-redundancy-features

Display Name

Backfill Minutes

Description

The number of minutes (up to 5 minutes) of streaming data to be requested after a disconnect. Only available for project with academic research access. See https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/recovery-and-redundancy-features

API Name

backfill-minutes

Default Value

0

Expression Language Scope

Not Supported

Sensitive

false

Required

true
Backoff Attempts
The number of reconnection tries the processor will attempt in the event of a disconnection of the stream for any reason, before throwing an exception. To start a stream after this exception occur and the connection is fixed, please stop and restart the processor. If the valueof this property is 0, then backoff will never occur and the processor will always need to be restartedif the stream fails.

Display Name

Backoff Attempts

Description

The number of reconnection tries the processor will attempt in the event of a disconnection of the stream for any reason, before throwing an exception. To start a stream after this exception occur and the connection is fixed, please stop and restart the processor. If the valueof this property is 0, then backoff will never occur and the processor will always need to be restartedif the stream fails.

API Name

backoff-attempts

Default Value

5

Expression Language Scope

Not Supported

Sensitive

false

Required

true
Backoff Time
The duration to backoff before requesting a new stream ifthe current one fails for any reason. Will increase by factor of 2 every time a restart fails

Display Name

Backoff Time

Description

The duration to backoff before requesting a new stream ifthe current one fails for any reason. Will increase by factor of 2 every time a restart fails

API Name

backoff-time

Default Value

1 mins

Expression Language Scope

Not Supported

Sensitive

false

Required

true
Base Path
The base path that the processor will use for making HTTP requests. The default value should be sufficient for most use cases.

Display Name

Base Path

Description

The base path that the processor will use for making HTTP requests. The default value should be sufficient for most use cases.

API Name

base-path

Default Value

https://api.twitter.com

Expression Language Scope

Not Supported

Sensitive

false

Required

true
Batch Size
The maximum size of the number of Tweets to be written to a single FlowFile. Will write fewer Tweets based on the number available in the queue at the time of processor invocation.

Display Name

Batch Size

Description

The maximum size of the number of Tweets to be written to a single FlowFile. Will write fewer Tweets based on the number available in the queue at the time of processor invocation.

API Name

batch-size

Default Value

1000

Expression Language Scope

Not Supported

Sensitive

false

Required

true
Bearer Token
The Bearer Token provided by Twitter.

Display Name

Bearer Token

Description

The Bearer Token provided by Twitter.

API Name

bearer-token

Expression Language Scope

Not Supported

Sensitive

true

Required

true
Connect Timeout
The maximum time in which client should establish a connection with the Twitter API before a time out. Setting the value to 0 disables connection timeouts.

Display Name

Connect Timeout

Description

The maximum time in which client should establish a connection with the Twitter API before a time out. Setting the value to 0 disables connection timeouts.

API Name

connect-timeout

Default Value

10 secs

Expression Language Scope

Not Supported

Sensitive

false

Required

true
Expansions
A comma-separated list of expansions for objects in the returned tweet. See https://developer.twitter.com/en/docs/twitter-api/expansions for proper usage. Possible field values include: author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id

Display Name

Expansions

Description

A comma-separated list of expansions for objects in the returned tweet. See https://developer.twitter.com/en/docs/twitter-api/expansions for proper usage. Possible field values include: author_id, referenced_tweets.id, referenced_tweets.id.author_id, entities.mentions.username, attachments.poll_ids, attachments.media_keys ,in_reply_to_user_id, geo.place_id

API Name

expansions

Expression Language Scope

Not Supported

Sensitive

false

Required

false
Maximum Backoff Time
The maximum duration to backoff to start attempting a new stream.It is recommended that this number be much higher than the 'Backoff Time' property

Display Name

Maximum Backoff Time

Description

The maximum duration to backoff to start attempting a new stream.It is recommended that this number be much higher than the 'Backoff Time' property

API Name

maximum-backoff-time

Default Value

5 mins

Expression Language Scope

Not Supported

Sensitive

false

Required

true
Media Fields
A comma-separated list of media fields to be returned as part of the tweet. Refer to https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media for proper usage. Possible field values include: alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, promoted_metrics, public_metrics, type, url, width

Display Name

Media Fields

Description

A comma-separated list of media fields to be returned as part of the tweet. Refer to https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/media for proper usage. Possible field values include: alt_text, duration_ms, height, media_key, non_public_metrics, organic_metrics, preview_image_url, promoted_metrics, public_metrics, type, url, width

API Name

media-fields

Expression Language Scope

Not Supported

Sensitive

false

Required

false
Place Fields
A comma-separated list of place fields to be returned as part of the tweet. Refer to https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place for proper usage. Possible field values include: contained_within, country, country_code, full_name, geo, id, name, place_type

Display Name

Place Fields

Description

A comma-separated list of place fields to be returned as part of the tweet. Refer to https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/place for proper usage. Possible field values include: contained_within, country, country_code, full_name, geo, id, name, place_type

API Name

place-fields

Expression Language Scope

Not Supported

Sensitive

false

Required

false
Poll Fields
A comma-separated list of poll fields to be returned as part of the tweet. Refer to https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll for proper usage. Possible field values include: duration_minutes, end_datetime, id, options, voting_status

Display Name

Poll Fields

Description

A comma-separated list of poll fields to be returned as part of the tweet. Refer to https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/poll for proper usage. Possible field values include: duration_minutes, end_datetime, id, options, voting_status

API Name

poll-fields

Expression Language Scope

Not Supported

Sensitive

false

Required

false
Queue Size
Maximum size of internal queue for streamed messages

Display Name

Queue Size

Description

Maximum size of internal queue for streamed messages

API Name

queue-size

Default Value

10000

Expression Language Scope

Not Supported

Sensitive

false

Required

true
Read Timeout
The maximum time of inactivity between receiving tweets from Twitter through the API before a timeout. Setting the value to 0 disables read timeouts.

Display Name

Read Timeout

Description

The maximum time of inactivity between receiving tweets from Twitter through the API before a timeout. Setting the value to 0 disables read timeouts.

API Name

read-timeout

Default Value

10 secs

Expression Language Scope

Not Supported

Sensitive

false

Required

true
Stream Endpoint
The source from which the processor will consume Tweets.
Display Name

Stream Endpoint

Description

The source from which the processor will consume Tweets.

API Name

stream-endpoint

Default Value

Sample Endpoint

Allowable Values
- Sample Stream
- Search Stream
Expression Language Scope

Not Supported

Sensitive

false

Required

true
Tweet Fields
A comma-separated list of tweet fields to be returned as part of the tweet. Refer to https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet for proper usage. Possible field values include: attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, public_metrics, referenced_tweets, reply_settings, source, text, withheld

Display Name

Tweet Fields

Description

A comma-separated list of tweet fields to be returned as part of the tweet. Refer to https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/tweet for proper usage. Possible field values include: attachments, author_id, context_annotations, conversation_id, created_at, entities, geo, id, in_reply_to_user_id, lang, non_public_metrics, organic_metrics, possibly_sensitive, promoted_metrics, public_metrics, referenced_tweets, reply_settings, source, text, withheld

API Name

tweet-fields

Expression Language Scope

Not Supported

Sensitive

false

Required

false
User Fields
A comma-separated list of user fields to be returned as part of the tweet. Refer to https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user for proper usage. Possible field values include: created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, protected, public_metrics, url, username, verified, withheld

Display Name

User Fields

Description

A comma-separated list of user fields to be returned as part of the tweet. Refer to https://developer.twitter.com/en/docs/twitter-api/data-dictionary/object-model/user for proper usage. Possible field values include: created_at, description, entities, id, location, name, pinned_tweet_id, profile_image_url, protected, public_metrics, url, username, verified, withheld

API Name

user-fields

Expression Language Scope

Not Supported

Sensitive

false

Required

false

Relationships

Name	Description
success	FlowFiles containing an array of one or more Tweets

Writes Attributes

Name	Description
mime.type	The MIME Type set to application/json
tweets	The number of Tweets in the FlowFile