GCPCredentialsService
Implementation: GCPCredentialsControllerService
The Controller Service used to obtain Google Cloud Platform credentials. | Project ID | gcp-project-id | | | Google Cloud Project ID Supports Expression Language: true (will be evaluated using Environment variables only) |
BigQuery API Endpoint | bigquery-api-endpoint | bigquerystorage.googleapis.com:443 | | Can be used to override the default BigQuery endpoint. Default is bigquerystorage.googleapis.com:443. Format must be hostname:port. Supports Expression Language: true (will be evaluated using Environment variables only) |
Dataset | bq.dataset | ${bq.dataset} | | BigQuery dataset name (Note - The dataset must exist in GCP) Supports Expression Language: true (will be evaluated using flow file attributes and Environment variables) |
Table Name | bq.table.name | ${bq.table.name} | | BigQuery table name Supports Expression Language: true (will be evaluated using flow file attributes and Environment variables) |
Record Reader | bq.record.reader | | Controller Service API: RecordReaderFactory Implementations: ScriptedReader ReaderLookup YamlTreeReader CSVReader AvroReader SyslogReader ProtobufReader GrokReader Syslog5424Reader JsonPathReader JsonTreeReader XMLReader CEFReader ExcelReader WindowsEventLogReader | Specifies the Controller Service to use for parsing incoming data. |
Transfer Type | bq.transfer.type | STREAM | - STREAM
![Use streaming record handling strategy Use streaming record handling strategy](../../../../../html/images/iconInfo.png) - BATCH
![Use batching record handling strategy Use batching record handling strategy](../../../../../html/images/iconInfo.png)
| Defines the preferred transfer type streaming or batching |
Append Record Count | bq.append.record.count | 20 | | The number of records to be appended to the write stream at once. Applicable for both batch and stream types |
Number of retries | gcp-retry-count | 6 | | How many retry attempts should be made before routing to the failure relationship. |
Skip Invalid Rows | bq.skip.invalid.rows | false | | Sets whether to insert all valid rows of a request, even if invalid rows exist. If not set the entire insert request will fail if it contains an invalid row. Supports Expression Language: true (will be evaluated using flow file attributes and Environment variables) |
Proxy Configuration Service | proxy-configuration-service | | Controller Service API: ProxyConfigurationService Implementation: StandardProxyConfigurationService | Specifies the Proxy Configuration Controller Service to proxy network requests. Supported proxies: HTTP + AuthN |
Relationships:
Name | Description |
---|
failure | FlowFiles are routed to this relationship if the Google BigQuery operation fails. |
success | FlowFiles are routed to this relationship after a successful Google BigQuery operation. |
Reads Attributes:
None specified.Writes Attributes:
Name | Description |
---|
bq.records.count | Number of records successfully inserted |
State management:
This component does not store state.Restricted:
This component is not restricted.Input requirement:
This component requires an incoming relationship.System Resource Considerations:
None specified.