Writes the contents of a FlowFile to a Google BigQuery table. The processor is record based so the schema that is used is driven by the RecordReader. Attributes that are not matched to the target schema are skipped. Exactly once delivery semantics are achieved via stream offsets.
| Display Name | API Name | Default Value | Allowable Values | Description |
|---|
| GCP Credentials Provider Service | GCP Credentials Provider Service | | Controller Service API:
GCPCredentialsService
Implementation: GCPCredentialsControllerService | The Controller Service used to obtain Google Cloud Platform credentials. |
| Project ID | gcp-project-id | | | Google Cloud Project ID
Supports Expression Language: true (will be evaluated using variable registry only) |
| BigQuery API Endpoint | bigquery-api-endpoint | bigquerystorage.googleapis.com:443 | | Can be used to override the default BigQuery endpoint. Default is bigquerystorage.googleapis.com:443. Format must be hostname:port.
Supports Expression Language: true (will be evaluated using variable registry only) |
| Dataset | bq.dataset | ${bq.dataset} | | BigQuery dataset name (Note - The dataset must exist in GCP)
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry) |
| Table Name | bq.table.name | ${bq.table.name} | | BigQuery table name
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry) |
| Record Reader | bq.record.reader | | Controller Service API:
RecordReaderFactory
Implementations: GrokReader
JsonTreeReader
WindowsEventLogReader
ReaderLookup
ParquetReader
CSVReader
Syslog5424Reader
ExcelReader
CEFReader
XMLReader
ScriptedReader
SyslogReader
JsonPathReader
AvroReader
YamlTreeReader | Specifies the Controller Service to use for parsing incoming data. |
| Transfer Type | bq.transfer.type | STREAM | - STREAM
 - BATCH
| Defines the preferred transfer type streaming or batching |
| Append Record Count | bq.append.record.count | 20 | | The number of records to be appended to the write stream at once. Applicable for both batch and stream types |
| Number of retries | gcp-retry-count | 6 | | How many retry attempts should be made before routing to the failure relationship. |
| Skip Invalid Rows | bq.skip.invalid.rows | false | | Sets whether to insert all valid rows of a request, even if invalid rows exist. If not set the entire insert request will fail if it contains an invalid row.
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry) |