BigQuery

The BigQuery resource allows tasks and runbooks to connect to a BigQuery database. For a high-level introduction to resources, see the resources page.

To create a new BigQuery resource, visit resources for your team, click the "New" button, and select the BigQuery resource type. Enter your connection details in the form:

Name should be a human-readable name for your resource.

Project ID, Location, and Dataset can all be found in the BigQuery console under Dataset Info.

Credentials are the private keys for the service account with access to this dataset. The service account should have the permissions "BigQuery Job User" and "BigQuery User". Credentials can be found in the Google Cloud console under IAM & Admin > Service Accounts > Keys. The key should be of JSON type and can only be viewed at creation time. See "Create and manage service account keys" for more information about managing service account keys.

If your database has restrictions on incoming network connections, you'll need to ensure the Airplane IP addresses are whitelisted:

To connect to a database inside a private VPC, you can self-host the agent and run the agent within your own VPC. Since tasks and SQL queries run on agents, these tasks will run from within your network and be able to connect.

You can issue SQL queries against your BigQuery resource from tasks by using the SQL SDK. See SQL SDK reference for more information.

When this resource is attached to a task, its fields are available in the JSON environment variable AIRPLANE_RESOURCES under the alias it was attached with. For more information, see the BigQuery field reference.