BigQuery

Connect to BigQuery databases from Airplane tasks and runbooks.
The BigQuery resource allows tasks and runbooks to connect to a BigQuery database. For a high-level introduction to resources, see the resources page.

Creating a BigQuery resource

To create a new BigQuery resource, visit resources for your team, click the "New" button, and select the BigQuery resource type. Enter your connection details in the form:
Name should be a human-readable name for your resource.
Project ID, Location, and Dataset can all be found in the BigQuery console under Dataset Info.
Credentials are the private keys for the service account with access to this dataset. The service account should have the permissions "BigQuery Job User" and "BigQuery User". Credentials can be found in the Google Cloud console under IAM & Admin > Service Accounts > Keys. The key should be of JSON type and can only be viewed at creation time. See "Create and manage service account keys" for more information about managing service account keys.

IP whitelisting

If your database has restrictions on incoming network connections, you'll need to ensure the Airplane IP addresses are whitelisted:
  • 34.121.89.145
  • 34.68.100.27
  • 34.72.182.225

Private networking

To connect to a database inside a private VPC, you can self-host the agent and run the agent within your own VPC. Since tasks and SQL queries run on agents, these tasks will run from within your network and be able to connect.

Accessing from tasks

You can issue SQL queries against your BigQuery resource from tasks by using the SQL SDK. See SQL SDK reference for more information.

Field reference

When this resource is attached to a task, its fields are available in the JSON environment variable AIRPLANE_RESOURCES under the alias it was attached with. For more information, see the BigQuery field reference.