Best practices guide for scale

Using Airplane with scalability and security in mind.

Hosting model
Airplane offers two options for executing tasks: Airplane-hosted agents and self-hosted agents.
Airplane-hosted agents are great for getting started, especially if your resources are accessible from the Internet. They require no maintenance and are included with all plans.
Self-hosted agents are common for customers who are scaling and have stricter security or compliance requirements. When agents are self-hosted in a customer's VPC, tasks are executed by the agents within the same network. Self-hosted agents are especially useful if you have private databases or APIs, or if you generally want tasks executed on your own infrastructure. See our docs on agents to learn more about setting this up.
There are a couple small tradeoffs to keep in mind when using self-hosted agents:
  • Setup time: you'll need to install agents initially. This takes just a few minutes if you have infrastructure experience. If not, this may require someone on your team with access to your cloud provider to help out.
  • Cost: since you're installing Airplane agents on your own servers, you'll need to consider server / cloud computing costs. This tends to be low for most Airplane use cases and is mostly dependent on how many tasks you're executing.

Comparing SQL, REST, and code-based tasks
If you want to create a task that executes a SQL query (for example), you have multiple options:
  • Setup a database resource, and then create a SQL task
  • Create an API endpoint in your own codebase, and then create a REST task that hits that endpoint
  • Create a JavaScript, Python or other code-based task that executes code that directly queries the database (or hits an API endpoint in your codebase that you've created).
There is no single best task type, and the right type depends on your use case. We have customers using any and all of the above methods. However, there are a few common patterns we've seen:
  • As a default, JavaScript, Python, or Docker will be simple to use and can be very flexible in what you can do.
  • Some organizations protect their sensitive data behind an API, in which case they might use a REST task or make REST calls from a Node/Python task.
  • Some organizations move faster and use SQL tasks, in which case they can build tasks rapidly without needing to spin up new API endpoints. A compromise between speed and safety is often using SQL for read-only tasks, and REST for write tasks.

Storing config database credentials and secrets
Many tasks in Airplane will require access to secrets: database credentials, API keys, passwords, etc. There are two options that enterprises use with Airplane, depending on your security posture:
  • Store secrets with Airplane: you can create resources and config variables directly in Airplane, which you can then reference in your tasks. This is the easiest-to-use option and what the majority of our customers do. However, this means that sensitive data is stored on Airplane's servers. We are SOC 2 Type 2 compliant and encrypt all data in transit and at rest.
  • Manage secret storage yourself: If you'd prefer not to store secret data on Airplane's servers, you can use any secrets manager (AWS Secrets Manager, Doppler, Hashicorp Vault, etc), and then refer to these secrets via their respective APIs in a code-based task. Note that if you use this approach, you won't be able to use Airplane's built-in task modes (SQL, REST, Mongo, etc) and instead will have to use Node, Python, or other code-based tasks exclusively. You can write a helper function that abstracts away the secrets management and import that function into every task that needs it. You won't lose any functionality, but this would mean require trading off a bit of ease of use.

Code lifecycle
You can create Airplane tasks quickly without managing any code on your machine by using cloud workspaces in Studio. This works fine, but if you're using Airplane on a team, we recommend integrating Airplane into your existing development workflow (version control, code review, testing, CI/CD, prod/staging/test environments, etc) using local workspaces.
Airplane has first-class support for all these concepts:
We recommend using GitHub or something similar and running deploys when certain branches are pushed:
  • You can start with your main branch deploying to a single production environment
  • Alternatively, you can have both main and prod branches, where main deploys to a staging environment using --env stage and prod deploys to a prod environment
To ensure that developers aren't creating or editing tasks outside of your CI/CD system, you can set up your developer group to have the "Restricted developer" role in the environment(s) you want to be CI/CD-only. See Permissions for how to set this up.

Airplane security practices
We're SOC 2 Type II compliant, HIPAA compliant, and have industry-standard security practices. Airplane services are designed with privacy, integration, and availability in mind. All customer data is encrypted in transit and at rest. Learn more here.
PrivacyTermsSupport