dbt Cloud environments
An environment determines how dbt Cloud will execute your project in the dbt Cloud IDE or dbt Cloud CLI (for development) and scheduled jobs (for deployment).
Critically, in order to execute dbt, environments define three variables:
- The version of dbt Core that will be used to run your project
- The warehouse connection information (including the target database/schema settings)
- The version of your code to execute
Each dbt Cloud project can have only one development environment, but there is no limit to the number of deployment environments, providing you the flexibility and customization to tailor the execution of scheduled jobs.
Use environments to customize settings for different stages of your project and streamline the execution process by using software engineering principles. This page will detail the different types of environments and how to intuitively configure your development environment in dbt Cloud.
Types of environments
In dbt Cloud, there are two types of environments:
- Deployment environment — Determines the settings used when jobs created within that environment are executed.
Types of deployment environments:- General
- Staging
- Production
- Development environment — Determines the settings used in the dbt Cloud IDE or dbt Cloud CLI, for that particular project.
Each dbt Cloud project can only have a single development environment but can have any number of deployment environments.
Development | Staging | Deployment | |
---|---|---|---|
Determines settings for | dbt Cloud IDE or dbt Cloud CLI | dbt Cloud Job runs | dbt Cloud Job runs |
How many can I have in my project? | 1 | Any number | Any number |
For users familiar with development on dbt Core, each environment is roughly analogous to an entry in your profiles.yml
file, with some additional information about your repository to ensure the proper version of code is executed. More info on dbt core environments here.
Common environment settings
Both development and deployment environments have a section called General Settings, which has some basic settings that all environments will define:
Setting | Example Value | Definition | Accepted Values |
---|---|---|---|
Name | Production | The environment name | Any string! |
Environment Type | Deployment | The type of environment | [Deployment, Development] |
dbt Version | 1.4 (latest) | The dbt version used | Any dbt version in the dropdown |
Default to Custom Branch | ☑️ | Determines whether to use a branch other than the repository’s default | See below |
Custom Branch | dev | Custom Branch name | See below |
- dbt Cloud allows users to select any dbt release. At this time, environments must use a dbt version greater than or equal to v1.0.0; lower versions are no longer supported.
- If you select a current version with
(latest)
in the name, your environment will automatically install the latest stable version of the minor version selected. - Go Versionless, which removes the need for manually upgrading environment, while ensuring you are always up to date with the latest fixes and features.
Custom branch behavior
By default, all environments will use the default branch in your repository (usually the main
branch) when accessing your dbt code. This is overridable within each dbt Cloud Environment using the Default to a custom branch option. This setting will have slightly different behavior depending on the environment type:
- Development: determines which branch in the dbt Cloud IDE or dbt Cloud CLI developers create branches from and open PRs against.
- Deployment: determines the branch is cloned during job executions for each environment.
For more info, check out this FAQ page on this topic!
Extended attributes
Extended attributes are are currently not supported for SSH tunneling
Extended attributes allows users to set a flexible profiles.yml snippet in their dbt Cloud Environment settings. It provides users with more control over environments (both deployment and development) and extends how dbt Cloud connects to the data platform within a given environment.
Extended attributes are set at the environment level, and can partially override connection or environment credentials, including any custom environment variables. You can set any YAML attributes that a dbt adapter accepts in its profiles.yml
.
![Extended Attributes helps users add profiles.yml attributes to dbt Cloud Environment settings using a free form text box. Extended Attributes helps users add profiles.yml attributes to dbt Cloud Environment settings using a free form text box.](/img/docs/dbt-cloud/using-dbt-cloud/extended-attributes.jpg?v=2)
The following code is an example of the types of attributes you can add in the Extended Attributes text box:
dbname: jaffle_shop
schema: dbt_alice
threads: 4
username: alice
password: '{{ env_var(''DBT_ENV_SECRET_PASSWORD'') }}'
Extended Attributes don't mask secret values
We recommend avoiding setting secret values to prevent visibility in the text box and logs. A common workaround is to wrap extended attributes in environment variables. In the earlier example, password: '{{ env_var(''DBT_ENV_SECRET_PASSWORD'') }}'
will get a value from the DBT_ENV_SECRET_PASSWORD
environment variable at runtime.
How extended attributes work
If you're developing in the dbt Cloud IDE, dbt Cloud CLI, or orchestrating job runs, extended attributes parses through the provided YAML and extracts the profiles.yml
attributes. For each individual attribute:
-
If the attribute exists in another source (such as your project settings), it will replace its value (like environment-level values) in the profile. It also overrides any custom environment variables (if not itself wired using the syntax described for secrets above)
-
If the attribute doesn't exist, it will add the attribute or value pair to the profile.
Only the top-level keys are accepted in extended attributes
This means that if you want to change a specific sub-key value, you must provide the entire top-level key as a JSON block in your resulting YAML. For example, if you want to customize a particular field within a service account JSON for your BigQuery connection (like 'project_id' or 'client_email'), you need to provide an override for the entire top-level keyfile_json
main key/attribute using extended attributes. Include the sub-fields as a nested JSON block.
Git repository caching enterprise
At the start of every job run, dbt Cloud clones the project's Git repository so it has the latest versions of your project's code and runs dbt deps
to install your dependencies.
For improved reliability and performance on your job runs, you can enable dbt Cloud to keep a cache of the project's Git repository. So, if there's a third-party outage that causes the cloning operation to fail, dbt Cloud will instead use the cached copy of the repo so your jobs can continue running as scheduled.
dbt Cloud caches your project's Git repo after each successful run and retains it for 8 days if there are no repo updates. It caches all packages regardless of installation method and does not fetch code outside of the job runs.
dbt Cloud will use the cached copy of your project's Git repo under these circumstances:
- Outages from third-party services (for example, the dbt package hub).
- Git authentication fails.
- There are syntax errors in the
packages.yml
file. You can set up and use continuous integration (CI) to find these errors sooner. - If a package doesn't work with the current dbt version. You can set up and use continuous integration (CI) to identify this issue sooner.
To enable Git repository caching, select Account settings from the gear menu and enable the Repository caching option.
Partial parsing
At the start of every dbt invocation, dbt reads all the files in your project, extracts information, and constructs an internal manifest containing every object (model, source, macro, and so on). Among other things, it uses the ref()
, source()
, and config()
macro calls within models to set properties, infer dependencies, and construct your project's DAG. When dbt finishes parsing your project, it stores the internal manifest in a file called partial_parse.msgpack
.
Parsing projects can be time-consuming, especially for large projects with hundreds of models and thousands of files. To reduce the time it takes dbt to parse your project, use the partial parsing feature in dbt Cloud for your environment. When enabled, dbt Cloud uses the partial_parse.msgpack
file to determine which files have changed (if any) since the project was last parsed, and then it parses only the changed files and the files related to those changes.
Partial parsing in dbt Cloud requires dbt version 1.4 or newer. The feature does have some known limitations. Refer to Known limitations to learn more about them.
To enable, select Account settings from the gear menu and enable the Partial parsing option.
Account access to Advanced CI features beta
To help increase the governance and improve the quality of the data, you can set up automation that tests code changes before merging them into production with CI jobs. You can also enable Advanced CI features, such as compare changes, that allow dbt Cloud account members to view details about the changes between what's currently in your production environment and the pull request's latest commit, providing observability into how data changes are affected by code changes.
To use Advanced CI features, your dbt Cloud account must have access to them. Ask your dbt Cloud administrator to enable Advanced CI features on your account, which they can do by selecting Account settings from the gear menu and choosing the Enable account access to Advanced CI option.
Create a development environment
To create a new dbt Cloud development environment:
- Navigate to Deploy -> Environments
- Click Create Environment.
- Select Development as the environment type.
- Fill in the fields under General Settings and Development Credentials.
- Click Save to create the environment.
Set developer credentials
To use the dbt Cloud IDE or dbt Cloud CLI, each developer will need to set up personal development credentials to your warehouse connection in their Profile Settings. This allows you to set separate target information and maintain individual credentials to connect to your warehouse.
Deployment environment
Deployment environments in dbt Cloud are necessary to execute scheduled jobs and use other features (like different workspaces for different tasks). You can have many environments in a single dbt Cloud project, enabling you to set up each space in a way that suits different needs (such as experimenting or testing).
Even though you can have many environments, only one of them can be the "main" deployment environment. This would be considered your "production" environment and represents your project's "source of truth", meaning it's where your most reliable and final data transformations live.
To learn more about dbt Cloud deployment environments and how to configure them, refer to the Deployment environments page. For our best practices guide, read dbt Cloud environment best practices for more info.