Dagster Cloud environment variables and secrets#

This guide is applicable to Dagster Cloud.

Environment variables, which are key-value pairs configured outside your source code, allow you to dynamically modify application behavior depending on environment.

Using environment variables, you can define various configuration options for your Dagster application and securely set up secrets. For example, instead of hard-coding database credentials - which is bad practice and cumbersome for development - you can use environment variables to supply user details. This allows you to parameterize your pipeline without modifying code or insecurely storing sensitive data.


Understanding environment variables and secrets#

Overview#

There are two ways to declare and manage variables in Dagster Cloud:

Dagster Cloud UIAgent configuration
Deployment type support Serverless, Hybrid Hybrid
How it worksEnvironment variables are managed in the Dagster Cloud UI. Values are pulled from storage and decrypted when your code is executed.Environment variables are defined in the agent's configuration. Variables set at the code location level will pass through Dagster Cloud, while those set at the deployment level bypass Dagster Cloud entirely. Refer to the Setting environment variables for Dagster Cloud agents guide for more info.
RequirementsAbility to modify your dagster.yaml and dagster_cloud.yaml files
Limitations
  • Maximum of 1,000 variables per full deployment
  • Variables must be less than or equal to 4KB in size
  • Variable names:
    • Must be 512 characters or less in length
    • Must start with a letter or underscore
    • Must contain only letters, numbers, and underscores
    • May not be the same as built-in (system) variables
Variable names:
  • Must start with a letter or underscore
  • Must contain only letters, numbers, and underscores
Storage and encryptionUses Amazon Key Management Services (KMS) and envelope encryption. Refer to the Storage and encryption section for more info.Dependent on agent type (ex: Kubernetes)
ScopeScoped by deployment (full and branch) and optionally, code locationScoped by code location. Variables can be set for a full deployment (all code locations) or on a per-code location basis.

Storage and encryption#

This section is applicable only if using the Dagster Cloud UI to manage environment variables.

To securely store environment variables defined using the Dagster Cloud UI, Dagster Cloud uses Amazon Key Management Services (KMS) and envelope encryption. Envelope encryption is a multi-layered approach to key encryption. Plaintext data is encrypted using a data key, and then the data under the data key is encrypted under another key.

Here's a look at how it works in Dagster Cloud:

Dagster Cloud encryption key hierarchy diagram

In Dagster Cloud, each customer account is assigned a unique key, which then encrypts the data associated with that account. All customer keys are encrypted using a non-exportable AWS KMS master key.

This approach isolates each account's data and reduces the risk of exposure by limiting the amount of data a single key can access.

Scope#

This section is applicable only if using the Dagster Cloud UI to manage environment variables.

Environment variables can be scoped to specific deployments and/or code locations. When creating or modifying an environment variable, you'll be prompted to select the deployment(s) to scope the variable to:

  • Full deployment - Variables with this scope will be available to selected code locations in the full deployment

  • Branch deployments - Variables with this scope will be available to selected code locations in Branch Deployments.

    Note: While viewing a Branch Deployment in Dagster Cloud, variables will be read-only. Environment variables must be managed in the Branch Deployment's parent full deployment, which will usually be prod.

  • Local - Variables with this scope will be included when downloading variables to a local .env file

By default, new environment variables default to all deployments and all code locations.

Same variable, different scopes and values#

You can create multiple instances of the same environment variable with different values for different scopes. Each instance of the variable can then be scoped to a deployment and code location (or locations). This approach can be useful for parameterizing behavior by environment without needing to modify your application code.

For example, let's say we want to use different database passwords in production and testing (Branch Deployments). In our code, we use the SNOWFLAKE_PASSWORD environment variable to pass in the database password. To use different passwords between production and Branch Deployments, we can create two instances of SNOWFLAKE_PASSWORD. One instance is scoped to the prod deployment and the other only to Branch Deployments:

Example SNOWFLAKE_PASSWORD variables configured with different values based on deployment

In this example, the value of SNOWFLAKE_PASSWORD would be production_password in the prod deployment and testing_password in a Branch Deployment.

Reserved variables#

This section is applicable only if using the Dagster Cloud UI to manage environment variables.

Built-in (system) Dagster Cloud environment variables are reserved and therefore unavailable for use. An error will surface in Dagster Cloud if a built-in variable name is used.


Managing environment variables#

The simplest way to manage environment variables is to use Dagster Cloud's built-in manager which allows you to create and manage environment variables right in the UI.

Using the Dagster Cloud UI#

To manage environment variables using the Dagster Cloud UI, you need:

Creating new variables#

  1. Sign in to your Dagster Cloud account.

  2. Click Deployment > Environment variables.

  3. Click + Add Environment Variable to add a new environment variable.

  4. In the window that displays, fill in the following:

    • Name - Enter a name for the environment variable. This is how the variable will be referenced in your code.
    • Value - Enter a value for the environment variable.
    • In Deployment Scope, select the deployment(s) where the variable should be accessible:
      • Full deployment - The variable will be available to selected code locations in the full deployment
      • Branch deployments - The variable will be available to selected code locations in Branch Deployments
      • Local - If selected, the variable will be included when exporting environment variables to a local .env file
    • In Code Location Scope, select the code location(s) where the variable should be accessible. At least one code location is required. Refer to the Scope section for more info.

    For example:

    Create new environment variable dialog window in Dagster Cloud
  5. When finished, click Save. Dagster Cloud will automatically re-deploy the workspace to apply the changes.

Editing, viewing, and deleting variables#

After the environment variable is created:

  • To edit an environment variable, click the Edit button.
  • To view an environment variable's value, click the eye icon in the variable's Value column. Click the icon again to conceal the value.
  • To delete an environment variable, click the Trash icon and confirm the deletion when prompted.

Exporting variables to a .env file#

All variables with the local deployment scope can be exported to an .env file and used locally. To create the file:

  1. In the Environment variables tab, click the menu next to + Add environment variable:

    Highlighted Download local variables file in Environment Variables tab of Dagster Cloud
  2. Click Download local environment variables

  3. Save the file. Note: If saved to the same folder where you run Dagit, Dagster will automatically load the variables in the .env file when Dagit is started.

Using agent configuration (Hybrid)#

Only supported for Hybrid deployments.

For Dagster Cloud Hybrid deployments, making environment variables accessible is accomplished by adding them to your agent's configuration. How this is accomplished depends on the agent type.

Refer to the Setting environment variables for Dagster Cloud agents guide for more info.


Accessing environment variables in Dagster code#

Ready to start using environment variables in your Dagster code? Refer to the Using environment variables and secrets in Dagster code guide for more info and examples.


Built-in environment variables#

Dagster Cloud provides a set of built-in, automatically populated environment variables, such as the name of a deployment or details about a branch deployment commit, that can be used to modify behavior based on environment.

All deployment variables#

The following variables are available in every deployment of your Dagster Cloud instance, including full (e.g., prod) and branch deployments.

PropertyDescription
DAGSTER_CLOUD_DEPLOYMENT_NAMEThe name of the Dagster Cloud deployment. For example, prod.
DAGSTER_CLOUD_IS_BRANCH_DEPLOYMENTIf 1, the deployment is a branch deployment. Refer to the Branch Deployment variables section for a list of variables available in branch deployments.

Branch Deployment variables#

The following environment variables are currently available only in a branch deployment.

For every commit made to a branch, the following metadata is available:

PropertyDescription
DAGSTER_CLOUD_GIT_SHAThe SHA of the commit.
DAGSTER_CLOUD_GIT_TIMESTAMPThe time the commit occurred.
DAGSTER_CLOUD_GIT_AUTHOR_EMAILThe email of the git user who authored the commit.
DAGSTER_CLOUD_GIT_AUTHOR_NAMEThe name of the git user who authored the commit.
DAGSTER_CLOUD_GIT_MESSAGEThe message associated with the commit.
DAGSTER_CLOUD_GIT_BRANCHThe name of the branch associated with the commit.
DAGSTER_CLOUD_GIT_REPOThe name of the repository associated with the commit.
DAGSTER_CLOUD_PULL_REQUEST_IDThe ID of the pull request associated with the commit.
DAGSTER_CLOUD_PULL_REQUEST_STATUSThe status of the pull request at the time of the commit. Possible values are OPEN, CLOSED, and MERGED.

Troubleshooting#

ErrorDescription and resolution
[variable] is a reserved environment variable name.Dagster reserves the names of built-in (system) variables. To resolve, change the variable's name to a name that isn't currently reserved and meets the other naming requirements.
Environment variables must be no greater than 4KB in size.To resolve, reduce the size of the environment variable's value to less than the maximum of 4KB.
Environment variable names must be no longer than 512 characters.To resolve, reduce the number of characters in the variable's name.
Invalid environment variable name [variable]The name of the environment variable doesn't meet one or several of Dagster's naming requirements. To resolve, change the variable's name to:
  • Start with a letter or underscore
  • Contain only letters, numbers, and underscores
Deployment [deployment_name] has reached the maximum of 1,000 environment variables.The maximum number of environment variables for the full deployment has been reached. New variables cannot be added. Remove any unneeded variables to reduce the total below the maximum, then add new variables.