Orchestration on Docker¶

APIs¶

dagster_docker.DockerRunLauncher RunLauncher[source]¶

Config Schema:

image (dagster.StringSource, optional):

The docker image to be used if the repository does not specify one.

network (dagster.StringSource, optional):

Name of the network to which to connect the launched container at creation time

registry (strict dict, optional):

Information for using a non local/public docker registry

Config Schema:

url (dagster.StringSource):

username (dagster.StringSource):

password (dagster.StringSource):

env_vars (List[String], optional):

The list of environment variables names to include in the docker container. Each can be of the form KEY=VALUE or just KEY (in which case the value will be pulled from the local environment)

container_kwargs (permissive dict, optional):

key-value pairs that can be passed into containers.create. See https://docker-py.readthedocs.io/en/stable/containers.html for the full list of available options.

networks (List[dagster.StringSource], optional):

Names of the networks to which to connect the launched container at creation time

Launches runs in a Docker container.

dagster_docker.docker_executor ExecutorDefinition[source]¶

Config Schema:

image (dagster.StringSource, optional):

The docker image to be used if the repository does not specify one.

network (dagster.StringSource, optional):

Name of the network to which to connect the launched container at creation time

registry (strict dict, optional):

Information for using a non local/public docker registry

Config Schema:

url (dagster.StringSource):

username (dagster.StringSource):

password (dagster.StringSource):

env_vars (List[String], optional):

The list of environment variables names to include in the docker container. Each can be of the form KEY=VALUE or just KEY (in which case the value will be pulled from the local environment)

container_kwargs (permissive dict, optional):

key-value pairs that can be passed into containers.create. See https://docker-py.readthedocs.io/en/stable/containers.html for the full list of available options.

networks (List[dagster.StringSource], optional):

Names of the networks to which to connect the launched container at creation time

retries (selector, optional):

Whether retries are enabled or not. By default, retries are enabled.

Default Value:

{
    "enabled": {}
}

Config Schema:

enabled (strict dict, optional):

Default Value:

{}

disabled (strict dict, optional):

Default Value:

{}

Executor which launches steps as Docker containers.

To use the docker_executor, set it as the executor_def when defining a job:

from dagster import job
from dagster_docker import docker_executor

@job(executor_def=docker_executor)
def docker_job():
    pass

Then you can configure the executor with run config as follows:

execution:
  config:
    registry: ...
    network: ...
    networks: ...
    container_kwargs: ...

If you’re using the DockerRunLauncher, configuration set on the containers created by the run launcher will also be set on the containers that are created for each step.