Contributing#

We love to see our community members get involved! If you are planning to contribute to Dagster, you will first need to set up a local development environment.

Environment Setup#

Apple M1-specific setup#

If you have a Mac with an M1 chip, you will need to set up a separate environment to emulate an x86 architecture before following the remaining setup instructions. To find out if your Mac has an M1 chip, follow the instructions here.

If you don't have a Mac with an M1 chip, skip ahead to the next section - Dagster development setup.

  1. Install rosetta. About rosetta.

    softwareupdate --install-rosetta --agree-to-license

  2. Create a duplicate terminal that will default to running the rosetta x86 emulator. Source stack overflow post.

    • Go to Finder > Applications and find your terminal app.
    • Right click the app and duplicate it.
    • Right click the new terminal > Get Info > Enable Open using Rosetta.
    • Click to open the terminal, type arch to verify it says i386 now.

  3. In your rosetta terminal, install homebrew.

    arch -x86_64 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

  4. To ensure that the correct installation of homebrew is automatically used in your rosetta terminal, add the following snippet to your ~/.bashrc, ~/.zshrc or profile manager of choice.

    arch_name="$(uname -m)"

if [ "${arch_name}" = "x86_64" ]; then
   echo "Running in x86 mode"
   eval $(/usr/local/bin/brew shellenv)
elif [ "${arch_name}" = "arm64" ]; then
   echo "Running in Arm mode"
   eval $(/opt/homebrew/bin/brew shellenv)
else
   echo "Unexpected uname -m result ${arch_name}"
fi

# brew libraries
export LDFLAGS="-L $(brew --prefix openssl)/lib"
export CFLAGS="-I $(brew --prefix openssl)/include"

After sourcing this file, you can ensure that the correct version of homebrew is being used by confirming the output of which brew is /usr/local/bin/brew.

Dagster development setup#

  1. Install Python. Python 3.6 or above recommended, but our CI/CD pipeline currently tests against up-to-date patch versions of Python 3.6, 3.7 and 3.8.

  2. Create and activate a virtualenv, using the tool of your choice. On macOS you can install pyenv with Homebrew:

    brew install pyenv pyenv-virtualenv

    Then add the following commands to your shell profile:

    eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

    and finally create and activate the virtualenev:

    pyenv install 3.7.4
pyenv virtualenv 3.7.4 dagster37
pyenv activate dagster37

  3. Ensure that you have node installed by running node -v, and that you have yarn installed. If you are on macOS, you can install yarn with Homebrew:

    brew install yarn

  4. Clone the Dagster repository to the destination of your choice:

    git clone git@github.com:dagster-io/dagster.git

  5. Run make dev_install at the root of the repository. This sets up a full Dagster developer environment with all modules and runs tests that do not require heavy external dependencies such as docker. This will take a few minutes. Note that certain sections of the makefile (sanity_check, which is part of rebuild_dagit) require POSIX compliant shells and will fail on CMD and powershell -- if developing on windows, using something like WSL or git-bash is recommended. Note also that if this command fails while installing python packages, the problem might be resolved by ensuring you are running an up-to-date version of pip (upgrade with pip install -U pip).

    make dev_install

  6. Run some tests manually to make sure things are working:

    python -m pytest python_modules/dagster/dagster_tests

Developing Dagster#

Some notes on developing in Dagster:

  • Black/Pylint: We use black to enforce a consistent code style, along with pylint. We test these in our CI/CD pipeline.
    • You can run make black from the repo root to automatically format your files.
  • Line Width: We use a line width of 100.
  • Import ordering: We use isort to standardize the order of imports.
    • You can run "make isort" from the repo root to automatically isort your files.
  • IDE: We recommend setting up your IDE to format with black on save and check pylint, but you can always run make black and make pylint in the root Dagster directory before submitting a pull request. If you're also using VS Code, you can see what we're using for our settings.json here.
  • Docker: Some tests require Docker Desktop to be able to run them locally.

Developing Dagit#

For development, run the Dagit GraphQL server on a different port than the webapp, with any pipeline. For example:

cd dagster/examples/docs_snippets/docs_snippets/intro_tutorial/basics/connecting_solids/
dagit -p 3333 -f complex_pipeline.py

Keep this running. Then, in another terminal, run the local development (autoreloading, etc.) version of the webapp:

cd dagster/js_modules/dagit
make dev_webapp

During development, you might find these commands useful. Run them from dagster/js_modules/dagit:

  • yarn ts: Typescript typechecking
  • yarn lint: Linting with autofix
  • yarn jest: An interactive Jest test runner that runs only affected tests by default

To run all of them together, run yarn test.

Developing Docs#

To run the Dagster documentation website locally, run the following commands:

cd docs
make next-watch-build   # Serves the docs website on http://localhost:3001

Troubleshooting tip: You may need to run make next-dev-install first to install dependencies. Also make sure that your Node version is >=12.13.0.

The API documentation is generated from ReStructured Text files (.rst), which extracts Python docstrings from the library files. The .rst files can be found in the docs/sphinx/sections/api/apidocs directory.

If you change any .rst files, be sure to run the following command in the docs directory:

make apidoc-build

The majority of our code snippets are pulled from real Python files. This allows us to test our code snippets and ensure they remain up-to-date.

In .mdx files, you'll see a code snippet with python file=/path/to/file.py startafter=start_marker endbefore=end_marker at the beginning of the block. For example:

You can find the corresponding Python file at dagster/examples/docs_snippets/docs_snippets/concepts/asset/asset_dependenyy.py. The code included in each snippet is the code in the file between the # start_marker and # end_marker comments.

To change the code snippet, update the .py file, then run the following from the docs directory:

make mdx-format

You can find more information about developing documentation in `docs/README.md`.

## Picking a Github Issue

We encourage you to start with an issue labeled with the tag [`good first issue`](https://github.com/dagster-io/dagster/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) on the [Github issue board](https://github.com/dagster-io/dagster/issues), to get familiar with our codebase as a first-time contributor.

Then, you can work on the issue labeled as [`good second issue`](https://github.com/dagster-io/dagster/labels/good%20second%20issue) which is more like a medium task.

When you are ready for more of a challenge, you can tackle issues with the [most 👍 reactions](https://github.com/dagster-io/dagster/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-%2B1-desc). We factor engagement into prioritization of the issues. You can also explore other labels and pick any issue based on your interest.

## Submit Your Code

To submit your code, [fork the Dagster repository](https://help.github.com/en/articles/fork-a-repo), create a [new branch](https://help.github.com/en/desktop/contributing-to-projects/creating-a-branch-for-your-work) on your fork, and open [a Pull Request (PR)](https://help.github.com/en/articles/creating-a-pull-request-from-a-fork) once your work is ready for review.

In the PR template, please describe the change, including the motivation/context, test coverage, and any other relevant information. Please note if the PR is a breaking change or if it is related to an open GitHub issue.

A Core reviewer will review your PR in around one business day and provide feedback on any changes it requires to be approved. Once approved and all the tests (including Buildkite!) pass, the reviewer will click the Squash and merge button in Github 🥳.

Your PR is now merged into Dagster! We’ll shout out your contribution in the weekly release notes.