How We Cut Dagster Cloud Deployments from 3 Minutes to 40 Seconds

Serverless Development and Feedback Loop

Dagster is a data orchestrator. On Dagster Cloud you can develop and deploy Dagster code without setting up a local environment or cloud infrastructure. When you push changes to GitHub, a GitHub Action builds and deploys the code directly to Dagster Cloud, where you can view and interact with Dagster objects via the UI.

Docker Image Problems

Initially we used a Docker‑based build process, but the edit‑deploy‑run cycle was slow. Each commit required 3–5 minutes for the new Docker image to appear in the UI, which frustrated developers making small changes.

We measured the steps involved when a line of code is changed and committed:

~20 s – Provide GitHub runner and download the action.

~10 s – Download the Docker‑based action.

~60 s – Build and upload the user’s Docker image (cached builds can be ~60 s; uncached >90 s).

~90 s – Run the Docker image in AWS.

~180 s – Total runtime.

The two longest phases are building the Docker image (60‑90 s) and deploying the Docker container (≈90 s).

Building Docker Images

Docker images consist of stacked layers, each created by a subset of Dockerfile commands.

Each layer is identified by a hash.

Only layers not already present in the registry are uploaded.

GitHub Actions cache can pull unchanged layers, reducing build time.

Docker builds are not deterministic; identical inputs may produce different hashes.

Launching Docker Containers

We use AWS Fargate, which takes 45‑90 s to configure and start a container and does not provide image caching. Starting a new container requires downloading all layers from the registry.

Other Constraints

After the Docker image is built and started, a code server runs the user’s code to extract metadata for the UI. This step can take a few seconds to 30 s depending on the metadata computation. Reproducibility is essential; we use Docker image hashes (and later PEX file hashes) to uniquely identify code and environment.

Alternative Solutions Overview

Switch from Fargate to EC2 to speed container start‑up, at the cost of increased operational burden.

Use a different Docker build system such as AWS CodeBuild, which would require deeper GitHub integration.

Switch to AWS Lambda for faster start‑up, but Lambda’s execution limits and custom base‑image constraints make it unsuitable for long‑running servers.

Build and upload only changed code to a long‑running code server, exploring packaging tools like rsync, poetry, nix, shiv, and pex, and considering EFS for Python environments.

We chose option 4 because moving a 100 MB Docker image for a small code change is wasteful; using Git‑style diffs with PEX proved effective.

What Is PEX?

PEX (Python Executable) bundles Python packages into a single executable file. Example: % pex dagster --python=python3.8 -o dagster.pex Key features:

Isolation : At runtime the PEX environment is isolated from other packages; only bundled packages are available.

Determinism : Same input packages produce identical PEX files, enabling content‑addressable storage.

Composition : Multiple PEX files can be merged at runtime (e.g., deps.pex + source.pex).

Cross‑platform builds : Using a Linux‑based python:*‑slim base image, PEX can build Linux PEX files on any platform as long as wheels are available.

Fast Deployment

We combine PEX with S3 storage to avoid the overhead of building and starting Docker images. When code is pushed to GitHub, the GitHub Action decides between a full build (deps.pex + source.pex) or a fast build (only source.pex) based on whether dependencies have changed.

In the fast‑build path, the entire deployment completes in about 40 seconds, compared to the previous >3‑minute process.

Timeline for fast build:

Trade‑offs and Issues

Multiple environments share the same memory and CPU on a code server; a heavy environment can affect others.

Docker can build Linux wheels for any OS, while PEX can only build Linux wheels; we still use Docker for source distribution when needed.

Custom base images are required for parity between Docker and fast‑deploy paths.

GitHub Workflow and PEX

Previously the GitHub Action packaged code into a Docker image, incurring a ~10 s download and start‑up cost. Now the action code is packaged into a PEX file and run directly on the GitHub runner, eliminating the Docker download step. We also reduced the workflow to a single job, saving another ~10 s.

Conclusion

Reducing deployment time from over three minutes to forty seconds is a significant acceleration. Using PEX on top of Docker gives us a reproducible, consistent environment, and we are excited to explore further possibilities with this combination.