Long-running operations like cluster initialization and deployment rollouts are now powered by Hatchet, an open-source durable execution engine.
Why Hatchet#
We needed a workflow engine that could handle multi-step infrastructure operations reliably. Hatchet stood out for a few reasons:
- TypeScript-native: Workflows are defined in the same language as the rest of our stack, no YAML or DSLs
- Durable execution: If a step fails mid-workflow, it retries from where it left off rather than restarting from scratch
- DAG-based workflows: Steps can run in parallel or sequence with explicit dependency graphs
- Step-level visibility: You can see exactly which step a workflow is on and what failed
What changed#
Cluster setup, deployment rollouts, and resource cleanup now run as Hatchet workflows. Each step has its own retry policy and timeout, so transient failures (like a cloud API being slow) are handled automatically without failing the entire operation.
Progress is tracked per-step, so the UI always reflects exactly where things stand.