Add DBOS decorators to your Python, TypeScript, Go, or Java functions to save workflow progress in Postgres. If the process crashes, the workflow can resume from its last checkpoint. You don’t need to run a separate orchestrator.
With ZenML, you write pipelines and agents in Python. Dynamic pipelines let you use loops and branches, pause for approval with wait(), and retry failed runs without repeating completed steps. ZenML saves each step’s output as a versioned artifact in your own storage and supports sandboxes for agent code. Change the stack configuration to run the same pipeline on Kubernetes, Vertex AI, SageMaker, AzureML, or other supported infrastructure.
DBOS remembers where a workflow got to. ZenML tracks completed steps and saves versioned outputs alongside the inputs and code used for each run. Its retry and resume features work without DBOS.
ZenML
Use ZenML if you are
- Orchestrating Python pipelines and agents (training, batch inference, evaluation, tool-use loops) and want versioned artifacts and lineage by default
- Pausing an agent run for a human approval with wait(), then resuming it from the dashboard, CLI, or API
- Running the same pipeline code on Kubernetes, Vertex AI, SageMaker, AzureML, or other supported infrastructure
- Putting two runs of the same pipeline side by side and needing to see exactly what changed
DBOS
Use DBOS if you are
- Adding crash recovery to transactional app workflows, and you already run Postgres
- Wanting durability as a library inside the service you already have, with nothing new to operate
- Writing in TypeScript, Go, or Java as well as Python, under one durability contract
- Depending on queues, rate limits, exactly-once messaging, and resume from inside a single function
DBOS is built for app workflows on Postgres. ZenML is built for pipelines and agents whose outputs you have to trust twice: once when they run, and again a month later.
The step is the unit of record
DBOS’s unit is the workflow step, and what it persists is the state needed to resume. ZenML’s unit is the @step, and it persists two things: that the step finished, so a retry or resume skips it, and what the step produced, versioned in the artifact store and attached to the run.
openai.chat.completions.create(prompt=…)OTel traces via instrumentation. You own alias resolution and secrets.- artifact
- brief · v4 · stored on the run
- metadata
- whatever you log on the step
- What’s stored: DBOS checkpoints step state to Postgres so execution can resume. ZenML writes the step’s output to your bucket (S3, GCS, Azure Blob, MinIO) and the run metadata to the ZenML server. Self-host the server or use ZenML Pro; the artifacts stay in your bucket either way.
- Naming: Return
Annotated[str, "brief"]and you fetch that artifact by name across runs, not by an opaque id. - Metadata: ZenML has no LLM-specific primitive. It gives you somewhere to put the numbers:
log_metadata()pins token counts, cost, or anything else to the step, and it shows up on the run next to the artifact.
Pause for approval, resume later
DBOS pauses a workflow with DBOS.sleep() or DBOS.recv(), checkpointed in Postgres, and carries on when the timer fires or a message lands. ZenML pauses a dynamic pipeline with wait(). The run goes PAUSED, someone answers the question, and the run picks up where it stopped.
DBOS.recv("approval", timeout_seconds=86400)Checkpointed in Postgres; the workflow resumes on delivery- status
- PAUSED · orchestration process torn down
- resolve
- dashboard · CLI · API → true
- resume
- zenml pipeline runs resume 9f2a
- Dynamic pipelines:
@pipeline(dynamic=True)builds the graph at run time from plain Python. Loop over agent tasks, branch on a result, fan out with.map().wait()lives here and only here: not inside a step body, not in a static pipeline. - What happens while it waits:
wait(schema=bool, question=...)polls until its timeout, then marks the runPAUSEDso the orchestration process can go away. Answer it from the dashboard, the API, orzenml pipeline runs wait-conditions resolve, thenzenml pipeline runs resumecarries on without re-running the steps that finished.on_pauseandon_resumehooks fire on the way. - Failure handling:
StepRetryConfig(max_retries=3, delay=10, backoff=2)retries a step. Execution modes decide whether the run fails fast, stops, or keeps going.zenml pipeline runs retryrestarts a failed dynamic run, and completed steps are reused rather than re-executed.
Artifact versioning across every step
A Postgres checkpoint table tells you a workflow completed. It doesn’t tell you this run’s embeddings differ from last week’s, and it won’t hand last week’s back to you. ZenML versions every step output, so a run is a set of artifacts you can open and load, not a log line.
- Versions: Re-run a step with different inputs and you get a new version of the same named artifact, not an overwrite. Both stay loadable. Steps you didn’t touch come from cache on every run, keyed on the step, its parameters, and its inputs.
- Cross-run load: A later run, or a notebook, pulls an earlier run’s output by name with
get_artifact_version(...).load()on the ZenML client. - Sandboxes: An agent that runs generated code gets a
sandboxstack component with local, Docker, Kubernetes, and Modal flavors:Client().active_stack.sandbox.create_session(), thensession.exec([...]). Docker and Modal add snapshot and restore. DBOS leaves isolation to your service.
Deploy a pipeline and call it by name
Both turn code into something you can call. Here’s the difference: what you get back after it runs.
- Serving:
zenml pipeline deploy run.weather_agent --name weather-agent-servicestands the pipeline up behind an HTTP endpoint, with local, Docker, Kubernetes, GCP Cloud Run, AWS App Runner, and Hugging Face deployer flavors. - Per-request lineage: Every request to that endpoint is a ZenML run, so one call gets the same artifacts, metadata, and step history a batch job does. Want to test a change to that agent against real production runs before it ships? Pair ZenML with Kitaru, replay-based regression testing for agents.
- DBOS: You invoke durable workflows from inside your own service, with queues, scheduled workflows, and DBOS Cloud on hand. Smaller footprint; the run record is yours to keep.
What makes ZenML different
| Feature | ZenML | DBOS | What that means |
|---|---|---|---|
| Durable human-in-the-loop wait and resume | Yes | Yes | DBOS: durable sleep, recv, and events. ZenML: wait() in a dynamic pipeline; the run pauses and resumes with completed steps reused. |
| Loops, branches, and fan-out decided at run time | Yes | Yes | ZenML dynamic pipelines use plain Python control flow, as DBOS workflows do. |
| Recover after failure without re-executing completed work | Partial support | Yes | DBOS resumes from inside the function via its Postgres checkpoint. ZenML retries at step granularity and reuses completed steps. |
| Caching on ordinary re-runs, not only on recovery | Yes | Not supported | ZenML skips steps whose parameters and inputs are unchanged on every run. |
| Versioned artifacts and lineage across runs | Yes | Not supported | Step outputs are stored, versioned, and loadable by name; DBOS persists state, not artifacts. |
| Isolated sandboxes for agent tool loops | Yes | Not supported | ZenML sandbox stack component (local, Docker, Kubernetes, Modal). DBOS leaves isolation to your service. |
| One stack abstraction for your clouds | Yes | Not supported | Configure once, every pipeline uses it. DBOS runs wherever your service runs. |
| Serve it and get lineage per request | Yes | Partial support | zenml pipeline deploy makes every request a tracked run. |
| Runs as a library with no orchestrator to operate | Not supported | Yes | DBOS needs only Postgres. ZenML expects a server and a configured stack. |
| Queues, rate limits, and exactly-once messaging | Not supported | Yes | First-class in DBOS. ZenML leaves queueing to the orchestrator. |
| TypeScript, Go, and Java as well as Python | Not supported | Yes | ZenML is Python only. |
| Open source, self-hostable | Yes | Yes |
How the two surfaces map
| Concept | DBOS | ZenML |
|---|---|---|
| Workflow boundary | @DBOS.workflow | @pipeline or @pipeline(dynamic=True) |
| Unit of work | @DBOS.step | @step |
| What persists | Workflow state in Postgres | Step completion plus outputs as versioned artifacts |
| Skipping completed work | Checkpoint resume after a crash | enable_cache on every run; completed steps reused on retry and resume |
| Durable pause | DBOS.sleep, DBOS.recv, events | wait(schema, question) in a dynamic pipeline; zenml pipeline runs resume |
| Where it runs | Inside your service, or DBOS Cloud | Stack orchestrator (zenml stack set) |
| Cross-run reuse | Bring your own store | get_artifact_version(...).load() |
| Serving | Invoke from your own service | zenml pipeline deploy --name ... |
Code comparison
from typing import Annotated
from zenml import log_metadata, pipeline, step, wait
@step
def plan(goal: str) -> Annotated[list[str], "tasks"]:
tasks = call_llm(f"Break down: {goal}")
# No llm() primitive: attach whatever you want recorded.
log_metadata(metadata={"model": "fast", "tasks": len(tasks)})
return tasks
@step
def research(task: str) -> Annotated[str, "finding"]:
return call_llm(f"Research: {task}")
@step
def deploy(findings: list[str]) -> Annotated[dict, "release"]:
return ship(findings)
@pipeline(dynamic=True)
def agent_pipeline(goal: str):
tasks = plan(goal)
findings = research.map(task=tasks) # fan-out decided at run time
# Pause for approval. The run goes PAUSED, the process can be
# torn down, and a human resolves it from the dashboard or CLI.
approved = wait(schema=bool, question="Approve and deploy?")
if approved:
deploy(findings)
agent_pipeline("prepare the release notes")
# Resolve and resume later; completed steps are reused:
# zenml pipeline runs wait-conditions resolve --run <id> --interactive
# zenml pipeline runs resume <id>
# Same code on another cloud:
# zenml stack set sagemaker && python agent_pipeline.pyfrom dbos import DBOS
@DBOS.step()
def plan(goal: str) -> list[str]:
return call_llm(f"Break down: {goal}")
@DBOS.step()
def research(task: str) -> str:
return call_llm(f"Research: {task}")
@DBOS.step()
def deploy(findings: list[str]) -> dict:
return ship(findings)
@DBOS.workflow()
def agent_workflow(goal: str) -> dict:
tasks = plan(goal)
findings = [research(t) for t in tasks]
# Durable wait: checkpointed in Postgres. Another process
# delivers the answer with DBOS.send(workflow_id, ok, "approval").
approved = DBOS.recv("approval", timeout_seconds=86400)
if not approved:
return {"status": "rejected"}
return deploy(findings)
# Durability comes from the library plus your Postgres.
# No orchestrator process to operate, and no artifact
# store: what a step returned is yours to persist.
DBOS(config={"name": "agent"})
DBOS.launch()One orchestrator for
pipelines and agents
If you want crash-proof app workflows with the smallest possible footprint, and you already run Postgres, DBOS is a clean answer. If the work is a Python pipeline or agent that needs an approval gate, versioned outputs, step retries, and the same code on Kubernetes today and Vertex AI next quarter, ZenML is the orchestrator. You don’t need DBOS underneath it.









