Kitaru vs OpenAI Agents SDK: replay-based evals for the agent you built

The OpenAI Agents SDK is OpenAI’s code-first harness: an opinionated toolkit for defining and running an agent in Python or TypeScript.

Kitaru isn’t another harness. It’s a replay-based eval layer that sits beside it: wrap the agent with KitaruRunner and every run lands as a session, with model calls, tools, hosted tools, and handoffs recorded as nodes you can inspect, replay, and evaluate. They don’t compete. They compose, and the first-class OpenAI Agents adapter does the wiring. Running that same agent durably in production is ZenML’s job; Kitaru replays and improves it. See ZenML vs OpenAI Agents SDK.

Kitaru

Use Kitaru if you are

Running OpenAI Agents SDK agents in production and want their runs turned into a suite you can run again
Deciding a model or prompt swap and want two runs over the same cohort compared side by side, not a vibe check
Turning the sessions that caught a failure into a cohort that gates the next change
Already tracing to Langfuse, LangSmith, Braintrust, Logfire, or Arize Phoenix and want those traces runnable, not just readable
Self-hosting eval infrastructure so traces and credentials stay in your own systems

OpenAI Agents SDK

Use the OpenAI Agents SDK if you are

Building an Agents SDK-native agent and want a tight code-first harness for tools, handoffs, guardrails, structured outputs, and OpenAI platform integrations
Standardizing on OpenAI's tracing, evals, sandbox agents, and hosted tools in one ecosystem
Streaming responses with run_streamed, or relying on a tool approval interruption that resumes later: the current Kitaru adapter records non-streaming runs only and fails closed on approval interruptions
Running short-lived or prototype agents where you don't need replay yet

The OpenAI Agents SDK builds the agent. Kitaru turns each run into something you can run again.

Record once, replay it

The OpenAI Agents SDK gives you the agent loop, and OpenAI’s tracing shows you what happened. Then what? A trace isn’t a run you can execute again. Kitaru’s adapter wraps Runner so every call becomes a session you can replay.

OpenAI Agents SDK alone

A trace of the run, read only

1model.requestgpt-5-nano

2tool.check_orderlive call

3handofftrace ends here

read the trace again: that's the only rerun you get

a trace can't answer “what if” about the run

Kitaru-wrapped

Recorded once, replayed for the experiment

1tool.check_orderanswered from recording

2tool.refund_paymentanswered from recording

3model.requestlive · gpt-5-mini

>>>kitaru experiment run start cheaper-model --cohort-version <id> --agent reviewer@2 --wait

named tools answered from the recording; only the model changes

One wrapper records everything. Install kitaru-openai-agents, put KitaruRunner(agent_id=...) in front of run_sync or run, and model calls, direct function tools, hosted tools, and handoffs land as nodes on one session. OpenAI’s own RunResult comes back exactly as it would without Kitaru.
Replay answers tool calls from the recording. A replay re-executes through the same runner. Direct FunctionTool calls follow a named policy: history returns the recorded result for the same canonical arguments, static returns a configured value, passthrough calls the real tool.
The model call stays live. No cached model responses. A replay can swap the root input, the starting agent’s instructions, the run-level model, and model settings, and the model call still runs live. That’s what lets you see what a different model or prompt actually does.

Nothing touches production unless you say so

Every OpenAI Agents SDK run means live tool calls too: the payment API, the ticketing system, whatever your agent touches. That’s fine in production. It’s a liability in a test.

Rerun outside KitaruEvery check hits your real systems again

tool call: refund_payment()

calls the live payments API

response comes back, maybe different this time

Real systems touched100%

Kitaru replayTool calls answered from the recording

tool policy: history, named per tool

recording answers the call

same inputs reproduce the original run

Real systems touched0%

Passthrough is the default, and stays the default. This adapter doesn’t support a history default. Leave the default at passthrough and add a named history or static entry for each direct function tool you want answered from the recording. Those tools never get called live.
Only direct function tools get substituted. A named policy has to match one ordinary, enabled, non-approval FunctionTool on the starting agent. Try to substitute a hosted tool, an MCP tool, an agent-as-tool, or a handoff target and the adapter rejects it before a session exists, so a misconfigured policy fails before OpenAI ever calls the model.
Set per replay, not per call. A Kitaru worker selects the replay via KITARU_REPLAY_ID; there’s no per-run replay argument to thread through your own code.

A session you can inspect, not just a trace you can read

OpenAI’s tracing is good: model calls, tool calls, handoffs, guardrails, and custom spans all land in the dashboard. What it doesn’t give you is something you can run again and evaluate. A Kitaru session is exactly that.

OpenAI traceWhat happened

model.request218ms

tool.search_docs412ms

model.request186ms

handoff9ms

model.request324ms

Read-only. Reproduce by re-running.

Kitaru sessionWhat you can replay

session 7c1b3 nodes · $0.15

model.requestrecorded

tool.search_docsrecorded

handoffrecorded

replayevaluatecompare

Every node, one session. Model calls, direct tool calls, hosted tools, and handoffs land as nodes on the session the adapter creates before OpenAI executes the agent. A session_observer callback hands you the session id so you can tie it to the native result.
Replay, then evaluate. A replayed session gets the same evaluators as the original: deterministic checks first, a model-graded check for what’s left.
Compare, not just read. Freeze the sessions that matter into a cohort version, run an experiment with one variable moved, and read the evaluations on both sides.

What makes Kitaru unique

Feature	Kitaru	OpenAI Agents SDK	What that means
First-class Agent abstraction with instructions, tools, handoffs, guardrails	Not supported	Yes
First-party tracing, evals, and hosted tools	Not supported	Yes
Streaming responses (run_streamed)	Not supported	Yes	The adapter records run and run_sync only.
Resume a tool approval interruption (RunState)	Not supported	Yes	Approval interruptions fail closed in the adapter; RunState input is rejected.
Every run recorded as a replayable session (model calls, tools, hosted tools, handoffs)	Yes	Not supported
Direct function tools answered from the recording during replay (named history or static policy)	Yes	Not supported	Hosted, MCP, and handoff tools are recorded but not substituted.
Cohorts: frozen session sets as regression suites	Yes	Not supported
Experiments: same cohort, one variable moved, compared side by side	Yes	Not supported
Import Langfuse / LangSmith / Braintrust / Logfire / Phoenix traces	Yes	Not supported
Self-hosted server and workers (Apache 2.0)	Yes	Not supported
First-class adapter for the other	Yes	Not supported

How the two surfaces map

Concept	OpenAI Agents SDK	Kitaru
Layer	Agent harness (how the agent thinks and acts)	Replay-based evals (how you test what it did)
Core unit	Agent run	Session: recorded model, tool, and handoff nodes, replayable
Composition	`Runner.run_sync(agent, input)`	Same agent through `KitaruRunner(agent_id=...).run_sync(agent, input)`
Tool calls in a replay	Hit real systems, or you mock them by hand	Passthrough by default; named `history` or `static` per direct function tool
“Did my change help?”	Read the trace, compare by eye	Two runs over the same cohort, evaluated on both sides
Streaming	`run_streamed()` with live events	Not supported by the current adapter
Tool approval interruptions	Native `RunState` interrupt and resume	Fails closed; not supported by the adapter
Where it runs	Whatever Python service you wrap the agent in	Self-hosted server; workers replay in your environment

Code comparison

OpenAI Agents SDK + KitaruRecommended

import uuid
from agents import Agent
from kitaru_openai_agents import KitaruRunner

reviewer = Agent(
  name="reviewer",
  instructions="Review the draft for compliance.",
  model="gpt-5-nano",
)
runner = KitaruRunner(agent_id=uuid.UUID(AGENT_ID))

# Runs exactly as before, and lands as a session:
# every model call, tool call, and handoff.
result = runner.run_sync(reviewer, case)
print(result.final_output)

# Later: freeze the sessions that matter, test a change.
# Default stays passthrough; name each tool to replay.
#   kitaru cohort create hard-cases --agent reviewer --tag needs-review
#   kitaru experiment create cheaper-model --agent reviewer \
#     --evaluator compliance-check@latest \
#     --override '{"model": {"gpt-5-nano": "gpt-5-mini"}}' \
#     --tool-policy '{"default": {"type": "passthrough"},
#       "tools": {"check_order": {"type": "history",
#       "scope": "cohort_version", "on_miss": "fail"}}}'
#   kitaru experiment run start cheaper-model \
#     --cohort-version <id> --agent reviewer@2 --evaluate-baselines --wait

OpenAI Agents SDK alone

from agents import Agent, Runner

reviewer = Agent(
  name="reviewer",
  instructions="Review the draft for compliance.",
  model="gpt-5-nano",
)

result = Runner.run_sync(reviewer, case)
print(result.final_output)
# One trace per run, in the OpenAI dashboard. The run that
# caught the bug is a transcript you can read, not a test
# you can run again with the model swapped.

Put replay-based evals under your OpenAI agents

If your OpenAI agent still fits in a notebook or a short-lived script, the SDK on its own is the right answer. If it’s shipping, writing to real systems and generating traffic you wish you could test against, Kitaru makes those runs runnable: record them as sessions, replay them as evals, and gate regressions with the cohort that caught the failure.

curl -fsSL https://kitaru.ai/install | bash