The OpenAI Agents SDK is the code-first harness for defining and running an agent. It’s OpenAI’s opinionated stack for building agent behavior in Python and TypeScript.
Kitaru is not another harness. It’s a replay-based eval layer that sits beside it: wrap the agent with KitaruRunner and every run lands as a Kitaru session — model calls, tools, hosted tools, and handoffs recorded as nodes you can inspect, replay, and score. The two are complementary, with a first-class OpenAI Agents adapter that does the wiring.
Kitaru
Use Kitaru if you are
- Running OpenAI Agents SDK agents in production and want their runs turned into a suite you can run again
- Deciding a model or prompt swap and want two runs over the same cohort compared side by side, not a vibe check
- Turning the sessions that caught a failure into a cohort that gates the next change
- Already tracing to Langfuse, LangSmith, or Braintrust and want those traces runnable, not just readable
- Self-hosting eval infrastructure so traces and credentials stay in your own systems
OpenAI Agents SDK
Use the OpenAI Agents SDK if you are
- Building an Agents SDK-native agent and want a tight code-first harness for tools, handoffs, guardrails, structured outputs, and OpenAI platform integrations
- Standardizing on OpenAI's tracing, evals, sandbox agents, and hosted tools in one ecosystem
- Streaming responses, or relying on a tool approval interruption that resumes later — the current Kitaru adapter doesn't record streamed runs or resume approvals durably
- Running short-lived or prototype agents where you don't need replay yet
The OpenAI Agents SDK builds the agent. Kitaru turns each run into something you can run again.
Record once, replay it
The OpenAI Agents SDK gives you the agent loop, and OpenAI’s own tracing shows what happened. What tracing doesn’t give you is a run you can execute again. Kitaru’s adapter wraps Runner so every call becomes a session you can replay.
OpenAI Agents SDK alone
A trace of the run — read only
1model.requestgpt-5-nano
2tool.check_orderlive call
3handofftrace ends here
read the trace again — that's the only rerun you get
a trace can't answer “what if” about the run
Kitaru-wrapped
Recorded once, replayed for the experiment
1tool.check_orderanswered from recording
2tool.refund_paymentanswered from recording
3model.requestlive · gpt-5-mini
>>>experiment.run(cohort=cohort, version="pr-311")
tool calls stay faithful to the recording — only the model changes
- One wrapper records everything.
KitaruRunner(agent_id=...)in front ofrun_syncorrun— model calls, tools, hosted tools, and handoffs land as nodes on one session, and OpenAI’s ownRunResultcomes back exactly as it would without Kitaru. - Replay answers tool calls from the recording. A replay re-executes through the same runner. Tool calls follow a policy — passthrough calls the real tool, static returns the recorded value — so a comparison doesn’t have to touch production systems.
- The model call stays live. There’s no cached model response. A replay’s model call runs live, which is what lets you see what a different model or prompt actually does.
Nothing touches production unless you say so
Every OpenAI Agents SDK run means live tool calls too — the payment API, the ticketing system, whatever your agent touches. That’s fine in production. It’s a liability in a test.
Rerun outside KitaruEvery check hits your real systems again
tool call: refund_payment()
response comes back — maybe different this time
Real systems touched100%
Kitaru replayTool calls answered from the recording
tool policy: static substitution
same inputs reproduce the original run
Real systems touched0%
- Passthrough is the default. A replay calls your real tool exactly like the original run did — useful when hitting the dependency is safe and part of the test.
- Static swaps in the recording. Match a tool to a static policy and its call is answered from the recorded or configured value instead — the real tool is never invoked.
- Set per replay, not per call. A Kitaru worker selects the replay via
KITARU_REPLAY_ID; there’s no per-run replay argument to thread through your own code.
A session you can inspect, not just a trace you can read
OpenAI’s tracing is genuinely useful — model calls, tool calls, handoffs, guardrails, and custom spans all land in the dashboard. What tracing doesn’t give you is something you can run again and score. Kitaru sessions do.
OpenAI traceWhat happened
model.requesttool.search_docsmodel.requesthandoffmodel.requestRead-only. Reproduce by re-running.
Kitaru sessionWhat you can replay
session 7c1b
model.requestrecordedtool.search_docsrecordedhandoffrecordedreplayscorecompare
- Every node, one session. Model calls, direct tool calls, hosted tools, and handoffs land as nodes on the session the adapter creates before OpenAI executes the agent.
- Replay, then evaluate. A replayed session gets scored by the same evaluators as the original — deterministic checks first, a model-graded check for what’s left.
- Compare, not just read. Freeze the sessions that matter into a cohort, run an experiment with one variable moved, and diff before against after.
What makes Kitaru unique
| Feature | Kitaru | OpenAI Agents SDK |
|---|---|---|
| First-class Agent abstraction with instructions, tools, handoffs, guardrails | Not supported | Yes |
| First-party tracing, evals, and hosted tools | Not supported | Yes |
| Streaming responses (run_streamed) | Not supported | Yes |
| Durable resume of a tool approval interruption | Not supported | Yes |
| Every run recorded as a replayable session (model calls, tools, hosted tools, handoffs) | Yes | Not supported |
| Tool calls answered from the recording during replay (static or passthrough policy) | Yes | Not supported |
| Cohorts: frozen session sets as regression suites | Yes | Not supported |
| Experiments: same cohort, one variable moved, compared side by side | Yes | Not supported |
| Import Langfuse / LangSmith / Braintrust / OTel traces | Yes | Not supported |
| Self-hosted server and workers (Apache 2.0) | Yes | Not supported |
| First-class adapter for the other | Yes | Not supported |
How the two surfaces map
| Concept | OpenAI Agents SDK | Kitaru |
|---|---|---|
| Layer | Agent harness (how the agent thinks and acts) | Replay-based evals (how you test what it did) |
| Core unit | Agent run | Session — recorded model, tool, and handoff nodes, replayable |
| Composition | Runner.run_sync(agent, input) | Same agent wrapped once by KitaruRunner |
| Tool calls in a replay | Hit real systems, or you mock them by hand | Passthrough (default) or static, answered from the recording |
| “Did my change help?” | Read the trace, compare by eye | Two runs over the same cohort, diffed |
| Streaming | run_streamed() with live events | Not supported by the current adapter |
| Tool approval interruptions | Native RunState interrupt and resume | Fails closed — not durably resumable yet |
| Where it runs | Whatever Python service you wrap the agent in | Self-hosted server; workers replay in your environment |
Code comparison
import uuid
from agents import Agent
from kitaru_openai_agents import KitaruRunner
reviewer = Agent(
name="reviewer",
instructions="Review the draft for compliance.",
model="gpt-5-nano",
)
runner = KitaruRunner(agent_id=uuid.UUID(AGENT_ID))
# Runs exactly as before — and lands as a session:
# every model call, tool call, and handoff.
result = runner.run_sync(reviewer, case)
print(result.final_output)
# Later: freeze the sessions that matter, test a change.
import kitaru
client = kitaru.KitaruClient()
cohort = client.cohorts.create("hard-cases", sessions=session_ids)
experiment = client.experiments.create(
"cheap-model",
model="gpt-5-mini",
)
before = experiment.run(cohort=cohort, version="v1")
after = experiment.run(cohort=cohort, version="pr-311")
client.compare(before, after)from agents import Agent, Runner
reviewer = Agent(
name="reviewer",
instructions="Review the draft for compliance.",
model="gpt-5-nano",
)
result = Runner.run_sync(reviewer, case)
print(result.final_output)
# One trace per run, in the OpenAI dashboard. The run that
# caught the bug is a transcript you can read — not a test
# you can run again with the model swapped.Put replay-based evals under your OpenAI agents
If your OpenAI agent still fits in a notebook or a short-lived script, the SDK on its own is the right answer. If it’s shipping — writing to real systems, generating traffic you wish you could test against — Kitaru makes those runs runnable: record them as sessions, replay them as evals, and gate regressions with the cohort that caught the failure.
uv add "kitaru[cli,worker]" kitaru-pydantic-ai