The Claude Agent SDK and Kitaru solve different problems. The Agent SDK is the agent harness: it builds agents that read files, run commands, call tools, and solve whatever task you hand to query(). Kitaru is a replay-based eval layer. Every run lands as a session, and a session re-executes. Unchanged, you get a faithful baseline. Forked with one thing changed, you get a comparison.
They don’t compete. They compose. Kitaru ships a first-class Claude Agent SDK adapter: KitaruClaudeRunner wraps the SDK’s one-shot query(), yields the same message objects your code already consumes, and records the prompt, model calls, tool calls, subagents, usage, cost, and failures as one session. Running that same agent durably in production is ZenML’s job; Kitaru replays and improves it. See ZenML vs Claude Agent SDK.
Kitaru
Use Kitaru if you are
- Running Agent SDK queries in production and want the run that failed on your desk, re-runnable without touching production
- Deciding a model or prompt swap and want two runs over the same cohort compared, not a vibe check
- Turning the sessions that caught a failure into a cohort that gates the next change
- Declaring your in-process SDK MCP tools once so replay answers them from the recording
- Self-hosting eval infrastructure so prompts, tool results, and credentials stay in your own systems
Claude Agent SDK
Use the Claude Agent SDK if you are
- Building autonomous coding agents that edit files, run bash, and search codebases
- Running interactive sessions with ClaudeSDKClient, or resuming, continuing, and forking provider sessions: the Kitaru adapter records one-shot string query() calls only
- Needing Claude's built-in tools or external MCP servers substituted in a test: Kitaru records them but can only pass them through
- Prototyping one-off tasks where you don't need a recorded session yet
The Claude Agent SDK runs the query. Kitaru turns each query into something you can run again.
Wrap query(), keep the messages
Install it next to the SDK with uv add kitaru-claude-agent-sdk. The adapter calls the SDK’s public query() and yields every message unchanged. It copies your ClaudeAgentOptions before adding its recording hooks, so the options and hook collections you passed in never change under you.
Register the agent once with kitaru agent register and hand its id to the runner. Under a Kitaru worker, KITARU_TASK_ID supplies the identity instead, so the same script records on your laptop, in production, and inside a replay without an edit. Wrap the stream in contextlib.aclosing() and a consumer that bails early still finalizes the partial session.
import contextlib, os, uuid
from claude_agent_sdk import ClaudeAgentOptions
from kitaru_claude_agent_sdk import KitaruClaudeRunner
runner = KitaruClaudeRunner(
agent_id=uuid.UUID(os.environ["KITARU_AGENT_ID"]),
)
stream = runner.query(
prompt="Investigate ticket 4821.",
options=ClaudeAgentOptions(model="claude-sonnet-4-5"),
)
# Same Claude messages as before. The run lands as a session.
async with contextlib.aclosing(stream) as messages:
async for message in messages:
print(message)Declare the tools replay is allowed to answer
The SDK’s in-process SdkMcpTool handler is the one tool boundary Kitaru can stand in for. Declare those tools through replayable_sdk_mcp_server() instead of building the server yourself, and pass the definition to every query that might use them. The policy identity is mcp__<server>__<tool>. The adapter builds a fresh SDK MCP server per query, so no handler or history state leaks between runs.
During a replay, history answers a wrapped tool with the recorded result that matches its identity and canonical arguments, static returns a canned case, error_result hands Claude a tool error on a miss, and passthrough calls the real handler. Only wrapped SDK MCP tools get substituted. Claude’s built-in tools and external MCP servers are recorded when the public messages expose them, but in a replay they can only pass through.
from claude_agent_sdk import ClaudeAgentOptions, SdkMcpTool
from kitaru_claude_agent_sdk import replayable_sdk_mcp_server
support_server = replayable_sdk_mcp_server(
name="support",
tools=[SdkMcpTool(name="lookup", description="Look up a record.",
input_schema=LOOKUP_SCHEMA, handler=lookup)],
)
stream = runner.query(
prompt="Look up ticket 4821.",
options=ClaudeAgentOptions(
tools=[], mcp_servers={},
allowed_tools=["mcp__support__lookup"],
),
replayable_servers=[support_server],
)Fail closed, or don't substitute at all
Substitution only fails closed when the adapter can prove every enabled tool is one it wrapped. So a replay with static or history substitution requires ClaudeAgentOptions(tools=[]). The adapter injects the wrapped tools itself, and before calling Claude it sets setting_sources=[] and strict_mcp_config=True on its private copy, so user settings, project settings, and .mcp.json can’t sneak in an unwrapped server.
A history miss under on_miss="fail" raises ToolPolicyMissError and marks the session failed. That’s the guarantee: nothing unrecorded reaches a live system. Passthrough is a live call, so a database write or an outbound message happens again. Put side-effecting tools behind a substituting policy, or run the replay in a sandbox you can throw away.
# Substituting replay: prove every enabled tool is wrapped.
options = ClaudeAgentOptions(tools=[], mcp_servers={},
allowed_tools=["mcp__support__lookup"])
# Rejected before Claude runs, because each can add an
# unwrapped tool: pre-existing mcp_servers, unwrapped
# allowed_tools, settings sources, plugins, skills,
# agent definitions, extra CLI args.
# Recording-only runs and all-passthrough replays keep
# your tool configuration exactly as it is.What Kitaru adds on top
A replay starts a fresh Claude query from the recorded root input. It doesn’t send the old assistant messages back or resume the provider session, so Claude is free to take a different path, and every model call is live. That’s the point: you see what a different model or prompt actually does.
# One change, the whole cohort, both sides evaluated.
kitaru cohort create hard-cases --agent support \
--tag needs-review --display-version week-32
kitaru experiment create cheaper-model --agent support \
--evaluator ticket-resolved@latest \
--override '{"model": {"claude-sonnet-4-5": "claude-haiku-4-5"}}' \
--tool-policy '{"default": {"type": "history",
"scope": "cohort_version", "on_miss": "fail"}}'
kitaru experiment run start cheaper-model \
--cohort-version <id> --agent support@2 \
--evaluate-baselines --wait- Sessions, recorded. One runner, no rewrite. Prompt, model calls, wrapped and built-in tool calls, subagents, usage, cost, and failures land as nodes on one session.
- Replay. Re-run from the top with the recorded prompt, or fork it: swap the prompt, the system prompt, or the model, either directly or mapped from the current one. A mapped override has to name the model set in your options, or the adapter stops before it calls Claude.
- Cohorts. The sessions you care about, frozen as a named, immutable version, so a run’s result keeps meaning what it meant.
- Experiments. The change as configuration: override, tool policy, evaluators. One run per cohort version and agent version, evaluated on both sides.
- Evaluators. Versioned Python callables over the session. Deterministic checks first, a model-graded check only for what’s left. Replay proves the new run completed. The evaluator says whether it helped.
- Self-hosted, Apache 2.0. One FastAPI + Postgres server. Workers run replays and evaluations in your environment, next to your Anthropic credentials.
What makes Kitaru unique
| Feature | Kitaru | Claude Agent SDK | What that means |
|---|---|---|---|
| Interactive coding agent toolkit (Read, Edit, Bash, WebSearch) | Not supported | Yes | Claude's built-in tools are best-in-class. Kitaru records them; it doesn't replace them. |
| Interactive sessions, resume, continue, and fork (ClaudeSDKClient) | Not supported | Yes | The adapter records one-shot string query() calls only. It rejects resume, continue, and fork options during replay. |
| Every query recorded as a replayable session | Yes | Not supported | KitaruClaudeRunner yields the same messages and writes the session beside them. |
| Replay with prompt, system prompt, or model replaced | Yes | Not supported | A fresh query from the recorded root input, every model call live. No model_params override in this release. |
| Tool calls answered from the recording | Partial support | Not supported | Static and history substitution for SDK MCP tools declared through replayable_sdk_mcp_server(). Built-in tools and external MCP servers are passthrough only. |
| Fail-closed isolation for substituting replays | Yes | Not supported | Requires tools=[]; unwrapped tools, settings sources, plugins, skills, and extra CLI args are rejected before Claude runs. |
| Cohorts: frozen session sets as regression suites | Yes | Not supported | |
| Experiments: same cohort, one variable moved, evaluated on both sides | Yes | Not supported | |
| Self-hosted server and workers (Apache 2.0) | Yes | Not supported |
How the two surfaces map
| Concept | Claude Agent SDK | Kitaru |
|---|---|---|
| Layer | Agent harness (how the agent thinks and acts) | Replay-based evals (how you test what it did) |
| Core unit | One query() and its message stream | Session: recorded model, tool, and subagent nodes, replayable |
| Composition | query(prompt=..., options=...) | Same call through KitaruClaudeRunner(agent_id=...).query(...) |
| Tool boundary | SdkMcpTool handlers, built-in tools, external MCP | Wrapped SDK MCP tools substituted per policy; everything else passthrough |
| Replay | Not a concept; a session is history on disk | Fresh query from the recorded root input; prompt, system prompt, or model replaced |
| Sessions, resume, fork | ClaudeSDKClient, resume, fork_session | Not supported by the adapter; one-shot string prompts only |
| “Did my change help?” | Read the transcript, compare by eye | Two runs over the same cohort, evaluated on both sides |
| Where it runs | Your process | Self-hosted server; workers replay in your environment |
Code comparison
import contextlib, os, uuid
from claude_agent_sdk import ClaudeAgentOptions
from kitaru_claude_agent_sdk import KitaruClaudeRunner
runner = KitaruClaudeRunner(
agent_id=uuid.UUID(os.environ["KITARU_AGENT_ID"]),
)
async def review(topic: str) -> None:
stream = runner.query(
prompt=f"Draft a short blog post on: {topic}",
options=ClaudeAgentOptions(
model="claude-sonnet-4-5",
tools=[], mcp_servers={},
allowed_tools=["mcp__support__lookup"],
),
replayable_servers=[support_server],
)
# Same messages as before. The run lands as a session.
async with contextlib.aclosing(stream) as messages:
async for message in messages:
print(message)
# Later: freeze the sessions that matter, test a change.
# kitaru cohort create hard-cases --agent support --tag needs-review
# kitaru experiment create cheaper-model --agent support \
# --evaluator ticket-resolved@latest \
# --override '{"model": {"claude-sonnet-4-5": "claude-haiku-4-5"}}' \
# --tool-policy '{"default": {"type": "history",
# "scope": "cohort_version", "on_miss": "fail"}}'
# kitaru experiment run start cheaper-model \
# --cohort-version <id> --agent support@2 --evaluate-baselines --waitimport asyncio
from claude_agent_sdk import query, ClaudeAgentOptions, ResultMessage
async def review(topic: str) -> str:
result_text = ""
async for msg in query(
prompt=f"Draft a short blog post on: {topic}",
options=ClaudeAgentOptions(allowed_tools=["Read"]),
):
if isinstance(msg, ResultMessage):
result_text = msg.result or ""
return result_text
asyncio.run(review("Replay-based evals"))
# One session history on disk. Useful for this run,
# not something you can run again with the model swapped.Put replay-based evals beside
your Agent SDK
If you’re driving the Agent SDK interactively, or your entrypoint is a ClaudeSDKClient conversation, stick with what you have. If you’re taking a one-shot query() build to production and wish you could test against its runs, wrap it with KitaruClaudeRunner: record the sessions, replay them as evals, and gate regressions with the cohort that caught the failure.