Pydantic AI and Kitaru solve different problems at different layers. Pydantic AI is an agent harness: the best-in-class Pythonic library for writing type-safe agent logic, with tool calling, structured outputs, dependency injection, and streaming. Kitaru is a replay-based eval layer: it records every run as a session, and a session re-executes. Unchanged, you get a faithful baseline. Forked with one thing changed, you get an answer.
They don’t compete. They compose. Kitaru ships a first-class Pydantic AI adapter: wrap the agent once with KitaruAgent and every run lands as a session with its model requests, tool calls, token usage, and cost. Running that same agent durably in production is ZenML’s job; Kitaru replays and improves it. See ZenML vs Pydantic AI.
Use Kitaru if you are
- Running Pydantic AI agents in production and can't test against real systems: a history tool policy answers tool calls from the recording
- Deciding a model or prompt swap and want a like-for-like comparison instead of a vibe check
- Turning the sessions that caught a failure into a regression suite that runs on every change
- Already tracing to Langfuse, LangSmith, Braintrust, Logfire, or Arize Phoenix and want those traces runnable, not just readable
- Self-hosting eval infrastructure so traces and credentials stay in your own systems
Use Pydantic AI if you are
- Writing agent logic in Python and want type-safe inputs, outputs, and tools
- Building short-lived or interactive agents, or using Pydantic AI's own Temporal, DBOS, Prefect, and Restate integrations where they fit
- Prototyping an agent before deciding whether it needs an eval loop around it
- Happy with your existing observability and only need better agent ergonomics
Pydantic AI gives you a great way to write an agent. Kitaru gives you a great way to prove a change to it.
Different questions
Pydantic AI is asking: how do I write a typed, ergonomic agent loop with first-class tools and structured outputs? Kitaru is asking: once this agent ships, which of its runs can I re-run? Did the model swap actually help, and what regressed?
In practice that means Pydantic AI keeps running your agent exactly as before. Kitaru sits beside it, recording each run and re-running the ones that matter.
Compose, don’t replace
Kitaru doesn’t ask you to give up the Pydantic AI agent you already wrote. Install it with uv add kitaru-pydantic-ai. KitaruAgent is a transparent WrapperAgent: run, run_sync, iter, tools, output types, and capabilities behave exactly as on the wrapped agent.
The adapter records every model request and tool call as a node on the session, and the same wrapper runs replays: model swaps and model_params apply at the model-request boundary, and tool calls get answered per policy. Under a history policy with on_miss="fail", anything unrecorded raises ToolPolicyMissError instead of touching a live system. If the process dies mid-run, the partial session is exactly the evidence you want.
from pydantic_ai import Agent
from kitaru_pydantic_ai import KitaruAgent
agent = Agent("openai:gpt-5.4", name="reviewer",
system_prompt="You're a compliance reviewer.")
reviewer = KitaruAgent(agent, agent_id=AGENT_ID)
# Runs exactly as before, and lands as a session:
# every model request, tool call, tokens, cost.
result = reviewer.run_sync(case)What Kitaru adds on top
Pydantic AI is an agent harness, and its own docs cover the Temporal, DBOS, Prefect, and Restate integrations for running it. What Kitaru adds is the eval loop on top of your recorded runs:
- Sessions, recorded. One wrapper, no rewrite. Every run lands as a session: model calls, tool calls, token usage, cost. Or import the traces you already collect in Langfuse, LangSmith, Braintrust, Logfire, or Arize Phoenix.
- Replay. A session re-executes against your real code, with tool calls answered from the recording. Unchanged, it reproduces the original, the faithful baseline that makes a diff trustworthy. Then fork it: a different model, a new prompt, your working tree registered as a new agent version.
- Cohorts. The sessions you care about, frozen as a named, immutable version, so a run’s result keeps meaning what it meant.
- Experiments. The change as configuration: override, tool policy, evaluators. Runs over the same cohort version, evaluated on both sides.
- Evaluators. Deterministic checks first: plain Python over the recorded nodes, versioned like the agent. Keep a model-graded check only for the residue.
- Self-hosted, Apache 2.0. One FastAPI + Postgres server. Replays and evaluations execute on workers in your environment, so traces and credentials don’t leave your systems.
What makes Kitaru unique
| Feature | Kitaru | Pydantic AI | What that means |
|---|---|---|---|
| Typed agent inputs, outputs, tools | Not supported | Yes | |
| Structured model outputs | Not supported | Yes | |
| Dependency injection for tools | Not supported | Yes | |
| Every run recorded as a replayable session | Yes | Not supported | |
| Replay with tool calls answered from the recording (history, static, passthrough) | Yes | Not supported | The API accepts an llm tool policy; this adapter doesn't support it. |
| Cohorts: frozen session sets as regression suites | Yes | Not supported | |
| Experiments: same cohort, one variable moved | Yes | Not supported | |
| Import Langfuse / LangSmith / Braintrust / Logfire / Phoenix traces | Yes | Not supported | |
| Self-hosted server and workers (Apache 2.0) | Yes | Not supported | |
| First-class adapter for the other | Yes | Not supported |
How the two surfaces map
| Concept | Pydantic AI | Kitaru |
|---|---|---|
| Layer | Agent harness (how the agent thinks) | Replay-based evals (how you test what it did) |
| Core unit | Agent run | Session: recorded model and tool calls, re-executable |
| Composition | Standalone Pythonic agent | Same agent wrapped once by KitaruAgent |
| Tool calls in tests | Hit real systems, or you mock them by hand | Answered from the recording per tool policy |
| “Did my change help?” | Read the traces, compare by eye | Two runs over the same cohort, evaluated on both sides |
| Crash mid-run | Run is gone | Partial session recorded, so the evidence survives |
| Where it runs | Whatever Python service you wrap the agent in | Self-hosted server; workers replay in your environment |
Code comparison
from pydantic_ai import Agent
from kitaru_pydantic_ai import KitaruAgent
reviewer = KitaruAgent(
Agent("openai:gpt-5.4", system_prompt="You're a compliance reviewer."),
agent_id=AGENT_ID,
)
# Production runs: each one lands as a session.
result = reviewer.run_sync(case)
# Later: freeze the sessions that matter, test a change.
# kitaru cohort create hard-cases --agent reviewer --tag needs-review
# kitaru experiment create cheaper-model --agent reviewer \
# --evaluator compliance-check@latest \
# --override '{"model": {"openai:gpt-5.4": "openai:gpt-5-mini"}}' \
# --tool-policy '{"default": {"type": "history",
# "scope": "cohort_version", "on_miss": "fail"}}'
# kitaru experiment run start cheaper-model \
# --cohort-version <id> --agent reviewer@2 --evaluate-baselines --waitfrom pydantic_ai import Agent
reviewer = Agent(
"openai:gpt-5.4",
system_prompt="You're a compliance reviewer.",
)
result = reviewer.run_sync(case)
# One log line per run. The run that caught the bug
# is a transcript you can read, not a test you can
# run again with the model swapped.Put replay-based evals under your Pydantic AI agents
If the Pydantic AI agent you wrote is still a notebook script or a short-lived interactive tool, Pydantic AI on its own is the right answer. If it’s shipping, writing to real systems and generating traffic you wish you could test against, Kitaru makes those runs runnable: record them as sessions, replay them as evals, and gate regressions with the cohort that caught the failure.
curl -fsSL https://kitaru.ai/install | bash