Pydantic AI and Kitaru solve different problems at different layers. Pydantic AI is an agent harness: it’s the best-in-class Pythonic library for writing type-safe agent logic — tool calling, structured outputs, dependency injection, streaming. Kitaru is a replay-based eval layer: it records every run as a session, and a session re-executes — unchanged for a faithful baseline, or forked with one thing changed.
They don’t compete. They compose. Kitaru ships a first-class Pydantic AI adapter — wrap the agent once with KitaruAgent and every run lands as a session: model requests, tool calls, token usage, cost.
Use Kitaru if you are
- Running Pydantic AI agents in production and can't test against real systems — replay answers tool calls from the recording
- Deciding a model or prompt swap and want a like-for-like comparison instead of a vibe check
- Turning the sessions that caught a failure into a regression suite that runs on every change
- Already tracing to Langfuse, LangSmith, or Braintrust and want those traces runnable, not just readable
- Self-hosting eval infrastructure so traces and credentials stay in your own systems
Use Pydantic AI if you are
- Writing agent logic in Python and want type-safe inputs, outputs, and tools
- Building short-lived or interactive agents, or using Pydantic AI's own durable execution integrations (Temporal, DBOS, Prefect, Restate) where they fit your stack
- Prototyping an agent before deciding whether it needs an eval loop around it
- Happy with your existing observability and only need better agent ergonomics
Pydantic AI gives you a great way to write an agent. Kitaru gives you a great way to prove a change to it.
Different questions
Pydantic AI is asking: how do I write a typed, ergonomic agent loop with first-class tools and structured outputs? Kitaru is asking: once this agent ships, which of its runs can I re-run? Did the model swap actually help — and what regressed?
In practice that means Pydantic AI keeps running your agent exactly as before. Kitaru sits beside it, recording each run — and re-running the ones that matter.
Compose, don’t replace
Kitaru doesn’t ask you to give up the Pydantic AI agent you already wrote. KitaruAgent is a transparent wrapper — run, run_sync, iter, tools, and output types behave exactly as on the wrapped agent.
The adapter records every model request and tool call as a node on the session, and the same wrapper executes replays: the recording answers tool calls, so nothing touches real systems. If the process dies mid-run, the partial session is exactly the evidence you want.
from pydantic_ai import Agent
from kitaru_pydantic_ai import KitaruAgent
agent = Agent("openai:gpt-5.4", name="reviewer",
system_prompt="You're a compliance reviewer.")
reviewer = KitaruAgent(agent, agent_id=AGENT_ID)
# Runs exactly as before — and lands as a session:
# every model request, tool call, tokens, cost.
result = reviewer.run_sync(case)What Kitaru adds on top
Pydantic AI is an agent harness — and its own docs cover durable execution through Temporal, DBOS, Prefect, and Restate integrations. Kitaru’s difference is the eval loop it builds on top of your recorded runs:
- Sessions, recorded. One wrapper, no rewrite. Every run lands as a session — model calls, tool calls, token usage, cost — or import the traces you already collect in Langfuse, LangSmith, Braintrust, or OTel.
- Replay. A session re-executes against your real code, with tool calls answered from the recording. Unchanged, it reproduces the original — the faithful baseline that makes a diff trustworthy. Then fork it: a different model, a new prompt, your working tree.
- Cohorts. The sessions you care about, frozen as a named, immutable set — so a run’s result keeps meaning what it meant.
- Experiments. Pure configuration: model, prompt, tool policy. Two runs over the same cohort, one variable moved, compared side by side.
- Evaluators. Deterministic checks first — a structured write diffs against production field by field. Keep a model-graded check only for the residue.
- Self-hosted, Apache 2.0. One FastAPI + Postgres server. Replays and evaluations execute on workers in your environment — traces and credentials don’t leave your systems.
What makes Kitaru unique
| Feature | Kitaru | Pydantic AI |
|---|---|---|
| Typed agent inputs, outputs, tools | Not supported | Yes |
| Structured model outputs | Not supported | Yes |
| Dependency injection for tools | Not supported | Yes |
| Every run recorded as a replayable session | Yes | Not supported |
| Replay with tool calls answered from the recording | Yes | Not supported |
| Cohorts: frozen session sets as regression suites | Yes | Not supported |
| Experiments: same cohort, one variable moved | Yes | Not supported |
| Import Langfuse / LangSmith / Braintrust / OTel traces | Yes | Not supported |
| Self-hosted server and workers (Apache 2.0) | Yes | Not supported |
| First-class adapter for the other | Yes | Not supported |
How the two surfaces map
| Concept | Pydantic AI | Kitaru |
|---|---|---|
| Layer | Agent harness (how the agent thinks) | Replay-based evals (how you test what it did) |
| Core unit | Agent run | Session — recorded model and tool calls, re-executable |
| Composition | Standalone Pythonic agent | Same agent wrapped once by KitaruAgent |
| Tool calls in tests | Hit real systems, or you mock them by hand | Answered from the recording per tool policy |
| “Did my change help?” | Read the traces, compare by eye | Two runs over the same cohort, diffed |
| Crash mid-run | Run is gone | Partial session recorded — the evidence survives |
| Where it runs | Whatever Python service you wrap the agent in | Self-hosted server; workers replay in your environment |
Code comparison
from pydantic_ai import Agent
from kitaru_pydantic_ai import KitaruAgent
reviewer = KitaruAgent(
Agent("openai:gpt-5.4", system_prompt="You're a compliance reviewer."),
agent_id=AGENT_ID,
)
# Production runs — each one lands as a session.
result = reviewer.run_sync(case)
# Later: freeze the sessions that matter, test a change.
import kitaru
client = kitaru.KitaruClient()
cohort = client.cohorts.create("hard-cases", sessions=session_ids)
experiment = client.experiments.create(
"cheap-model",
model="gpt-5-mini",
tool_policy=History(scope="cohort", on_miss="fail"),
)
before = experiment.run(cohort=cohort, version="v1")
after = experiment.run(cohort=cohort, version="pr-311")
client.compare(before, after)from pydantic_ai import Agent
reviewer = Agent(
"openai:gpt-5.4",
system_prompt="You're a compliance reviewer.",
)
result = reviewer.run_sync(case)
# One log line per run. The run that caught the bug
# is a transcript you can read — not a test you can
# run again with the model swapped.Put replay-based evals under your Pydantic AI agents
If the Pydantic AI agent you wrote is still a notebook script or a short-lived interactive tool, Pydantic AI on its own is the right answer. If it’s shipping — writing to real systems, generating traffic you wish you could test against — Kitaru makes those runs runnable: record them as sessions, replay them as evals, and gate regressions with the cohort that caught the failure.
uv add "kitaru[cli,worker]" kitaru-pydantic-ai