Kitaru by ZenML · Replay-based evals for AI agents
Your agent's best eval data is already in production
Turn real traces into replayable evals and test changes against what your agent has actually seen.
14-day free trial · Full access · No credit card
Watch the launch video — 90 seconds
You already suspect what's wrong.
Now you can check.
Every trace you import becomes a session, a replayable record of one real agent run. Take the ones that went sideways, change the one thing you suspect, and run them again. Two runs, side by side, and you know.
What does good even mean here?
Read twenty of them and write what you notice. Your notes become the cohorts.
Can I believe my own check?
Apply it blind to sessions you never read, then reveal the labels.
Can we ship the cheaper model?
Same cohort, one model swapped — the answer is two runs, compared.
Will it stay fixed?
The same experiment runs on every commit, on the cases that caught it.
Start where you are
Already have traces? Good. Starting fresh? Also good.
Import what your current tools already collected, or wrap your agent in one line and record new runs. Both roads end at the same thing: sessions you can replay.
Have traces? Import them.
$ kitaru session import langfuse-export.jsonl \
--importer kitaru/langfuse@latest \
--agent support-agent@latest \
--tag imported-baseline \
--media-type application/x-ndjson \
--wait
reading langfuse-export.jsonl … 452 traces
converting to sessions … done412 sessions created · 38 skipped · 2 failed
replay readiness ready 361 · partial 44 · unavailable 7
Have an agent? Wrap it.
from kitaru_pydantic_ai import KitaruAgent
from pydantic_ai import Agent
agent = KitaruAgent(Agent(
"openai:gpt-5-mini",
system_prompt="You are a compliance reviewer.",
tools=[search_docs, fetch_policy],
), agent_id=AGENT_ID)
result = await agent.run(task)Want the cheaper model?
Here's how you find out if it's safe.
Freeze a set of real production sessions. Replay them with the model you have and the model you want. Then see exactly which cases changed, and whether those changes are acceptable.
Pick the production cases that matter
Start with a cohort you found during investigation, or hand-pick your own. Versioning the set means you compare every change against the same cases.
Change one thing
Keep the agent setup fixed and swap the model. Now any difference in the replay comes from the change you are testing.
See what actually changed
Replay the same cohort with both setups and compare cost, latency, evaluator results, and behavior side by side.
Out of 200 real sessions, 196 came back unchanged. Now you only have four differences to review before deciding whether to ship.
Illustrative numbers from a sample run.
Frequently asked
questions
Everything you need to know about Kitaru.
How is this different from Langfuse, Braintrust or LangSmith?
They tell you what happened: traces you read, dashboards you check. Kitaru re-runs what happened. Your agent's real code executes again against the recorded world, so you can test your next change against your last thousand sessions before it ships. Kitaru also imports trace files exported from Langfuse, LangSmith, Braintrust, Logfire or Arize Phoenix, so your observability tool stays your system of record.
So is this an observability tool?
No. It sits beside your observability stack. Traces tell you what happened; Kitaru re-runs them against your actual code: a debugger with a memory rather than another dashboard of spans.
Do I have to change my agent's code?
Not to get started. Import your traces and you already get the session views, investigations, cohorts and evaluators; your code stays untouched. An adapter enters only when you want to replay sessions against a change: one line for the supported frameworks, or a small custom one for CLI-harness agents like Claude Code or Gemini CLI.
My agent writes to real systems. Isn't replay dangerous?
Replay answers the agent's tool calls from the recording, so nothing touches real systems. Per-tool policies control the rest: answer from history and stop the run when a call has no recorded answer, pin a static result, or deliberately pass a specific tool through live. We don't test in prod. We make prod's past your test bench.
The model isn't deterministic. How is replay trustworthy?
The recorded world is held constant, same inputs and same tool responses, so replay removes every source of variation except the model itself. For the variation that remains, you create multiple experiment runs over the same cohort and compare the distributions, so you can tell a real regression from run-to-run noise instead of judging from a single sample.
Where do the eval criteria come from? We never wrote any down.
From the people who already judge the agent every day. Your coding assistant, using Kitaru's investigation skill, interviews you over real sessions, pins your judgments to the exact evidence in the trace, and drafts evaluators from them. Each evaluator is checked against your verdicts before it gates anything.
What frameworks does it work with?
Recording adapters wrap your existing agent in one line, with no rewrite. Python: PydanticAI, the OpenAI Agents SDK, and LangGraph (including LangChain agents and Deep Agents). TypeScript: the Vercel AI SDK and Mastra. For a custom harness or a framework we don't support yet, Kitaru ships a skill that walks your coding assistant through generating a new adapter for it.
Can I use traces I already have?
Yes. Importers bring in trace files exported from Langfuse, LangSmith, Braintrust, Pydantic Logfire or Arize Phoenix, or you can write a one-page custom importer for your own store. The Phoenix importer reads JSONL downloaded from the Phoenix UI or JSON retrieved with the Phoenix CLI; it does not connect to the Phoenix API or use Phoenix credentials. Traces in raw OpenTelemetry format convert to Kitaru's JSONL import format.
My agent is TypeScript. Can I use Kitaru?
Yes, natively. TypeScript agents record and replay through the Vercel AI SDK and Mastra adapters, with a framework-neutral TypeScript SDK alongside — and the adapter-generation skill covers custom TypeScript harnesses too. The CLI, workers and evaluators run on Python today, so there's Python in the loop even when the agent itself is TypeScript.
Is it open source? Can I self-host?
Yes: Apache 2.0, self-hosted by default. The server and workers run in your infrastructure and replay executes in your environment, so your traces never have to leave your systems. ZenML Pro offers a managed version if you want one.
Does this replace human review?
No. Evals change how much humans review, not whether they do. The goal is that people spend their review time on the sessions that deserve it, with evidence attached.
Who is this for, and who isn't it for?
Teams shipping agents to customers whose regression process is honestly a few samples and a vibe check. Kitaru installs the rigor loop. It fits badly for single-dev prototypes and for teams buying a fully managed agent platform: if you're buying an agent platform, Kitaru will feel low-level. If you're building one, that's the point.
Something's broken. How do I reach you?
Three routes, all reaching a human: the Slack community, kitaru.ai/help (goes straight to GitHub issues), or support@kitaru.ai.
What arrives as a complaint
leaves as a regression test.
uv add "kitaru[cli,worker]" kitaru-pydantic-ai14-day free trial · Full access · No credit card · Open source (Apache 2.0)
Replays run from recordings — production never sees a test.