Kitaru18 mins

LangSmith vs Braintrust vs Kitaru: Agent Tracing, Evals, and Replay Compared

LangSmith vs Braintrust vs Kitaru compared on agent trace capture, test sets from production runs, human review, experiment comparison, replay, and pricing.

Kitaru comparison card for LangSmith vs Braintrust vs Kitaru for Agent Tracing, Evals, and Replay, showing the logos of Kitaru, LangSmith and Braintrust
On this page

It’s Friday afternoon, and your support agent somehow approved a refund that should have escalated.

You still found the bad tool call, fixed the prompt, and got a better answer in testing. Yet you’re unsure whether that fix will hold up across the tens of thousands of requests your agent has already handled.

Tracing and evaluating agents?
Keep your AI getting better. An always-on AI engineering team finds what’s broken and proves the fix works.
Start free
Tracing and evaluating agents?
Keep your AI getting better. An always-on AI engineering team finds what’s broken and proves the fix works.
Start free
From $39 a month · 14 days free
  • AI engineers on your agent
  • Set up in minutes
  • No credit card
Tracing and evaluating agents?
Keep your AI getting better.
Start free

LangSmith and Braintrust use a similar approach, and both help you inspect production runs, collect test cases, and compare executions. Kitaru works on top of both these platforms. It imports your existing traces and re-runs your agent’s real code against them, with one thing changed.

Nevertheless, all three help you inspect the root cause. But because the mechanisms differ, each tool can and cannot tell you different things.

In this LangSmith vs Braintrust vs Kitaru comparison, we’ll cover tracing, production-derived test sets, human review, and regression testing. Then we’ll compare pricing and help you decide where each tool fits.

LangSmith vs Braintrust vs Kitaru: Key Takeaways

  • LangSmith: An observability and evaluation platform with detailed trace views, dataset automation, and structured human-review queues. A strong choice for LangChain and LangGraph teams that want to investigate behavior and compare experiments.
  • Braintrust: A tracing and evaluation platform with the best trace viewer of the three. Human review comes with assignment and a Kanban board, and experiment comparison comes with pinned baselines and a CI action that comments on pull requests, useful when evaluation is already part of your release process.
  • Kitaru: An open-source agent replay and evaluation tool for ML engineers and AI developers. It imports traces from LangSmith and Braintrust, groups production sessions into immutable cohorts, and replays agent code with controlled changes.

What Problems Are These Tools Actually Solving?

All three can help you catch agent failures. They start from different questions, though, and that changes what you’ll do with them every day.

LangSmith

LangSmith Observability landing page with the headline Know what your agents are really doing, next to a dashboard of agent trace counts, hallucination scores and feedback charts

LangSmith began as the tracing layer for LangChain and LangGraph and is framework-agnostic today. Each trace is a collection of runs for a single operation. Around tracing sit datasets, experiments, and evaluators, plus annotation queues, online evaluators, and a hosted deployment runtime.

For testing, LangSmith centers on datasets of inputs and optional reference outputs. Your target function runs against those examples, and LangSmith stores the results as experiments. You can also score production traces with online evaluators or send selected runs into annotation queues.

Braintrust

Braintrust landing page with the headline See what happens in production, next to a conversation trace showing tool calls such as get_order_status and a conversation quality scorer

Braintrust brings production tracing, datasets, scorers, and experiments into one platform. Its Eval() function runs a task against a dataset, scores the outputs, and saves the results as an experiment you can compare with previous runs. Unlike LangSmith, Braintrust also lets you reference a logged trace directly in a dataset instead of copying its input.

Kitaru

Kitaru landing page with the headline Insights from your agent's traces, a one-line install command and a terminal showing trace import, analyzers, evaluators and replay commands

In Kitaru, your existing LangSmith or Braintrust project can remain the primary trace store. Freeze selected sessions in a cohort version and attach evaluators to the experiment. You can override the model, prompt, or model parameters. To test a code change, select a different registered agent version for the run.

Comparing LangSmith, Braintrust, and Kitaru Features

The table summarizes how the three tools handle your use cases. The sections after it go deep into each one.

FeatureKitaruLangSmithBraintrust
Agent trace capture and inspectionRecords sessions or imports captured trace data. Includes a session inspector and insight cards from selected analyzers.Trajectory and Details views; inputs, outputs, timing, tokens, errors, and metadata per runSpans, thread, and timeline views; Messages, Details, Review, and Raw tabs; in-trace search
Test sets built from production runsCohorts of whole sessions; immutable versions preserve the exact test populationRuns become examples manually, via automation rules, or from annotation queues; automatic versioning and splitsDatasets from logs, feedback, uploads, or Loop; rows copy a span or reference a trace; experiments pin to versions
Human review and annotationInvestigations: per-session questions, answers pinned to evidence, verdicts that calibrate evaluatorsAnnotation queues with rubrics, reservations, multi-reviewer thresholds, and a pairwise modeCategorical, continuous, and free-form review scores; assignment, kanban, blind review; corrections to expected
Experiment comparison and regression detectionReplays a cohort with a tool policy and can score baseline and replay sessions with the same evaluators. In CI, use --wait to catch failed runs and check evaluator results separately to enforce quality thresholds.Compare view over two or more experiments; red/green per row against a source experimentPinned or git-branch baselines; score delta column; sort by regressions; GitHub action posts to the PR

Feature 1. Agent Trace Capture and Inspection

Everything downstream depends on what the trace holds and whether anyone reads it.

Kitaru

Animated walkthrough of a Kitaru session: an imported returns-agent session shown as a tree of LLM and tool calls, then a click on the get_return_policy node to read its exact input, output and metadata

Kitaru records instrumented model calls, tool calls, and other captured events in an ordered session. Imported sessions contain the data captured by the source trace. Its LangSmith, Braintrust, or any other importer you want to use maps existing trace data into that structure; for example, the LangSmith importer maps a thread to one session and llm and tool runs to llm_call and tool_call nodes.

You can inspect a session or compare it with a replay in the dashboard. For larger imports, post-import analyzers read a whole import at once and store insight cards with supporting session references and investigation prompts.

Those analyzers must be selected when you import, and Kitaru doesn’t replace the monitoring charts, alerts, and broader trace exploration offered by observability platforms.

LangSmith

LangSmith trace view with a waterfall of an assistant run, a get_context step and a ChatOpenAI call, and the selected run's JSON input question and output answer in the Details pane

Source

LangSmith’s trace view separates the conversation and the execution layer. For instrumented threads, LangSmith offers Trajectory, Turns, and Details views. Turns requires thread_id metadata, and Trajectory requires renderable messages.

Trajectory is the conversation layer. It shows the conversation with model responses, tool calls, and results. Details lets you inspect individual runs, including inputs, outputs, tokens, timing, and errors.

The process goes like this: you scan the trajectory, click the tool call that looks wrong, and land in Details at the run that produced it.

Say a research agent cites the wrong document; you can trace the issue to the retrieval call and check the returned results in Details.

You can also group related interactions into threads using a thread_id. Without it, LangSmith displays individual traces with less conversation-level context.

Braintrust

Braintrust trace view with the Spans tab listing nested pipeline spans, cost, tokens and latency, and the Messages tab of the root span open on the right

Braintrust’s tracing captures inputs and outputs, tool calls, timing, token usage, costs, and errors. It provides three views: spans, thread, and timeline views.

The spans view shows the execution tree, with tabs to inspect messages, run details, and raw JSON. The timeline shows execution order, while the thread view groups related interactions and displays raw span data by default.

These views account for improved readability and make individual traces easier to debug. However, the limit is who uses it and how. With three layouts and a raw tab, a human still reads one trace at a time, while reviewing thousands of traces still takes considerable manual effort.

Bottom line: Braintrust and LangSmith win on inspection, Braintrust for breadth and LangSmith for the trajectory-to-details workflow.

Feature 2. Build Test Sets From Production Runs

It’s important to know how each tool turns a production run into something you can run again next week, and whether it stays fixed.

Kitaru

Animated walkthrough of creating a Kitaru cohort: three sessions selected into a returns-regression cohort, then the cohort's immutable v1 version listing those sessions

Source

Kitaru takes a different approach with cohorts. Each cohort version is a named set of sessions belonging to one agent, and it is the unit an experiment replays. You can combine known failures, successful counterexamples, and common production paths in one test population.

Suppose a new policy prompt fixes an incorrect refund. Include that failed request, plus legitimate refunds the agent must still approve. The cohort stays unchanged between experiments, so you can compare the same cases after each revision.

Two limits apply. No trace export contains the code, so replay needs a registered run command for the agent version that produced the runs. And no automation rule feeds a cohort; new versions are explicit.

LangSmith

LangSmith Select generated examples screen listing dataset examples with context and question inputs on the left and reference answer outputs on the right

Source

In LangSmith, a dataset contains examples with inputs, optional reference outputs, and metadata. You can select production traces and add them to a dataset, or use automation rules to collect runs that match conditions such as errors or low feedback scores.

Versioning is automatic. LangSmith versions datasets as examples change. However, the example does not carry the run as an executable unit. When you run an experiment, your target function executes again on the stored input. You’ll need your own mocks or fixtures if you want to reproduce earlier tool responses.

Braintrust

Braintrust Logs table of production conversation traces with topic tags, next to the Loop assistant panel asked to create a dataset from a popular use case

Source

Braintrust builds datasets from production logs, user feedback, uploads, or directly from Loop, Braintrust’s built-in agent.

A log becomes a row in one of two ways.

  • “Add this span to dataset” copies the span, and the span’s output typically becomes the row’s expected value.
  • The row’s input holds the pointer to the logged trace. Copying span data is a separate action. That is the closest either trace tool gets to Kitaru’s whole-session unit.

The catch is, a row that references a trace depends on that trace remaining available. Starter and Pro retention is 14 and 30 days.

Bottom line: Kitaru wins. A cohort holds whole sessions and a version never changes, so the cases you test after a fix are the exact cases that failed before it. A LangSmith example keeps the input and re-executes the rest, and a Braintrust trace reference lasts only as long as the retention window. Neither makes the whole run the unit you test again, so a repeat is a fresh execution rather than a replay of what happened.

Feature 3. Human Review and Annotation

An evaluator can’t define your refund policy. Your domain expert still has to judge what counts as acceptable behavior. The question is how each tool gets your domain expert’s opinion onto a run, and what it does with it afterward.

Kitaru

Animated walkthrough of a Kitaru review: a note pinned to the get_return_policy tool call saying an empty policy lookup should block the refund, then a session verdict chosen from acceptable, problematic or uncertain

Kitaru organizes human judgment through investigations. A coding assistant can select sessions, prepare focused questions, and point to the exact trace evidence behind each question. A domain expert reviews the evidence and records a verdict like acceptable, problematic, or uncertain.

By design, a coding agent authors it, not you. The agent maps the sessions, drafts the questions, and pins evidence; the reviewer opens the dashboard’s review page, answers, and sets the verdict. You can then run an evaluator against reviewed sessions and compare its judgments with the human answers before using it to block a release.

Kitaru, however, lacks the mature assignment and queue management available in the other two products.

LangSmith

LangSmith thread view of a research-assistant run with the Add to Annotation Queue menu open, listing queues such as Evaluator alignment and Chat LangChain Golden Dataset

Source

LangSmith’s annotation queues give human reviewers a focused workflow for attaching feedback to specific runs or threads, in single-run or pairwise style. If you’re tuning a support agent’s tone, reviewers can judge which answer is more helpful without agreeing on a perfect numerical score first.

You can reserve items to avoid duplicate work and require several reviewers to complete an item before it’s done.

The queue’s labels can also inform an LLM-as-a-judge evaluator. LangSmith includes a workflow for comparing judge scores with human feedback, so you can check whether the evaluator follows your reviewers’ decisions.

Braintrust

Braintrust trace with two user reviews attached, showing a Response quality score of Good, Partial or Bad and a Failure reason note from a human reviewer

Source

Braintrust’s human review scores are configured per project as categorical, continuous, or free-form text written to metadata or expected. You can score logged behavior, receive assignments, and track work through the Review page. Teams can also correct an expected value directly when the recorded output was wrong.

Corrections are first-class, and so expected values define the correct or ideal output for a given input. For example, a reviewer might replace a mistaken refund amount with the approved amount. That correction then becomes the reference for future evaluations instead of accidentally treating the production mistake as ground truth.

However, Starter includes one human-review score per project. Unlimited review scores require Pro or Enterprise, even though all plans support unlimited users.

Bottom line: Kitaru wins. LangSmith and Braintrust run the better queues, but both still send your domain expert into a trace to read it before scoring anything. Kitaru does the reading for them. The coding agent pins the evidence and asks one focused question, so the expert only judges. Those verdicts land on the same sessions you replay, so the evaluator you calibrate against them is checked on the exact cases it will gate.

Feature 4. Experiment Comparison and Regression Detection

A higher average score means little if the new model breaks a refund your customers care about. Here the experiment’s mechanics matter.

Kitaru

Animated walkthrough of a Kitaru experiment: the original session and its replay side by side, with the changed agent version, tools, tool policy and output, then the evaluator verdict flipping from fail to pass

An experiment in Kitaru is a named change, replayed across a cohort and evaluated. It holds an override (model, prompt, params), a tool policy, and the evaluator list. For code changes, you register a new agent version.

The key difference is the tool policy, which keeps the comparison controlled:

  • history returns a recorded result for a matching tool call
  • static returns a canned result
  • passthrough calls the live tool

If no recorded result matches, the policy can fail, return a controlled error, or allow a live call.

Consider a refund agent that previously called refund_payment(order_id=4821). During replay, a history policy can return the saved response without issuing a second refund. A new tool call with no recorded match can stop the test instead of touching the payment system.

That safety isn’t automatic. The server defaults to live passthrough if no policy is supplied, and adapter support varies. The OpenAI Agents adapter, for example, requires named history overrides for supported direct function tools instead of a global history default.

Kitaru recommends replaying an unchanged agent first to check fidelity. Once that works, you can change one variable and compare baseline and candidate evaluations.

The good part is, a failed replay remains visible instead of quietly disappearing from the result count. The bad part is, the comparison views are lighter than the other two, with no charts and no pairwise mode.

LangSmith

LangSmith Experiments tab with feedback and latency bar charts, a table of experiments with correctness and similarity scores, and three experiments selected for Compare

Source

LangSmith runs a target function against dataset examples through evaluate(). Change the prompt or model, run another experiment, and open the comparison view.

The comparison view starts from a selection. You can select two or more experiments and then click Compare. LangSmith marks regressions in red and improvements in green. The UI also supports output diffs and side-by-side trace inspection, which helps you see why a score changed.

The rows are fresh executions, so if the agent calls a payment service during an experiment, LangSmith doesn’t automatically substitute the original production response. You must control that dependency in your application or test environment.

LangSmith can run your agent again on the same input; it can’t re-run a recorded production session with the model swapped, and the tool results held fixed.

Braintrust

Braintrust experiment comparison of five models against a sonnet-4.5 base, with score deltas, improvement, tradeoff and regression labels, and duration and token counts per experiment

Source

Braintrust’s comparison starts from a baseline or the most recent experiment on the same git branch. Once set, a score delta column appears in the table, and each row is color-coded. It matches test cases, calculates score changes, and helps you sort through regressions.

In CI, the braintrustdata/eval-action runs evaluations and comments on pull requests. bt eval returns a non-zero exit code if any eval throws an exception. To block a merge when quality falls below your threshold, you can define a custom reporter or another explicit failure rule.

Bottom line: Braintrust has the most complete CI reporting workflow. LangSmith wins on the detailed, in-UI comparisons. Both sides of Kitaru’s comparison are the same recorded session, with the world held constant by a tool policy, so Kitaru wins on control.

LangSmith vs Braintrust vs Kitaru: Pricing Comparison

All three offer a free starting point, but their bills follow different meters. Here’s what matters when usage grows.

Kitaru

Kitaru is free and open source under Apache 2.0. You host the server, import and record sessions without limits, and replay on your own workers with your own model keys.

The paid tier is ZenML Pro. It hosts Kitaru and ZenML for your team and adds AI engineers that read every trace your agent sends, flag evals that are wrong or missing, and draft the fix.

  • Pro Starter: $39 per month, flat. One AI agent, one ML project, and unlimited seats, with a hosted dashboard, trace import, evaluators, cohorts and replay, scheduled eval runs, and alerts when an eval starts drifting. There is no cap on sessions.
  • Pro Enterprise: Custom pricing for more agents, projects, and workspaces. Adds on-prem or air-gapped deployment, your own LLM endpoint, SSO (SAML / OIDC), SCIM, custom roles, audit logs, and an SLA with a named engineer.

Note: Starter has a 14-day free trial with full access and no credit card. One agent means one shipped product, even if it runs several sub-agents. Replays use your own model keys, so those model costs stay on your provider’s bill.

ZenML pricing cards: Open Source free and self-hosted, Pro Starter at $39 per month with a 14-day free trial, one AI agent, one ML project and unlimited seats, and Pro Enterprise with custom pricing

LangSmith

LangSmith meters usage across its services in LangChain Standard Units. One LSU costs $1.

  • Developer: Free up to 5k base traces per month
  • Plus: $39 per seat per month
  • Enterprise: custom

Base traces are retained for 14 days, while extended retention is 180 days by default. Traces beyond the included allowance cost extra. LangSmith also meters certain deployment and agent services separately, so tracing volume isn’t the only thing to watch on a larger bill.

LangSmith pricing cards: Developer at $0 per seat with up to 5k base traces a month, Plus at $39 per seat with up to 10k base traces, and Enterprise with custom pricing

Source

Braintrust

Braintrust combines a platform fee with processed-data and scoring allowances. All plans include unlimited users, projects, datasets, and experiments.

  • Starter: Free; 1 GB of processed data, 10,000 scores, $10 in model credits, and 14-day retention.
  • Pro: $249 per month and includes 5 GB of processed data, 50,000 scores, $100 in monthly model credits, and 30 days of retention.
  • Enterprise: Custom pricing

Braintrust pricing cards: Starter at $0 a month with 1 GB processed data and 10k scores, Pro at $249 a month with 5 GB and 50k scores, and Enterprise with custom pricing

Source

Final Recommendation: Which One Should You Choose?

The three tools are not interchangeable. The right choice depends on what you already have and what you need to prove.

  • Choose LangSmith if you build on LangChain or LangGraph and want trace debugging, online evaluations, dataset management, and structured review queues in one place.
  • Choose Braintrust if your team already treats evals as part of every release. It connects tracing, reviewer corrections, versioned datasets, baseline comparisons, and GitHub Actions.
  • Choose Kitaru if your agent is live and you need to test a model, prompt, or code change against complete production-derived sessions. It can sit alongside LangSmith or Braintrust without replacing either.

We’re biased on the last point, so here is the trade-off in full. Kitaru isn’t an observability dashboard, and neither does it have alerting, online evaluators, regression sorting, or charts in its comparison views.

What it has is the mechanism the other two don’t. It freezes production sessions into immutable cohorts, replays them against your code, and holds the world constant with a tool policy while you change the agent.

To investigate yesterday’s bad answer, start in LangSmith or Braintrust. To test whether tomorrow’s fix breaks other real cases, Kitaru adds a different test.

Ready? Try Kitaru out for free.

Keep your AI getting better
An always-on AI engineering team that watches your agents, finds what’s broken, and proves the fix works.
Start free
From $39 a month · 14 days free
  • AI engineers on your agent
  • Set up in minutes
  • No credit card

AI orchestration,
on the infra you choose

Get new posts by email

New posts and product updates, sent when we publish. No spam.

By subscribing, you agree to our privacy policy.