Improving a Claims Agent
Who keeps a claims agent improving, batch after batch?
- ZenML AI engineer
- +4.0 points
- Codex
- +0.7 points
Agent improvement · Model switch · Public data
Evals your underwriters
would sign off on
Our AI engineer reads your agent's traces next to your evals and your underwriting guidelines, and shows you what the evals miss.
The problem
The dashboard says 91%.Underwriters still redo the quotes.
Your agent quotes, triages claims or drafts appeals, and its traces already land in your tracing tool. The evals say it's fine. Your underwriters keep finding the same mistakes.
- 01
The manual moves, the judges don't
Guidelines change every quarter. The judges were written against last year's.
- 02
Someone will ask how you tested it
A pass rate is not an answer when a regulator or a broker asks why a quote was priced the way it was.
- 03
Nobody wants the job
Every hour an underwriter spends grading an agent is an hour not spent underwriting.
How it works
You answer a few questions. Good evals fall out.
You don't write evals from scratch. Your AI engineer asks only what it can't work out alone.
- 01
Connect your repos
Add the GitHub app and your tracing keys. Revoke them any time.
- 02
A short interview
A few questions about what it found. Tap to answer.
- 03
Hear what your evals miss
What your evals missed last week, in your order.
- 04
Fifteen minutes a week
Ten quick questions on three real sessions.
Security
Built for regulated data
Run it on-prem and policyholder data, loss runs and inspection reports never leave your walls. You choose where it runs and which models it calls.
- 01
Runs where you need it
Your AI engineer runs in our cloud by default, or on-prem in your own cloud, air-gapped if you need it. Kitaru is open source and runs on your machine.
- 02
Your models, your keys
Judges and replays call the providers you choose with your own keys, including models you host yourself.
- 03
SOC 2 Type II and ISO 27001
Independently audited for security, availability and confidentiality.
- 04
SSO, roles and audit logs
SAML or OIDC single sign-on, SCIM provisioning, custom roles and org-wide audit logs.
- 05
Your encryption keys
Bring your own keys for encryption, and choose the region your data lives in.
- 06
A record of every change
Reviewed sessions, verdicts and replays are kept, so you can show how each change was tested before it shipped.
Benchmarks
Tested on insurance work
We run real claims, appeals and underwriting work through open and frontier models, and let our AI engineer improve the agent batch after batch. Every result publishes its method, its noise and its limits.
All insurance benchmarksGain over the agent left alone, 800 appeals · higher is better Underwriting Agent
Which model should an underwriting agent run on?
Cost per correct decision, models tied at the top · lower is better - Claude Haiku 5.5
- $0.0038
- GLM 5.3 Flash
- $0.0092
- GLM 5.3
- $0.0737
Model choice · Prompting · Cost · Public data
Trusted to run production AI, from pipelines to agents, at
Questions from security and risk
Where do our traces go?
Your traces stay in the tracing tool you already use. By default, your AI engineer runs in our cloud and reads them through keys you can revoke. Deploy it on-prem and it runs inside your own cloud, so nothing leaves your infrastructure.
Can we use our own models?
Yes. Judges and replays call the providers you pick with your keys, and you can point them at models you host yourself.
Which certifications do you hold?
ZenML holds a SOC 2 Type II report and ISO 27001 certification. Reports and policies are in our trust center.
Can it run air-gapped?
Yes. ZenML Pro supports multi-region and air-gapped deployments. See deployment options.
Can we show model risk how it was tested?
Yes. Every reviewed session, verdict, check and replay is kept with who decided and when. Export it as evidence for model validation, your model risk committee or an audit.
Which agents does it work with?
Any agent whose traces you already collect: quoting, underwriting triage, claims intake, appeals and customer service. It reads traces from Langfuse, LangSmith, Braintrust, Logfire, MLflow or Phoenix.
Does it help with state AI rules?
It gives you the testing records that AI governance rules ask for, such as the NAIC model bulletin many states have adopted. It doesn't make you compliant on its own; it gives your compliance team the evidence.
Bring one agent
and a week of traces
We'll show you what your evals miss on your own data, and which checks your underwriters would actually sign off on.