Calibre has run over 10,000 ticket pipelines in production. Here's what we've learned about error recovery, cost tracking, rate limiting, API failure handling, and the monitoring dashboard that keeps it all operational.
Running an AI agent in production is different from running a demo. A demo works on the ideal path: the API is up, the token budget is infinite, the tool output is clean, and the task is well-scoped. Production is the complement of that list: APIs fail, tokens cost money, tool output is messy, and the ticket that triggers the run was written by someone who had already moved on to another task before finishing the description.
Calibre has run over 10,000 ticket pipelines in production across 5+ developers. These are the lessons that don’t show up in the demo.
The agent loop makes many external calls per turn: Claude API, GitHub API, Jira API, MySQL queries, shell commands. Any of them can fail. The default behavior of most agent frameworks is to crash or retry infinitely. Both are wrong.
Transient failures (network timeouts, 429 rate limits, 503 service unavailable): retry with exponential backoff. Three retries, starting at 1 second, maxing at 10 seconds. If all three fail, the failure is no longer transient.
Permanent failures (401 auth, 400 bad request, 404 not found): don’t retry. Log the failure and return a structured error to the agent. The agent can decide whether to adapt (e.g., the file it was looking for doesn’t exist, try a different path) or escalate.
Semantic failures (the API returned 200 but the response is an error message): these are the hardest. The tool executed successfully but the operation didn’t produce the expected result. The only defense is episodic logging that captures the full request and response for later forensic analysis.
The implementation pattern is a typed result:
type ToolResult<T> =
| { ok: true; value: T }
| { ok: false; error: ToolError };
class ToolError {
constructor(
public readonly kind: 'transient' | 'permanent' | 'semantic',
public readonly message: string,
public readonly recoverable: boolean,
) {}
}The dispatch loop checks result.ok before continuing. A permanent error pauses the current phase and reports to the agent. Three transient errors in a row abort the entire run.
Cost is the operational concern that everyone ignores until the first $500 bill. Calibre tracks cost at four levels:
Per API call: every Claude invocation logs the model, input tokens, output tokens, and calculated cost. This is the atomic unit of cost tracking.
Per phase: the refine phase might cost $0.30, the execute phase $2.10, the validate phase $0.80. Knowing phase costs tells you where the expensive decisions happen.
Per run: total cost of a single ticket pipeline. Median cost is ~$3.50. Range is $0.80 (simple documentation PR) to $25 (complex migration with multiple back-and-forth reviews).
Per developer: costs are attributed to the developer who triggered the run via X-Developer-Id. This surfaces who is using the system and whether usage patterns are reasonable.
The cost ceiling is the primary safety mechanism. If a run exceeds $15, the agent enters “cost-critical mode”: it switches to cheaper models (Haiku instead of Sonnet), reduces context retention, and warns the developer. The run is paused for human approval before exceeding a hard cap.
The agent makes API calls faster than a human would. This means it hits rate limits that a human never would. Three patterns matter:
Client-side rate limiting: the agent should never send requests faster than the API’s documented limit. A token bucket per API endpoint with a conservative rate (e.g., 80% of the documented limit) prevents 429s before they happen.
Adaptive backoff: if a 429 arrives despite the rate limiter, the agent must back off. The backoff duration comes from the Retry-After header if present, or a binary exponential backoff starting at 5 seconds. The agent should not resume making requests for that API until the backoff window expires.
Concurrent request limiting: the agent may issue multiple tool calls in a single turn (parallel dispatch). Each of those tool calls may hit external APIs. Limit concurrent outbound API calls to 3 per agent instance. More than that and you’re compounding rate limit risk across parallel requests.
Calibre has a web dashboard that shows, in real time:
The dashboard is intentionally minimal: the goal is not to visualize every metric, it’s to surface the five signals that require human attention. Everything else goes to episodic logs that are queried during incident investigation.
After 10,000+ runs, here’s what the data shows:
The edge cases are real but manageable. A Jira ticket with a broken description (no steps to reproduce, unclear expected behavior) adds ~8 minutes because the agent has to explore the codebase to reverse-engineer the problem. A PR that touches a migration is always reviewed by a human regardless of confidence. And every three months, an API change on the Jira or GitHub side breaks something and requires a configuration update.
The lesson: build for the failures, monitor the costs, and make the system visible. An agent running in a dark cave will eventually break and no one will know.
Related: These operational patterns are embedded in Calibre’s phase-gated pipeline infrastructure — episodic logging captures every decision, cost ceilings cap every run, and policy flags let humans override the agent’s judgment.