A knowledge base that never prunes itself becomes noise. Track which bullets were shown vs cited, detect which ones came from PRs that were closed unmerged, and soft-delete them automatically. The feedback loop that keeps a playbook accurate.
A knowledge base for AI agents starts useful and degrades over time — unless you build the feedback loop in from the beginning.
The typical pattern: an agent solves a problem, you extract the lesson, you add it to the playbook. The playbook grows. Months later, 40% of the playbook is stale patterns from tickets that turned out to be edge cases, deprecated APIs, or fixes that were later reverted. The agent loads all of it into context. Signal-to-noise drops. Retrieval quality degrades.
The fix isn’t manual curation. It’s measuring lesson usage and auto-demoting bullets that fail the signal test.
Every time a lesson is retrieved and shown to an agent, that’s a load event. Every time the agent cites a lesson in its own output (referencing its ID while reasoning), that’s a cite event.
These are different signals:
Record both events with the phase name, ticket ID, and success flag:
recordLessonLoaded({
lessonIds: retrievedBullets.map(b => b.id),
consumer: 'refine',
ticketId,
phase: 'refine',
});
// Later, after the phase completes:
recordLessonCited({
lessonIds: [...citedIds],
ticketId,
phase: 'refine',
phaseSuccess: success,
});These events accumulate in a .lesson-usage.jsonl file — one line per event, easy to query.
The strongest negative signal is: a bullet was learned from a ticket whose PR was eventually closed without merge.
A closed-unmerged PR means a human reviewed the fix and rejected it. If the playbook has a lesson derived from that fix, the lesson is likely a bad pattern. It should be removed from future agent context.
The demote logic:
needsHumanReview: true was set (meaning the agent itself flagged uncertainty)learned_from: TICKET-IDdeprecated: true on those bulletsif (ticket.planGate?.needsHumanReview && ticket.pr?.state === 'closed') {
const bullets = playbook.filter(b => b.learned_from === ticket.ticket);
for (const bullet of bullets) {
bullet.deprecated = true;
bullet.demoted_on = new Date().toISOString();
bullet.demoted_reason = 'pr_closed_unmerged';
}
}Bullets are never deleted — they’re soft-deprecated. The deprecated: true flag causes the loader to exclude them from retrieval. The record is preserved for audit purposes.
Retrieval quality matters as much as KB quality. The naive approach — deriving query keywords from the ticket ID (CTMS-4337 → keywords ['CTMS', '4337']) — matches zero bullets in practice. Ticket IDs carry no semantic content.
The correct approach derives keywords from the ticket’s content: summary, description, and labels. BM25 tokenizes internally, so you don’t need to over-normalize — just pass the raw text.
function synthesizeKeywords(ticket: JiraTicket): string[] {
return [
ticket.summary,
ticket.description?.slice(0, 500) ?? '',
...(ticket.labels ?? []),
].filter(s => s.length > 0);
}Cap description length. Long ticket bodies dominate term-frequency weighting and drown out the summary and label signal that’s actually most relevant for playbook matching.
Beyond auto-demote, run periodic pruning on bullets that have been in the corpus for more than 30 days without any load or cite events. These are lessons that no ticket has triggered — dead weight.
Add safety gates:
The combination of auto-demote (reacts to explicit rejection signals) and scheduled pruning (clears unused lessons) keeps the playbook from accumulating noise indefinitely.
A 200-bullet playbook where 40 bullets are stale or counterproductive is worse than a 100-bullet playbook where every bullet applies. Context window space is finite. Retrieval recall degrades as noise increases. The agent reads irrelevant bullets and either ignores them (wasted tokens) or is confused by them (active harm).
The feedback loop — retrieve, record, cite, measure, demote — turns the knowledge base from a write-only log into a living system that improves with every run.