Lexical retrieval first
Infrastructure errors share exact tokens with runbooks. BM25 keeps the first stage transparent, fast, offline-testable, and free of embedding infrastructure.
Applied AI and reliability engineering
Incident Triage Assistant
Grounded incident guidance with deterministic retrieval evidence.
- Role
- Solo developer across retrieval, structured AI output, CLI, Streamlit, tests, and evaluation
- Evidence status
- Retrieval verified; live Claude path pending a local API credential
- Verified
- 2026-07-22
01
Problem and scope
The assistant turns an alert, log excerpt, or stack trace into a structured triage. A dependency-free BM25 stage retrieves relevant runbooks, then Claude is designed to return a typed root cause, severity, remediation plan, commands, citations, and escalation guidance.
02
Architecture
Incident input
Alert, error, log excerpt, or stack trace
BM25 retrieval
Ranked lookup across seven infrastructure runbooks
Claude reasoning
Prompt grounded in the top retrieved runbooks
Typed triage
Root cause, severity, steps, commands, citations
Text equivalent: the stages above flow from left to right, with each stage passing its result to the next.
03
Engineering decisions
Typed output boundary
A Pydantic schema makes the model response usable by a CLI, interface, ticket, or future integration without brittle text parsing.
Retriever protocol
The retrieval interface can accept a semantic implementation later without changing the triage engine.
04
Verification evidence
- All 16 offline tests passed on July 22, 2026.
- All 10 labelled retrieval cases returned the expected runbook within the top three.
- MRR was 1.000, so the expected runbook ranked first in every evaluation case.
- The live Claude generation path was not run because no Anthropic key was available on the machine.
$ python -m pytest
16 passed in 0.17s
$ python evals/run_evals.py
hit@3: 10/10 (100%) · MRR: 1.000
△ live Claude evaluation pending credential05
Limitations
- The page does not present illustrative model output as a verified live result.
- BM25 can miss paraphrased incidents with little token overlap; the Retriever protocol is the planned seam for semantic retrieval.
- The seven bundled runbooks are a focused demonstration set, not a production operations knowledge base.
Next step