Technical overviewApplied AI and reliability engineering

Incident Triage Assistant

A structured incident triage grounded in the runbooks it retrieves.

Source code

Inspect the implementation, commit history, and project documentation.

View source on GitHub(opens in a new tab)
Role
Solo developer: retrieval, structured AI output, CLI, Streamlit, tests, evaluation
Evidence status
Retrieval verified; live Claude path pending a local API credential
Verified
2026-07-22

01

Purpose and scope

Turns an alert, log, or stack trace into a structured triage: a dependency-free BM25 stage retrieves relevant runbooks, then Claude returns a typed root cause, severity, remediation plan, commands, and citations.

What it is used for

  • Turn an alert or stack trace into a first-pass triage.
  • Ground remediation commands in the most relevant runbooks.
  • Expose the same typed result through a CLI or Streamlit.
  • Python
  • Claude API
  • BM25
  • Pydantic
  • Streamlit
  • pytest

02

Architecture

Input channels

Operator interface

Command-line interface

Accepts alerts, logs, and stack traces

Web interface

Streamlit interface

Interactive submission and results

Incident text

Retrieval boundary

Application service

Triage engine

Coordinates retrieval, prompting, validation

Retriever

BM25 index

Ranks passages by lexical overlap

Knowledge base

Seven runbooks

Infrastructure remediation knowledge

Top-ranked context

AI boundary

Grounding

Prompt assembly

Combines incident and retrieved evidence

External model

Claude API

Generates a structured proposal

Type boundary

Pydantic schema

Validates fields before use

Validated result

Structured triage

Output

Diagnosis

Probable root cause and severity

Output

Action plan

Fix steps and exact commands

Output

Evidence and escalation

Runbook citations and escalation

Text equivalent: An incident enters through the CLI or Streamlit; the triage engine queries a BM25 index of seven runbooks, assembles the top-ranked context for Claude, and Pydantic validates the response before the structured triage returns to either interface.

03

Engineering decisions

Lexical retrieval first

Infrastructure errors share exact tokens with runbooks, so BM25 keeps the first stage transparent, fast, and offline-testable.

Typed output boundary

A Pydantic schema makes the response usable by a CLI, UI, or ticket without brittle text parsing.

Retriever protocol

The retrieval interface can accept a semantic implementation later without changing the engine.

04

Verification evidence

  • All 16 offline tests passed on July 22, 2026.
  • All 10 labelled retrieval cases returned the expected runbook in the top three, and MRR was 1.000.
  • The live Claude path was not run because no Anthropic key was available on the machine.
verified-2026-07-22
$ python -m pytest
16 passed in 0.17s
$ python evals/run_evals.py
hit@3: 10/10 (100%) · MRR: 1.000
△ live Claude evaluation pending credential

05

Limitations