Runbook 15 — CI failure triage agent

Quick links

Navigation

Use as a checklist.

Objective Done Inputs Mechanics Commands Expected Failure modes Operational value

Objective

Read the structured failure artifacts from Sprint 14.
Produce a deterministic first-pass diagnosis for each failed test.
Point the reviewer to the right evidence first.
Reduce time-to-orientation without pretending to fix the issue.

Rule

The triage agent is a disciplined first responder, not an autonomous fixer.

Definition of done

} return { classification: 'UNKNOWN', confidence: 0.4, reason: 'No current deterministic rule matched.', recommendedNextStep: 'Inspect trace and extend taxonomy if pattern is legitimate.' &125; }

Commands

node scripts/triage-agent.js
cat test-results/triage/triage-report.json | jq .

GitLab shape

triage_failures:
  stage: report
  script:
    - node scripts/triage-agent.js
    - cat test-results/triage/triage-report.json
  artifacts:
    when: always
    paths:
      - test-results/triage/

Expected outputs

✔ Every failed test is classified into a bounded category.
✔ The report gives a reason and a next step rather than echoing raw stack text.
✔ The reviewer knows which artifact to open first.
✔ Unknown remains available when the rule set is incomplete.

Failure modes

Triage adds no value: it only repeats raw error text. Action: summarize category, reason, next step.
Everything becomes TIMING: action: prioritize specific rules above generic timeout handling.
LLM enrichment contradicts deterministic rule: action: deterministic classification wins.
Bad fix suggestions: action: forbid timeout inflation, blind retries, and DOM-coupled selector drift.

Operational value

Faster failure orientation in CI.
Consistent first-pass diagnosis across reviewers.
Clear evidence-first workflow for new hires.
Foundation for future trend reporting and agent memory.