Runbook 05 — Test data + state reset (idempotent runs)

Quick links

Navigation

Use as a checklist.

Objective Done Invariants Mechanics Commands Expected Failure modes Operational value

Objective

Run is repeatable: execute once, twice, or back-to-back with consistent outcomes.
Run is idempotent: data creation does not collide with existing data.
Run is parallel-safe: design supports multiple workers (even if workers=1 today).
State is explicit: seed via API, validate via UI, no cross-test dependencies.

Auth contract

API seeding uses an authenticated request context (JWT attached per request).
UI runs use storage state generated by global setup (.auth/storageState.json).
No test performs ad-hoc UI login as a prerequisite.

Classification: API 401/403 = environment/credentials/endpoint contract. Do not treat as flaky.

Rule

Each test must pass when run alone. Each test must not require another test to run first.

System contract

INPUT	Authenticated user (API + UI) Clean or irrelevant prior state Unique test data generator
OUTPUT	Deterministic records created via API UI reflects created state immediately
DETERMINISM	Unique identifiers per run No shared state between tests No ordering dependency
FAILURE SIGNAL	UI cannot find seeded data → state propagation or API issue API 401/403 → auth/environment contract failure Duplicate/collision → test data generator failure

Definition of done

✔ Each test creates its own records (or validates records it created via API).
✔ Re-run produces distinct records (unique slugs/identifiers).
✔ UI reads validate API-seeded state without UI setup flows.
✔ CI output includes artifacts when configured (report + failure evidence).

Invariants

Unique data	All created entities use a unique suffix (timestamp + random).
No ordering	No “A then B” dependencies. Any test can run first.
API seeds	Setup performed via API unless the UI flow is the subject under test.
Auth centralized	Auth created once (global setup) and reused (storage state + API context).
Config explicit	`WEB_BASE_URL` and `API_BASE_URL` are required (no implicit defaults).

Local file references

Unique data generator: helpers/testData.js (timestamp + random suffix).

Mechanics

Targeting

Config hard-fails if API_BASE_URL missing.
Config hard-fails if WEB_BASE_URL (or legacy BASE_URL) missing.

Auth lifecycle

fixtures/global-setup.js performs API auth preflight and writes storage state.
Unauthenticated flows use .auth/emptyState.json.

State creation + UI validation

API creates article(s) via authenticated client (helpers/apiClient.js).
UI navigates to confirm list visibility and detail-page open.

Use API to create preconditions.
Use UI to validate user-visible reads and navigation.
Use unique identifiers per created record.

Don’t

Do not rely on “existing” feed content.
Do not chain tests via shared runtime state.
Do not add fixed sleeps to paper over state issues.

Commands

Set environment (example)

export WEB_BASE_URL="http://sut.testlab:3000"
export API_BASE_URL="http://sut.testlab:3001/api"
export RW_USER_EMAIL="YOUR_USER_EMAIL"
export RW_USER_PASSWORD="YOUR_USER_PASSWORD"

npx playwright test specs/sprint5   --project=auth-chromium   --workers=1   --reporter=line

npx playwright test specs/sprint5   --project=auth-chromium   -g "S5.003"   --workers=1   --reporter=line

Reset behavior

Re-run is expected to create new records. If the app enforces uniqueness constraints, update test data generator to avoid collisions.

Expected outputs

✔ S5.001 creates article via API and opens it via UI.
✔ S5.002 creates two independent articles (distinct slug/ID).
✔ S5.003 finds API-created article in Global Feed and opens detail page.
✔ Failure evidence present when enabled in config: trace/video/screenshot + HTML report.

Evidence locations

test-results/ (per-test artifacts)
playwright-report/ (HTML report)

Failure modes

Config fails fast: missing/empty API_BASE_URL or WEB_BASE_URL. Action: print env and correct file load.
Auth fails in global setup: API login preflight fails. Action: validate RW_USER_EMAIL/RW_USER_PASSWORD and API_BASE_URL includes /api.
UI cannot find seeded article: list assertion fails. Action: confirm feed scope (Global vs Personal), then verify article exists via API read-back before UI assertion.
Data collision: create call fails or UI asserts wrong entity. Action: enforce unique suffix and avoid reused titles/slugs.
Hidden dependency: test requires prior UI steps. Action: move setup into API seeding / fixtures; remove UI prerequisite flow.

Triage commands

echo "WEB_BASE_URL=$WEB_BASE_URL"
echo "API_BASE_URL=$API_BASE_URL"
echo "RW_USER_EMAIL=$RW_USER_EMAIL"

# Optional reachability
curl -sS -I "$WEB_BASE_URL/" | head -n 3
curl -sS -I "$API_BASE_URL/tags" | head -n 3

Operational value

Re-run without cleanup.
Run tests in any order.
Increase workers later without redesign.
Diagnose failures using CI logs + artifacts.

System flow

UPSTREAM	Sprint 4 — Flake control and environment discipline
DOWNSTREAM	Sprint 6 — Time-to-signal (fast failure visibility) Sprint 14 — Structured failure artifacts Sprint 15 — Failure triage classification Sprint 17 — Release gate decision

Why this matters

If state is not deterministic here, all downstream signals become unreliable. This runbook ensures failures are attributable to the system under test—not test design.