Friction: manual E2E on new projects + taxonomy triage loop #231

New issue

david · 2026-08-16T22:12:27Z

david commented

2026-08-16 22:12:27 +00:00

Context

Friction capture depends on real LLM output, which is non-deterministic — a gated automated E2E cannot reliably assert that agents emit the ## Friction section. Per the interview decision, validation is operator-run manual E2E against new projects (no gated automated E2E for friction). This issue is the manual validation checklist + the data-driven taxonomy triage loop.

Depends on: foundation, implement wiring, remediate wiring, aggregator, and docs issues all merged.

Manual E2E checklist

Run the full pi-loop pipeline (npm start -- <ISSUE-REF>) against 2–3 fresh projects covering varied ecosystems (e.g. one TypeScript/Node repo, one Python repo, one repo with sparse documentation), and verify each item:

Per-run artifacts — .pi-loop/runs/<run-id>/friction.json exists after implement and after remediate; validates against the schema (schemaVersion: 1, entries stamped with stage / runId / model / recordedAt); FRICTION.md renders the same entries.
Content quality — entries use the fixed categories sensibly (no other dominating in these runs); descriptions are specific (what was missing / what failed); no checkbox-syntax entries; no friction section corrupted the remediate annotated findings (remainingFindings in mr-result.json matches the unchecked findings).
Aggregate — .pi-loop/friction/friction.json + FRICTION.md are recomputed after each run; counts (byCategory, byStage, byModel, runs, totalEntries) are correct; re-running the pipeline on the same repo does not duplicate entries (idempotent — run-id+stage dedup by construction).
Failure path — kill a run mid-remediate (Ctrl-C); re-run the same command (resume); verify the aggregate is recomputed, the resumed stage contributes once, and the run completes with friction artifacts for the completed stages.
No diff leak — the MR's diff (implementation.diff / final.diff / the MR on the origin platform) contains no friction files; .pi-loop/ is untouched by the agent and excluded from the diff (ADR-009 preflight enforces the .gitignore rule).
Console — the run summary prints the Friction: line (n events (m categories) / no events / aggregate unavailable).
No downstream surfacing — friction does not appear in the MR description or the Jira/Forgejo write-back comment (by design).

Taxonomy triage loop (deferred, data-driven)

After the manual runs (and once real usage accumulates), inspect the aggregate's byCategory:
- If other dominates (> ~30% of entries), open a follow-up issue proposing new categories derived from the actual other descriptions (the taxonomy is deliberately fixed until the data justifies extension — ADR-016).
- If a category is consistently empty across many runs, consider whether it earns its slot (only remove with data; keeping an unused category is cheap).
Report the findings and the taxonomy decision back to this issue, then close it.

Acceptance criteria

Items 1–7 of the checklist pass on at least 2 projects (record which repos/tickets were used in a comment on this issue).
The taxonomy triage decision is documented (extend / keep as-is) with the supporting counts.
Any taxonomy change is tracked as a separate issue (schema + parser + docs must change together).

Dependencies

Foundation, implement wiring, remediate wiring, aggregator, and docs issues merged.
Live credentials + an LLM-configured environment (PILOOP_MODEL, provider tokens) to run real pipelines.

Notes

This issue is the manual E2E for the whole friction feature — no automated E2E test is added (deliberate, per the design decision).
Expected runtime: a few hours of operator time across the sample projects.

## Context Friction capture depends on real LLM output, which is non-deterministic — a gated automated E2E cannot reliably assert that agents emit the `## Friction` section. Per the interview decision, validation is **operator-run manual E2E against new projects** (no gated automated E2E for friction). This issue is the manual validation checklist + the data-driven taxonomy triage loop. Depends on: foundation, implement wiring, remediate wiring, aggregator, and docs issues all merged. ## Manual E2E checklist Run the full pi-loop pipeline (`npm start -- <ISSUE-REF>`) against **2–3 fresh projects** covering varied ecosystems (e.g. one TypeScript/Node repo, one Python repo, one repo with sparse documentation), and verify each item: 1. **Per-run artifacts** — `.pi-loop/runs/<run-id>/friction.json` exists after implement and after remediate; validates against the schema (`schemaVersion: 1`, entries stamped with `stage` / `runId` / `model` / `recordedAt`); `FRICTION.md` renders the same entries. 2. **Content quality** — entries use the fixed categories sensibly (no `other` dominating in these runs); descriptions are specific (what was missing / what failed); no checkbox-syntax entries; no friction section corrupted the remediate annotated findings (`remainingFindings` in `mr-result.json` matches the unchecked findings). 3. **Aggregate** — `.pi-loop/friction/friction.json` + `FRICTION.md` are recomputed after each run; counts (`byCategory`, `byStage`, `byModel`, `runs`, `totalEntries`) are correct; re-running the pipeline on the same repo does not duplicate entries (idempotent — run-id+stage dedup by construction). 4. **Failure path** — kill a run mid-remediate (Ctrl-C); re-run the same command (resume); verify the aggregate is recomputed, the resumed stage contributes once, and the run completes with friction artifacts for the completed stages. 5. **No diff leak** — the MR's diff (`implementation.diff` / `final.diff` / the MR on the origin platform) contains **no** friction files; `.pi-loop/` is untouched by the agent and excluded from the diff (ADR-009 preflight enforces the `.gitignore` rule). 6. **Console** — the run summary prints the `Friction:` line (`n events (m categories)` / `no events` / `aggregate unavailable`). 7. **No downstream surfacing** — friction does not appear in the MR description or the Jira/Forgejo write-back comment (by design). ## Taxonomy triage loop (deferred, data-driven) - After the manual runs (and once real usage accumulates), inspect the aggregate's `byCategory`: - If `other` dominates (> ~30% of entries), open a follow-up issue proposing new categories derived from the actual `other` descriptions (the taxonomy is deliberately fixed until the data justifies extension — ADR-016). - If a category is consistently empty across many runs, consider whether it earns its slot (only remove with data; keeping an unused category is cheap). - Report the findings and the taxonomy decision back to this issue, then close it. ## Acceptance criteria - Items 1–7 of the checklist pass on at least 2 projects (record which repos/tickets were used in a comment on this issue). - The taxonomy triage decision is documented (extend / keep as-is) with the supporting counts. - Any taxonomy change is tracked as a **separate** issue (schema + parser + docs must change together). ## Dependencies - Foundation, implement wiring, remediate wiring, aggregator, and docs issues merged. - Live credentials + an LLM-configured environment (`PILOOP_MODEL`, provider tokens) to run real pipelines. ## Notes - This issue is the manual E2E for the whole friction feature — no automated E2E test is added (deliberate, per the design decision). - Expected runtime: a few hours of operator time across the sample projects.

david added the

enhancement

pi-loop

labels

2026-08-16 22:12:37 +00:00

david removed the

enhancement

label

2026-08-18 02:18:36 +00:00

Sign in to join this conversation.