Manual E2E validation checklist for MR-skip feature #258

New issue

david · 2026-08-18T02:15:32Z

david commented

2026-08-18 02:15:32 +00:00

Summary

Manually validate the full MR-skip feature end-to-end against a real sandbox repo/issue once all the automated-code steps have landed, covering single-run, resume, and batch scenarios that unit tests cannot fully exercise (real git state, real platform REST calls, real resume across process invocations).

Background

Depends on: Add git-native "nothing to ship" helper
Depends on: Add mr-skipped.json artifact type + validator
Depends on: Wire "nothing to ship" skip into runMrStage
Depends on: Wire MR-skip outcome into runPipeline
Depends on: Recognize mr-skipped.json in resume/completeness detection
Depends on: Add mr-skipped outcome to batch mode

Per this repo's established testing convention (see docs/adr/016-friction-logging.md's precedent, which this feature's design record explicitly follows — see docs/adr/017-mr-skip-when-no-changes.md §11): deterministic logic gets unit tests (covered by the six preceding steps), but full end-to-end behavior — real git operations, real branch push/no-push behavior, real resume across separate CLI invocations, real batch continuation — is validated manually by the operator rather than via a new automated gated e2e (PILOOP_E2E=1) scenario. This step is that manual validation pass, not a code change.

This issue has no code deliverable — it's a checklist to work through and confirm once the preceding six steps are merged, run against pi-loop's normal sandbox validation setup (a throwaway repo + throwaway issue, per this repo's existing PILOOP_E2E=1 / gated e2e conventions).

Implementation Details

N/A — no source changes. This is a manual verification pass performed by running the CLI (npm start -- <ISSUE-REF> and its batch variants) against a sandbox target repo, using an issue/scenario engineered to produce no net diff after implement/remediate/docs (e.g. an issue asking for a change that's already present, or a trivial no-op request).

Acceptance Criteria

Single-issue run producing no net diff: running npm start -- <KEY> against a no-op issue results in mr-skipped.json being written to the run dir, no branch pushed to origin (verified via git branch -r or the Forgejo/GitLab/GitHub UI), no MR/PR created, no comment/write-back posted on the issue, and a CLI summary that reads as the dedicated "nothing to ship" message.
Resume after an mr-skipped run: re-running npm start -- <KEY> against the same issue/run dir causes the CLI to report the run as already complete (no full pipeline re-run), rather than restarting any stage.
Batch run including an mr-skipped issue: running npm start -- ISSUE-A ISSUE-B --auto-merge where one issue is engineered to produce no net diff confirms the batch does not stop on that issue, batch-manifest.json records { status: 'mr-skipped', reason: 'no-changes', runDir: ... } for it, and the next issue in the batch still processes correctly (its preflight isClean/pull succeed despite the prior issue leaving an empty feature branch checked out locally).
Local branch inspection: after the single-issue run in the first bullet, the local feature branch for the skipped issue still exists (git branch --list 'feature/<KEY>/*') and was never deleted.

Test Plan

Walk through each acceptance-criteria scenario against a sandbox repo, in order:

Trigger a single no-op issue run; inspect the run dir for mr-skipped.json, check git branch -r / the platform UI for the absence of a pushed branch/MR, check the issue for the absence of a write-back comment, and read the CLI's final summary output.
Re-run the same command against the same issue key; confirm the CLI's resume-detection logs report the run as already complete and no stages execute.
Run a batch command with two issues (one no-op, one normal) and --auto-merge; inspect batch-manifest.json after completion for the mr-skipped entry, and confirm the second issue's outcome is completed with no batch-level abort.
Run git branch --list 'feature/<KEY>/*' in the sandbox repo after step 1 and confirm the branch is present.

Record the outcome of each step (pass/fail + notes) as a comment on this issue once complete.

## Summary Manually validate the full MR-skip feature end-to-end against a real sandbox repo/issue once all the automated-code steps have landed, covering single-run, resume, and batch scenarios that unit tests cannot fully exercise (real git state, real platform REST calls, real resume across process invocations). ## Background **Depends on:** Add git-native "nothing to ship" helper **Depends on:** Add mr-skipped.json artifact type + validator **Depends on:** Wire "nothing to ship" skip into runMrStage **Depends on:** Wire MR-skip outcome into runPipeline **Depends on:** Recognize mr-skipped.json in resume/completeness detection **Depends on:** Add mr-skipped outcome to batch mode Per this repo's established testing convention (see `docs/adr/016-friction-logging.md`'s precedent, which this feature's design record explicitly follows — see `docs/adr/017-mr-skip-when-no-changes.md` §11): deterministic logic gets unit tests (covered by the six preceding steps), but full end-to-end behavior — real git operations, real branch push/no-push behavior, real resume across separate CLI invocations, real batch continuation — is validated manually by the operator rather than via a new automated gated e2e (`PILOOP_E2E=1`) scenario. This step is that manual validation pass, not a code change. This issue has no code deliverable — it's a checklist to work through and confirm once the preceding six steps are merged, run against pi-loop's normal sandbox validation setup (a throwaway repo + throwaway issue, per this repo's existing `PILOOP_E2E=1` / gated e2e conventions). ## Implementation Details N/A — no source changes. This is a manual verification pass performed by running the CLI (`npm start -- <ISSUE-REF>` and its batch variants) against a sandbox target repo, using an issue/scenario engineered to produce no net diff after implement/remediate/docs (e.g. an issue asking for a change that's already present, or a trivial no-op request). ## Acceptance Criteria - [ ] **Single-issue run producing no net diff**: running `npm start -- <KEY>` against a no-op issue results in `mr-skipped.json` being written to the run dir, no branch pushed to origin (verified via `git branch -r` or the Forgejo/GitLab/GitHub UI), no MR/PR created, no comment/write-back posted on the issue, and a CLI summary that reads as the dedicated "nothing to ship" message. - [ ] **Resume after an mr-skipped run**: re-running `npm start -- <KEY>` against the same issue/run dir causes the CLI to report the run as already complete (no full pipeline re-run), rather than restarting any stage. - [ ] **Batch run including an mr-skipped issue**: running `npm start -- ISSUE-A ISSUE-B --auto-merge` where one issue is engineered to produce no net diff confirms the batch does not stop on that issue, `batch-manifest.json` records `{ status: 'mr-skipped', reason: 'no-changes', runDir: ... }` for it, and the next issue in the batch still processes correctly (its preflight `isClean`/`pull` succeed despite the prior issue leaving an empty feature branch checked out locally). - [ ] **Local branch inspection**: after the single-issue run in the first bullet, the local feature branch for the skipped issue still exists (`git branch --list 'feature/<KEY>/*'`) and was never deleted. ## Test Plan Walk through each acceptance-criteria scenario against a sandbox repo, in order: 1. Trigger a single no-op issue run; inspect the run dir for `mr-skipped.json`, check `git branch -r` / the platform UI for the absence of a pushed branch/MR, check the issue for the absence of a write-back comment, and read the CLI's final summary output. 2. Re-run the same command against the same issue key; confirm the CLI's resume-detection logs report the run as already complete and no stages execute. 3. Run a batch command with two issues (one no-op, one normal) and `--auto-merge`; inspect `batch-manifest.json` after completion for the `mr-skipped` entry, and confirm the second issue's outcome is `completed` with no batch-level abort. 4. Run `git branch --list 'feature/<KEY>/*'` in the sandbox repo after step 1 and confirm the branch is present. Record the outcome of each step (pass/fail + notes) as a comment on this issue once complete.

david added the

enhancement

label

2026-08-18 02:15:32 +00:00

Sign in to join this conversation.