Watch

issue-289: ship create-issues reference docs instead of deferring to a missing process #290

Merged

david merged 1 commit from bug/issue-289/commit-working-tree-changes-create-issues-ships-its-reference-docs into main

2026-10-06 22:22:10 +00:00

david commented

2026-10-06 11:07:24 +00:00

Owner

What & why

The create-issues skill's Documentation Required section has been naming docs/reference/<slug>/ folders and telling implementers that "A separate process downloads these". That process never existed — it was pi-loop's still-unimplemented research stage (ADR-014) — so the folders were never created. The cost was measurable:

99 .pi-loop/runs/* directories reference docs/reference/... paths
57 shipped issue bodies carry the literal "A separate process downloads" sentence
missing-docs is the second-largest friction category (35 entries)
Implementers recorded the absence in findings (extensions/vision/findings.md: "the separate download process did not run"; extensions/penpot/findings.md: "the docs/reference/ directory does not exist"), and hand-authored fixtures from schemas to compensate

This change makes create-issues own the download end to end: a new Phase 6 turns the researched URLs into a JSON manifest, vendors each page verbatim with a dependency-free fetcher, and ships the tree through the commit-docs workflow — so the folders exist on the default branch before the issues that reference them are implemented.

Per-file breakdown

Added

skills/create-issues/scripts/fetch-reference-docs.mjs — dependency-free (Node ≥ 20.19 built-in fetch) reference-doc fetcher.
- Reads a { groups: [{ slug, name, version, sources: [{ url, summary }] }] } manifest; --out, --dry-run, --refresh, --max-bytes flags.
- All-or-nothing: fetches every source for every group into memory before writing a single file, so a failed fetch leaves no partial tree and exits non-zero.
- Writes pages verbatim to docs/reference/<slug>/<file>, with provenance in _meta.json (url, file, contentType, bytes, sha256, summary) and a regenerated docs/reference/README.md index.
- Idempotent: re-runs are no-ops unless a group's version or URL set changed (--refresh forces); a shrunken URL set removes orphaned files.
- Guards: only http(s) URLs accepted; bodies capped at 5 MB by default; a source cannot overwrite the provenance file.
skills/create-issues/scripts/fetch-reference-docs.test.mjs — 27-case suite covering manifest validation, dry-run (no network/writes), verbatim writes + provenance + index rendering, orphan removal, all-or-nothing failure, version/URL-set staleness, --refresh, the provenance-overwrite guard, and bad-args/bad-manifest exit codes. Verified locally: 27 pass / 0 fail.

Modified

skills/create-issues/SKILL.md — the substantive change:
- New Phase 6: Vendor the Reference Docs — 6.1 write the manifest (one group per slug, version as the staleness gate, GitHub-first URLs), 6.2 --dry-run preview, 6.3 fetch verbatim, 6.4 ship via the commit-docs workflow (date-driven branch, CI gate, ask before merge), 6.5 report the merge state and say so explicitly when the PR is unmerged (folders not yet on the default branch).
- Phase 4 no longer claims a separate process: it points at the folder and gives a fallback if it's absent, and explicitly forbids writing "a separate process" into issue bodies.
- Phase 5 confirmation now covers the docs and asks to create milestones, issues and vendor the listed folders.
- Old Phase 6 renumbered to Phase 7 throughout (incl. 7.1.1 Forgejo fail-fast dependency and the 7.4 backfill exemption).
- Preconditions add Node ≥ 20.19 + outbound network access, with an abort rather than a false promise.
- Error Handling + Quick Reference updated: a failed fetch aborts before any issue is created; an unmerged docs PR is reported as not-yet-on-the-default-branch.
package.json — npm test now also runs skills/create-issues/scripts/*.test.mjs.
README.md — create-issues row rewritten (milestones → steps, self-contained bodies, vendors dependency docs into docs/reference/<slug>/, shipped as a docs PR); repo-layout note now mentions create-issues/ shipping scripts/fetch-reference-docs.mjs.
CHANGELOG.md — new Fixed entry documenting the phantom-download problem and the fix.

Notes for the reviewer

The docs-PR shipping flow is delegated to the existing commit-docs skill, so this PR introduces no new merge behaviour of its own.
The script intentionally never summarises or rewrites a page — verbatim vendoring is what makes _meta.json's sha256 meaningful as a staleness check.

Closes #289

## What & why The `create-issues` skill's **Documentation Required** section has been naming `docs/reference/<slug>/` folders and telling implementers that *"A separate process downloads these"*. That process never existed — it was pi-loop's still-unimplemented research stage (ADR-014) — so the folders were never created. The cost was measurable: - 99 `.pi-loop/runs/*` directories reference `docs/reference/...` paths - 57 shipped issue bodies carry the literal "A separate process downloads" sentence - `missing-docs` is the second-largest friction category (35 entries) - Implementers recorded the absence in findings (`extensions/vision/findings.md`: *"the separate download process did not run"*; `extensions/penpot/findings.md`: *"the `docs/reference/` directory does not exist"*), and hand-authored fixtures from schemas to compensate This change makes `create-issues` own the download end to end: a new **Phase 6** turns the researched URLs into a JSON manifest, vendors each page verbatim with a dependency-free fetcher, and ships the tree through the `commit-docs` workflow — so the folders exist on the default branch before the issues that reference them are implemented. ## Per-file breakdown ### Added - **`skills/create-issues/scripts/fetch-reference-docs.mjs`** — dependency-free (Node ≥ 20.19 built-in `fetch`) reference-doc fetcher. - Reads a `{ groups: [{ slug, name, version, sources: [{ url, summary }] }] }` manifest; `--out`, `--dry-run`, `--refresh`, `--max-bytes` flags. - **All-or-nothing:** fetches every source for every group into memory before writing a single file, so a failed fetch leaves no partial tree and exits non-zero. - Writes pages **verbatim** to `docs/reference/<slug>/<file>`, with provenance in `_meta.json` (`url`, `file`, `contentType`, `bytes`, `sha256`, `summary`) and a regenerated `docs/reference/README.md` index. - Idempotent: re-runs are no-ops unless a group's `version` or URL set changed (`--refresh` forces); a shrunken URL set removes orphaned files. - Guards: only `http(s)` URLs accepted; bodies capped at 5 MB by default; a source cannot overwrite the provenance file. - **`skills/create-issues/scripts/fetch-reference-docs.test.mjs`** — 27-case suite covering manifest validation, dry-run (no network/writes), verbatim writes + provenance + index rendering, orphan removal, all-or-nothing failure, version/URL-set staleness, `--refresh`, the provenance-overwrite guard, and bad-args/bad-manifest exit codes. Verified locally: **27 pass / 0 fail**. ### Modified - **`skills/create-issues/SKILL.md`** — the substantive change: - **New Phase 6: Vendor the Reference Docs** — 6.1 write the manifest (one group per slug, version as the staleness gate, GitHub-first URLs), 6.2 `--dry-run` preview, 6.3 fetch verbatim, 6.4 ship via the `commit-docs` workflow (date-driven branch, CI gate, ask before merge), 6.5 report the merge state and say so explicitly when the PR is unmerged (folders not yet on the default branch). - **Phase 4** no longer claims a separate process: it points at the folder and gives a fallback if it's absent, and explicitly forbids writing "a separate process" into issue bodies. - **Phase 5** confirmation now covers the docs and asks to create milestones, issues **and** vendor the listed folders. - Old Phase 6 renumbered to **Phase 7** throughout (incl. 7.1.1 Forgejo fail-fast dependency and the 7.4 backfill exemption). - **Preconditions** add Node ≥ 20.19 + outbound network access, with an abort rather than a false promise. - **Error Handling** + **Quick Reference** updated: a failed fetch aborts before any issue is created; an unmerged docs PR is reported as not-yet-on-the-default-branch. - **`package.json`** — `npm test` now also runs `skills/create-issues/scripts/*.test.mjs`. - **`README.md`** — `create-issues` row rewritten (milestones → steps, self-contained bodies, vendors dependency docs into `docs/reference/<slug>/`, shipped as a docs PR); repo-layout note now mentions `create-issues/` shipping `scripts/fetch-reference-docs.mjs`. - **`CHANGELOG.md`** — new **Fixed** entry documenting the phantom-download problem and the fix. ## Notes for the reviewer - The docs-PR shipping flow is delegated to the existing `commit-docs` skill, so this PR introduces no new merge behaviour of its own. - The script intentionally never summarises or rewrites a page — verbatim vendoring is what makes `_meta.json`'s `sha256` meaningful as a staleness check. Closes #289

david added the

bug

label

2026-10-06 11:07:24 +00:00

david added 1 commit

2026-10-06 11:07:24 +00:00