documentation: add vision extension implementation plan #259

Merged

david merged 1 commit from documentation/2026-09-17/vision-extension-implementation-plan into main

2026-09-17 03:34:29 +00:00

david commented

2026-09-17 03:34:17 +00:00

Owner

What & why

Adds the implementation plan for a new vision extension to the repo, plus an incidental npm lockfile metadata refresh.

extensions/vision/IMPLEMENTATION_PLAN.md captures the output of a design interview that turns a one-shot Moonshot image-understanding CLI (/tmp/cpv/vision.py) into a first-class pi extension: an agent-callable vision tool backed by DeepSeek's deepseek-flash vision model. The plan is written to be executable without re-reading the interview transcript — every decision, contract, and default is stated inline.

This is a planning document only. No extension code, tests, or package.json wiring is included in this PR; the plan's milestones (§14) describe that work as follow-up.

Key decisions recorded in the plan:

Single tool, single turn. vision accepts a prompt, one or more local image paths, an optional detail level, and an optional thinking flag; it returns model text, structured details, and a pi-mapped usage (token counts plus peak/off-peak cost).
Local images only. No http(s) URL or Files API inputs, no caching, no conversation state, no streaming, no image generation/editing, no per-call model override — each call is an independent read-only request that writes nothing to disk.
Fail-fast configuration (§4). src/config.ts loads .env via dotenv.parse without mutating process.env (mirroring mongodb/src/env.ts), with process.env taking precedence. DEEPSEEK_API_KEY absent or empty ⇒ resolveConfig returns null and the extension registers zero tools. Invalid numerics for VISION_MAX_TOKENS / VISION_TIMEOUT throw a ToolError (category config) at load time rather than silently defaulting.
Injectable validation limits (§5) so DeepSeek's image-size/count/format limits can be exercised cheaply in tests.
Typed error contract (§7) — a ToolError with a category, mapped to actionable messages.
Fully offline test plan (§12) covering config, validation, client, usage, error mapping, tool, and factory registration, plus §15 live-verification items that require a real API key and are therefore excluded from CI.
Dependency-ordered milestones (§14) and a risks/maintenance section (§16).

The plan also specifies the intended file layout (§3): index.ts with a registerVisionTools seam, a src/ module per concern, a co-located *.test.ts per module, and a README.md.

Per-file breakdown

Added

extensions/vision/IMPLEMENTATION_PLAN.md (+452)
- §1 Goal, §2 Scope (in/out), §3 proposed file layout
- §4 Configuration table and VisionConfig / resolveConfig contract
- §5 local image validation and ValidationLimits
- §6 OpenAI-compatible request build / fetch / response parse
- §7 ToolError categories and mapping
- §8 DeepSeek usage → pi Usage with peak/off-peak costing
- §9 the vision ToolDefinition
- §10 factory + registerVisionTools registration seam
- §11 packaging and wiring, §12 offline test plan, §13 README outline
- §14 milestones, §15 live verification items, §16 risks and maintenance

Modified

package-lock.json (−1)
- Drops the stale "hasShrinkwrap": true field from the @earendil-works/pi-coding-agent@0.84.1 entry. Metadata-only lockfile churn produced by npm; no dependency versions or integrity hashes change.

Notes

No issue is attached: documentation is a no-issue, date-driven label in this repo's AGENTS.md conventions.
The scope deliberately stops at the plan; review here is about whether the plan is the right design to implement, not about code correctness.

## What & why Adds the implementation plan for a new `vision` extension to the repo, plus an incidental npm lockfile metadata refresh. `extensions/vision/IMPLEMENTATION_PLAN.md` captures the output of a design interview that turns a one-shot Moonshot image-understanding CLI (`/tmp/cpv/vision.py`) into a first-class pi extension: an agent-callable `vision` tool backed by DeepSeek's `deepseek-flash` vision model. The plan is written to be executable without re-reading the interview transcript — every decision, contract, and default is stated inline. This is a **planning document only**. No extension code, tests, or `package.json` wiring is included in this PR; the plan's milestones (§14) describe that work as follow-up. Key decisions recorded in the plan: - **Single tool, single turn.** `vision` accepts a `prompt`, one or more local image paths, an optional `detail` level, and an optional `thinking` flag; it returns model text, structured `details`, and a pi-mapped `usage` (token counts plus peak/off-peak cost). - **Local images only.** No `http(s)` URL or Files API inputs, no caching, no conversation state, no streaming, no image generation/editing, no per-call model override — each call is an independent read-only request that writes nothing to disk. - **Fail-fast configuration** (§4). `src/config.ts` loads `.env` via `dotenv.parse` without mutating `process.env` (mirroring `mongodb/src/env.ts`), with `process.env` taking precedence. `DEEPSEEK_API_KEY` absent or empty ⇒ `resolveConfig` returns `null` and the extension registers **zero tools**. Invalid numerics for `VISION_MAX_TOKENS` / `VISION_TIMEOUT` throw a `ToolError` (category `config`) at load time rather than silently defaulting. - **Injectable validation limits** (§5) so DeepSeek's image-size/count/format limits can be exercised cheaply in tests. - **Typed error contract** (§7) — a `ToolError` with a category, mapped to actionable messages. - **Fully offline test plan** (§12) covering config, validation, client, usage, error mapping, tool, and factory registration, plus §15 live-verification items that require a real API key and are therefore excluded from CI. - **Dependency-ordered milestones** (§14) and a risks/maintenance section (§16). The plan also specifies the intended file layout (§3): `index.ts` with a `registerVisionTools` seam, a `src/` module per concern, a co-located `*.test.ts` per module, and a `README.md`. ## Per-file breakdown ### Added - `extensions/vision/IMPLEMENTATION_PLAN.md` (+452) - §1 Goal, §2 Scope (in/out), §3 proposed file layout - §4 Configuration table and `VisionConfig` / `resolveConfig` contract - §5 local image validation and `ValidationLimits` - §6 OpenAI-compatible request build / fetch / response parse - §7 `ToolError` categories and mapping - §8 DeepSeek usage → pi `Usage` with peak/off-peak costing - §9 the `vision` `ToolDefinition` - §10 factory + `registerVisionTools` registration seam - §11 packaging and wiring, §12 offline test plan, §13 README outline - §14 milestones, §15 live verification items, §16 risks and maintenance ### Modified - `package-lock.json` (−1) - Drops the stale `"hasShrinkwrap": true` field from the `@earendil-works/pi-coding-agent@0.84.1` entry. Metadata-only lockfile churn produced by npm; no dependency versions or integrity hashes change. ## Notes - No issue is attached: `documentation` is a no-issue, date-driven label in this repo's `AGENTS.md` conventions. - The scope deliberately stops at the plan; review here is about whether the plan is the right design to implement, not about code correctness.

david added the

documentation

label

2026-09-17 03:34:17 +00:00

david added 1 commit

2026-09-17 03:34:17 +00:00