david/pi-extensions-and-skills

Watch

Fork 0

Implement extensions/vision/src/tool.ts (the vision ToolDefinition) #258

New issue

Closed

opened 2026-09-17 03:34:09 +00:00 by david · 1 comment

david commented

2026-09-17 03:34:09 +00:00

Owner

Summary

Create extensions/vision/src/tool.ts: the vision tool definition — its TypeBox parameter schema, name/label/description, prompt metadata, and the execute() that validates images, calls the client, maps usage, and handles the response edge cases.

Background

Depends on: #253, #255, #256, #257

This is the module the LLM actually sees. It owns three things that the lower layers deliberately do not:

The public contract — prompt, images, optional detail, optional thinking — plus the prompt metadata (description, promptSnippet, promptGuidelines) that tells the model when to reach for this tool. promptGuidelines bullets are appended flat to pi's guidelines with no tool-name prefix, so each bullet must name vision explicitly (pinned pi docs).
The assembly — resolve the effective thinking value (per-call param, else config.defaultThinking), validate images against ctx.cwd, build the body, call the client, map usage, and shape details.
The response edge cases — a model that returns only reasoning (content empty) must not silently return an empty string to the LLM.

Every failure must be thrown as a ToolError; a raw Error from a lower layer still needs to be wrapped by a final catch in category unexpected. pi marks a tool failed only when execute() throws.

StringEnum is deliberately not used: this repo's precedent is Type.Union([Type.Literal(...)]) (see extensions/forgejo/src/index.ts), and the plan pins that choice.

Documentation Required

A separate process downloads these into the listed folders before this issue is implemented. Check the folder for the actual reference material before starting.

docs/reference/pi-coding-agent/

Local package docs: node_modules/@earendil-works/pi-coding-agent/docs/extensions.md — the ToolDefinition contract: name/label/description/parameters/execute(toolCallId, params, signal, onUpdate, ctx), promptSnippet, promptGuidelines, the flat-guidelines rule ("Each guideline must name the tool it refers to"), the return shape (content, details, usage), and "Signaling errors" (throw to fail).
Same file, the "Usage accounting" section — usage returned from execute is persisted on the tool result.

docs/reference/typebox/

https://github.com/sinclairzx81/typebox — Type.Object, Type.String, Type.Array, Type.Optional, Type.Union, Type.Literal, and Type.Static for the inferred params type. This is the typebox v1 package (sinclairzx81) already in package.json; import from "typebox" like the other extensions do.

docs/reference/deepseek-api/

https://api-docs.deepseek.com/api/create-chat-completion/ — choices[0].message.content, reasoning_content, finish_reason, and the usage fields.
https://api-docs.deepseek.com/guides/thinking_mode/ — thinking is enabled by default and reasoning tokens count toward the output cap; this is what the empty-content/length errors must explain.

Implementation Details

Public surface:

export interface VisionToolDetails {
  model: string;
  thinking: boolean;
  detail?: string;
  images: { path: string; mime: string; width: number; height: number; bytes: number }[];
  durationMs: number;
  finishReason: string;
}

export function visionTool(client: VisionClient, config: VisionConfig): ToolDefinition;

Parameters (Type.Object, repo precedent Type.Union([Type.Literal(...)])):

{
  prompt: Type.String({ description: "What to determine from the image(s)." }),
  images: Type.Array(Type.String(), {
    description: "Local image file paths (absolute or relative to the working directory).",
    minItems: 1,
  }),
  detail: Type.Optional(Type.Union([
    Type.Literal("low"), Type.Literal("high"),
    Type.Literal("original"), Type.Literal("auto"),
  ], { description: "Image processing detail level. Omit for the API default." })),
  thinking: Type.Optional(Type.Boolean({
    description: "Enable the model's thinking mode for a harder question. Default false; retry with true if the first answer is insufficient.",
  })),
}

Definition metadata:

name: "vision", label: "Vision".
description: names the tool, states it sends local images to deepseek-flash and returns text, and says to use it when the current model cannot read an image.
promptSnippet: "Read or analyze local image files with DeepSeek vision".
promptGuidelines: an explicit bullet beginning "Use vision when …" that names the tool (guidelines are un-prefixed).

execute(toolCallId, params, signal, onUpdate, ctx):

resolve thinking = params.thinking ?? config.defaultThinking;
await loadAndValidateImages(params.images, ctx.cwd);
buildRequestBody({ prompt, images, detail: params.detail, thinking }, config);
await client.complete(body, signal) (pi may pass signal: undefined; the client still applies the deadline);
map usage with mapUsage(response.usage);
return { content: [{ type: "text", text }], details, usage }.

Response handling (order matters):

Non-empty message.content ⇒ return it.
Empty content with finish_reason === "length" ⇒ ToolError (response) whose message mentions VISION_MAX_TOKENS and that reasoning tokens count toward the cap when thinking is on.
Empty content otherwise ⇒ ToolError (response) mentioning that the model may have returned only reasoning and the call can be retried with thinking: true.
reasoning_content is discarded (never returned to the LLM).
Wrap the whole execute body in a try/catch: re-throw a ToolError unchanged, wrap anything else in new ToolError(..., "unexpected"). No raw stack trace reaches the model.

details is populated for every successful call: model, effective thinking, detail (only when supplied), the per-image metadata (path/mime/width/height/bytes, no data URL), durationMs (measure around the client call or the whole execute — pick one and document it), and finishReason.

Test file extensions/vision/src/tool.test.ts (TDD, fake VisionClient — never the network):

schema/name/label/description/promptSnippet/promptGuidelines assertions, including that the guidelines bullet names vision.
success: returns text + details + mapped usage through a fake client; details.thinking reflects the param and, when the param is omitted, config.defaultThinking.
validation failure (bad path) throws ToolError with category === "image" and the client is never called.
empty response ⇒ ToolError (response).
finish_reason: "length" with empty content ⇒ ToolError mentioning VISION_MAX_TOKENS.
abort: the fake client rejects/never resolves and the passed-in signal is aborted ⇒ ToolError (aborted) without leaking a raw error.
an unexpected throw from the fake client ⇒ ToolError (unexpected).

Acceptance Criteria

The tool registers as vision/Vision with a description, promptSnippet, and a promptGuidelines bullet that names the tool.
The parameter schema accepts prompt, a non-empty images array, and optional detail/thinking with the documented descriptions.
execute resolves thinking from the param with config.defaultThinking as the fallback and passes it through to the request body.
On success the tool returns the model text plus details (model, thinking, detail, per-image metadata, durationMs, finishReason) and a mapped usage.
Empty content with finish_reason: "length" raises a ToolError naming VISION_MAX_TOKENS; empty content otherwise raises a response ToolError.
Every thrown error is a ToolError, including the unexpected wrapper; reasoning_content never reaches the LLM.
extensions/vision/src/tool.test.ts passes with node --test extensions/vision/src/tool.test.ts and uses only a fake client.

Test Plan

cd /Users/david/Projects/pi-extensions-and-skills
node --test extensions/vision/src/tool.test.ts

Expected: all tests pass with no network access and no real image files required beyond temp fixtures.

Manual shape inspection (no network — uses a fake client):

node --experimental-strip-types -e "
import('./extensions/vision/src/tool.ts').then(({ visionTool }) => {
  const client = { complete: async () => ({ choices: [{ message: { content: 'ok' }, finish_reason: 'stop' }] }) };
  const config = { apiKey: 'x', model: 'deepseek-flash', baseUrl: 'https://api.deepseek.com', maxTokens: 8000, timeoutMs: 1000, defaultThinking: false };
  const tool = visionTool(client, config);
  console.log(tool.name, tool.label, Object.keys(tool.parameters.properties));
})
"

Expected: vision Vision [ 'prompt', 'images', 'detail', 'thinking' ].

## Summary Create `extensions/vision/src/tool.ts`: the `vision` tool definition — its TypeBox parameter schema, name/label/description, prompt metadata, and the `execute()` that validates images, calls the client, maps usage, and handles the response edge cases. ## Background **Depends on:** #253, #255, #256, #257 This is the module the LLM actually sees. It owns three things that the lower layers deliberately do not: 1. **The public contract** — `prompt`, `images`, optional `detail`, optional `thinking` — plus the prompt metadata (`description`, `promptSnippet`, `promptGuidelines`) that tells the model when to reach for this tool. `promptGuidelines` bullets are appended flat to pi's guidelines with no tool-name prefix, so each bullet must name `vision` explicitly (pinned pi docs). 2. **The assembly** — resolve the effective `thinking` value (per-call param, else `config.defaultThinking`), validate images against `ctx.cwd`, build the body, call the client, map usage, and shape `details`. 3. **The response edge cases** — a model that returns only reasoning (`content` empty) must not silently return an empty string to the LLM. Every failure must be thrown as a `ToolError`; a raw `Error` from a lower layer still needs to be wrapped by a final `catch` in category `unexpected`. pi marks a tool failed only when `execute()` throws. `StringEnum` is deliberately **not** used: this repo's precedent is `Type.Union([Type.Literal(...)])` (see `extensions/forgejo/src/index.ts`), and the plan pins that choice. ## Documentation Required A separate process downloads these into the listed folders before this issue is implemented. Check the folder for the actual reference material before starting. **`docs/reference/pi-coding-agent/`** - Local package docs: `node_modules/@earendil-works/pi-coding-agent/docs/extensions.md` — the `ToolDefinition` contract: `name`/`label`/`description`/`parameters`/`execute(toolCallId, params, signal, onUpdate, ctx)`, `promptSnippet`, `promptGuidelines`, the flat-guidelines rule ("Each guideline must name the tool it refers to"), the return shape (`content`, `details`, `usage`), and "Signaling errors" (throw to fail). - Same file, the "Usage accounting" section — `usage` returned from `execute` is persisted on the tool result. **`docs/reference/typebox/`** - https://github.com/sinclairzx81/typebox — `Type.Object`, `Type.String`, `Type.Array`, `Type.Optional`, `Type.Union`, `Type.Literal`, and `Type.Static` for the inferred params type. This is the `typebox` v1 package (sinclairzx81) already in `package.json`; import from `"typebox"` like the other extensions do. **`docs/reference/deepseek-api/`** - https://api-docs.deepseek.com/api/create-chat-completion/ — `choices[0].message.content`, `reasoning_content`, `finish_reason`, and the `usage` fields. - https://api-docs.deepseek.com/guides/thinking_mode/ — thinking is enabled by default and reasoning tokens count toward the output cap; this is what the empty-content/length errors must explain. ## Implementation Details Public surface: ```ts export interface VisionToolDetails { model: string; thinking: boolean; detail?: string; images: { path: string; mime: string; width: number; height: number; bytes: number }[]; durationMs: number; finishReason: string; } export function visionTool(client: VisionClient, config: VisionConfig): ToolDefinition; ``` Parameters (`Type.Object`, repo precedent `Type.Union([Type.Literal(...)])`): ```ts { prompt: Type.String({ description: "What to determine from the image(s)." }), images: Type.Array(Type.String(), { description: "Local image file paths (absolute or relative to the working directory).", minItems: 1, }), detail: Type.Optional(Type.Union([ Type.Literal("low"), Type.Literal("high"), Type.Literal("original"), Type.Literal("auto"), ], { description: "Image processing detail level. Omit for the API default." })), thinking: Type.Optional(Type.Boolean({ description: "Enable the model's thinking mode for a harder question. Default false; retry with true if the first answer is insufficient.", })), } ``` Definition metadata: - `name: "vision"`, `label: "Vision"`. - `description`: names the tool, states it sends local images to `deepseek-flash` and returns text, and says to use it when the current model cannot read an image. - `promptSnippet: "Read or analyze local image files with DeepSeek vision"`. - `promptGuidelines`: an explicit bullet beginning "Use vision when …" that names the tool (guidelines are un-prefixed). `execute(toolCallId, params, signal, onUpdate, ctx)`: 1. resolve `thinking` = `params.thinking ?? config.defaultThinking`; 2. `await loadAndValidateImages(params.images, ctx.cwd)`; 3. `buildRequestBody({ prompt, images, detail: params.detail, thinking }, config)`; 4. `await client.complete(body, signal)` (pi may pass `signal: undefined`; the client still applies the deadline); 5. map usage with `mapUsage(response.usage)`; 6. return `{ content: [{ type: "text", text }], details, usage }`. Response handling (order matters): - Non-empty `message.content` ⇒ return it. - Empty `content` **with** `finish_reason === "length"` ⇒ `ToolError` (`response`) whose message mentions `VISION_MAX_TOKENS` and that reasoning tokens count toward the cap when thinking is on. - Empty `content` otherwise ⇒ `ToolError` (`response`) mentioning that the model may have returned only reasoning and the call can be retried with `thinking: true`. - `reasoning_content` is discarded (never returned to the LLM). - Wrap the whole `execute` body in a `try/catch`: re-throw a `ToolError` unchanged, wrap anything else in `new ToolError(..., "unexpected")`. No raw stack trace reaches the model. `details` is populated for every successful call: `model`, effective `thinking`, `detail` (only when supplied), the per-image metadata (path/mime/width/height/bytes, no data URL), `durationMs` (measure around the client call or the whole execute — pick one and document it), and `finishReason`. Test file `extensions/vision/src/tool.test.ts` (TDD, fake `VisionClient` — never the network): - schema/name/label/description/promptSnippet/promptGuidelines assertions, including that the guidelines bullet names `vision`. - success: returns text + `details` + mapped `usage` through a fake client; `details.thinking` reflects the param and, when the param is omitted, `config.defaultThinking`. - validation failure (bad path) throws `ToolError` with `category === "image"` and the client is never called. - empty response ⇒ `ToolError` (`response`). - `finish_reason: "length"` with empty content ⇒ `ToolError` mentioning `VISION_MAX_TOKENS`. - abort: the fake client rejects/never resolves and the passed-in signal is aborted ⇒ `ToolError` (`aborted`) without leaking a raw error. - an unexpected throw from the fake client ⇒ `ToolError` (`unexpected`). ## Acceptance Criteria - [ ] The tool registers as `vision`/`Vision` with a description, `promptSnippet`, and a `promptGuidelines` bullet that names the tool. - [ ] The parameter schema accepts `prompt`, a non-empty `images` array, and optional `detail`/`thinking` with the documented descriptions. - [ ] `execute` resolves `thinking` from the param with `config.defaultThinking` as the fallback and passes it through to the request body. - [ ] On success the tool returns the model text plus `details` (model, thinking, detail, per-image metadata, durationMs, finishReason) and a mapped `usage`. - [ ] Empty content with `finish_reason: "length"` raises a `ToolError` naming `VISION_MAX_TOKENS`; empty content otherwise raises a `response` `ToolError`. - [ ] Every thrown error is a `ToolError`, including the `unexpected` wrapper; `reasoning_content` never reaches the LLM. - [ ] `extensions/vision/src/tool.test.ts` passes with `node --test extensions/vision/src/tool.test.ts` and uses only a fake client. ## Test Plan ```bash cd /Users/david/Projects/pi-extensions-and-skills node --test extensions/vision/src/tool.test.ts ``` Expected: all tests pass with no network access and no real image files required beyond temp fixtures. Manual shape inspection (no network — uses a fake client): ```bash node --experimental-strip-types -e " import('./extensions/vision/src/tool.ts').then(({ visionTool }) => { const client = { complete: async () => ({ choices: [{ message: { content: 'ok' }, finish_reason: 'stop' }] }) }; const config = { apiKey: 'x', model: 'deepseek-flash', baseUrl: 'https://api.deepseek.com', maxTokens: 8000, timeoutMs: 1000, defaultThinking: false }; const tool = visionTool(client, config); console.log(tool.name, tool.label, Object.keys(tool.parameters.properties)); }) " ``` Expected: `vision Vision [ 'prompt', 'images', 'detail', 'thinking' ]`.

david added this to the Vision: M5 - Tool definition milestone

2026-09-17 03:34:09 +00:00

david added the

feature

label

2026-09-17 03:34:09 +00:00

david self-assigned this

2026-09-17 03:34:09 +00:00

david referenced this issue

2026-09-17 03:34:28 +00:00

Implement extensions/vision/index.ts (factory, registration, unconfigured no-op) #260

david referenced this issue