Implement extensions/vision/src/client.ts (request build, fetch, parse, error mapping) #257

New issue

Closed

opened 2026-09-17 03:33:38 +00:00 by david · 1 comment

david commented

2026-09-17 03:33:38 +00:00

Owner

Summary

Create extensions/vision/src/client.ts: build the OpenAI-compatible chat-completions body for a vision call, POST it to DeepSeek with a hard deadline, parse the response, and translate every HTTP, network, timeout, abort, and parse failure into a typed ToolError with an actionable message.

Background

Depends on: #253, #255, #256

This is the only module in the extension that talks to the network. It is deliberately split into a pure body builder (buildRequestBody) and an injectable client (createHttpVisionClient(config, fetchImpl)), so the whole request/response surface — URL, headers, exact JSON body, every error branch — is testable with a fake fetch and never needs a real key or the network.

Two behaviours matter beyond "make a request":

The thinking toggle is always sent explicitly. DeepSeek defaults thinking mode to enabled (pinned docs), so omitting the field would make the model's default leak into the tool's documented thinking: false default. Sending {"type":"disabled"} when the flag is false overrides it deterministically.
The deadline is independent of pi's cancellation signal. pi may pass signal: undefined, but a wall-clock timeout (VISION_TIMEOUT, default 120 s) must still apply, so the two signals are combined with AbortSignal.any and the resulting abort is classified as either timeout (deadline) or aborted (pi's signal).

Documentation Required

A separate process downloads these into the listed folders before this issue is implemented. Check the folder for the actual reference material before starting.

docs/reference/deepseek-api/

https://api-docs.deepseek.com/api/create-chat-completion/ — the request fields (model, max_tokens, thinking, messages, content blocks, image_url.detail) and the response envelope (choices[0].message.content, finish_reason, usage).
https://api-docs.deepseek.com/guides/vision/ — the image_url block shape (base64 data URL), the detail values (low/high/original/auto), and the 48 MiB body limit.
https://api-docs.deepseek.com/guides/thinking_mode/ — {"thinking":{"type":"enabled|disabled"}} and the fact that thinking mode is enabled by default.
https://api-docs.deepseek.com/quick_start/error_codes/ — the documented status codes (400, 401, 402, 422, 429, 500, 503) and their causes, used to shape the hints.

docs/reference/nodejs/

https://nodejs.org/api/globals.html#fetch — the built-in fetch (no dependency needed).
https://nodejs.org/api/globals.html#abortsignaltimeoutdelay — AbortSignal.timeout(ms).
https://nodejs.org/api/globals.html#abortsignalanysignals — AbortSignal.any([...]) for combining pi's signal with the deadline.
https://nodejs.org/api/errors.html#abortstop — recognizing an abort (check the signal's reason/aborted, not the message string).

Implementation Details

Public surface:

export interface ChatRequestBody { /* the exact request JSON below */ }

export interface ChatCompletionResponse {
  choices: {
    message: { content?: string | null; reasoning_content?: string | null };
    finish_reason?: string;
  }[];
  usage?: DeepSeekUsage;
}

export interface VisionClient {
  complete(body: ChatRequestBody, signal: AbortSignal | undefined): Promise<ChatCompletionResponse>;
}

export function buildRequestBody(
  params: { prompt: string; images: ValidatedImage[]; detail?: string; thinking: boolean },
  config: VisionConfig,
): ChatRequestBody;

export function createHttpVisionClient(
  config: VisionConfig,
  fetchImpl?: typeof fetch, // defaults to global fetch
): VisionClient;

The body must be exactly:

{
  "model": "<config.model>",
  "max_tokens": <config.maxTokens>,
  "thinking": { "type": "enabled" | "disabled" },
  "messages": [
    {
      "role": "user",
      "content": [
        { "type": "text", "text": "<prompt>" },
        { "type": "image_url", "image_url": { "url": "data:image/png;base64,...", "detail": "low" } }
      ]
    }
  ]
}

detail is present inside image_url only when the caller supplied it (no null/undefined key).
Text block first, then one image block per ValidatedImage, preserving order.
thinking is always {"type":"enabled"} or {"type":"disabled"}.

Transport:

POST ${config.baseUrl}/chat/completions with headers Content-Type: application/json and Authorization: Bearer ${config.apiKey}.
Deadline: AbortSignal.any([toolSignal, AbortSignal.timeout(config.timeoutMs)]), where toolSignal is included only when defined (pi may pass undefined). Pass the combined signal to fetchImpl.
Non-2xx: read the body, try to parse DeepSeek's { "error": { "message", "type", "code" } }, and throw ToolError with category http, message including the status and the API message (fall back to the raw text when the body is not JSON). Hints by status: 401 ⇒ auth hint (set DEEPSEEK_API_KEY / check the key); 429 ⇒ rate-limit hint; 5xx ⇒ "DeepSeek server error, retry" hint. The category for every non-2xx stays http; the 401/429 hints appear in the message (they do not change the category).
Network rejection (DNS/TLS/connection reset) ⇒ ToolError category network.
Deadline fired ⇒ ToolError category timeout, message naming VISION_TIMEOUT and its current value.
pi's signal fired ⇒ ToolError category aborted.
Classify by inspecting the combined signal / its reason, not by matching error message text (Node throws a DOMException or an AbortError whose wording is not stable).
JSON.parse failure ⇒ ToolError category response.
Missing/empty choices[0].message ⇒ ToolError category response.
Never let a raw fetch/parse exception escape; every path throws ToolError.

Test file extensions/vision/src/client.test.ts (TDD, injected fake fetch, no network):

buildRequestBody asserts method/URL/headers and the exact JSON body: thinking enabled vs disabled, detail present vs absent, text block first, image order preserved.
Fake fetch returning 200 with a valid envelope ⇒ parsed response returned.
400/401/429/500 with DeepSeek error bodies ⇒ ToolError with category === "http" and a message containing the status and API message; assert the 401/429/5xx hints appear.
A non-2xx with a non-JSON body ⇒ http error, no crash.
fetch rejecting (simulated network failure) ⇒ network.
A never-resolving fetch plus a tiny timeoutMs ⇒ timeout.
Aborting the passed-in signal ⇒ aborted.
Malformed JSON body on a 200 ⇒ response; missing choices ⇒ response.

Acceptance Criteria

buildRequestBody produces exactly the JSON shape above: thinking always explicit, detail only when supplied, text block before image blocks, image order preserved.
The request goes to ${baseUrl}/chat/completions as a POST with Content-Type and Bearer auth headers.
The deadline applies even when the caller passes signal: undefined, and combines with pi's signal when present.
Every failure path throws ToolError with the documented category: http for every non-2xx (with the 401/429/5xx hints in the message), network, timeout, aborted, and response for parse/shape failures — each with an actionable message.
No raw fetch/parse exception ever escapes complete().
extensions/vision/src/client.test.ts covers the exact body, the 2xx parse, each error branch, and the timeout/abort distinction, using an injected fake fetch.
node --test extensions/vision/src/client.test.ts passes and no test touches the network.

Test Plan

cd /Users/david/Projects/pi-extensions-and-skills
node --test extensions/vision/src/client.test.ts

Expected: all tests pass with zero network access (an injected fake fetch is the only transport).

Optional live sanity check once a key exists in .env (not required for this issue and not part of the automated suite):

node --experimental-strip-types -e "
import('./extensions/vision/src/client.ts').then(async (c) => {
  const client = c.createHttpVisionClient({ apiKey: process.env.DEEPSEEK_API_KEY, model: 'deepseek-flash', baseUrl: 'https://api.deepseek.com', maxTokens: 16, timeoutMs: 20000, defaultThinking: false });
  const body = c.buildRequestBody({ prompt: 'Say hi', images: [], thinking: false }, { apiKey: 'x', model: 'deepseek-flash', baseUrl: 'https://api.deepseek.com', maxTokens: 16, timeoutMs: 20000, defaultThinking: false });
  console.log(await client.complete(body, undefined));
})
"

Expected: a response envelope with choices[0].message.content; a missing key yields a ToolError with an auth hint rather than a stack trace.

## Summary Create `extensions/vision/src/client.ts`: build the OpenAI-compatible chat-completions body for a `vision` call, POST it to DeepSeek with a hard deadline, parse the response, and translate every HTTP, network, timeout, abort, and parse failure into a typed `ToolError` with an actionable message. ## Background **Depends on:** #253, #255, #256 This is the only module in the extension that talks to the network. It is deliberately split into a pure body builder (`buildRequestBody`) and an injectable client (`createHttpVisionClient(config, fetchImpl)`), so the whole request/response surface — URL, headers, exact JSON body, every error branch — is testable with a fake `fetch` and never needs a real key or the network. Two behaviours matter beyond "make a request": - **The `thinking` toggle is always sent explicitly.** DeepSeek defaults thinking mode to *enabled* (pinned docs), so omitting the field would make the model's default leak into the tool's documented `thinking: false` default. Sending `{"type":"disabled"}` when the flag is false overrides it deterministically. - **The deadline is independent of pi's cancellation signal.** pi may pass `signal: undefined`, but a wall-clock timeout (`VISION_TIMEOUT`, default 120 s) must still apply, so the two signals are combined with `AbortSignal.any` and the resulting abort is classified as either `timeout` (deadline) or `aborted` (pi's signal). ## Documentation Required A separate process downloads these into the listed folders before this issue is implemented. Check the folder for the actual reference material before starting. **`docs/reference/deepseek-api/`** - https://api-docs.deepseek.com/api/create-chat-completion/ — the request fields (`model`, `max_tokens`, `thinking`, `messages`, `content` blocks, `image_url.detail`) and the response envelope (`choices[0].message.content`, `finish_reason`, `usage`). - https://api-docs.deepseek.com/guides/vision/ — the `image_url` block shape (base64 data URL), the `detail` values (`low`/`high`/`original`/`auto`), and the 48 MiB body limit. - https://api-docs.deepseek.com/guides/thinking_mode/ — `{"thinking":{"type":"enabled|disabled"}}` and the fact that thinking mode is enabled by default. - https://api-docs.deepseek.com/quick_start/error_codes/ — the documented status codes (400, 401, 402, 422, 429, 500, 503) and their causes, used to shape the hints. **`docs/reference/nodejs/`** - https://nodejs.org/api/globals.html#fetch — the built-in `fetch` (no dependency needed). - https://nodejs.org/api/globals.html#abortsignaltimeoutdelay — `AbortSignal.timeout(ms)`. - https://nodejs.org/api/globals.html#abortsignalanysignals — `AbortSignal.any([...])` for combining pi's signal with the deadline. - https://nodejs.org/api/errors.html#abortstop — recognizing an abort (check the signal's `reason`/`aborted`, not the message string). ## Implementation Details Public surface: ```ts export interface ChatRequestBody { /* the exact request JSON below */ } export interface ChatCompletionResponse { choices: { message: { content?: string | null; reasoning_content?: string | null }; finish_reason?: string; }[]; usage?: DeepSeekUsage; } export interface VisionClient { complete(body: ChatRequestBody, signal: AbortSignal | undefined): Promise<ChatCompletionResponse>; } export function buildRequestBody( params: { prompt: string; images: ValidatedImage[]; detail?: string; thinking: boolean }, config: VisionConfig, ): ChatRequestBody; export function createHttpVisionClient( config: VisionConfig, fetchImpl?: typeof fetch, // defaults to global fetch ): VisionClient; ``` The body must be exactly: ```json { "model": "<config.model>", "max_tokens": <config.maxTokens>, "thinking": { "type": "enabled" | "disabled" }, "messages": [ { "role": "user", "content": [ { "type": "text", "text": "<prompt>" }, { "type": "image_url", "image_url": { "url": "data:image/png;base64,...", "detail": "low" } } ] } ] } ``` - `detail` is present inside `image_url` **only** when the caller supplied it (no `null`/`undefined` key). - Text block first, then one image block per `ValidatedImage`, preserving order. - `thinking` is always `{"type":"enabled"}` or `{"type":"disabled"}`. Transport: - `POST ${config.baseUrl}/chat/completions` with headers `Content-Type: application/json` and `Authorization: Bearer ${config.apiKey}`. - Deadline: `AbortSignal.any([toolSignal, AbortSignal.timeout(config.timeoutMs)])`, where `toolSignal` is included only when defined (pi may pass `undefined`). Pass the combined signal to `fetchImpl`. - Non-2xx: read the body, try to parse DeepSeek's `{ "error": { "message", "type", "code" } }`, and throw `ToolError` with category `http`, message including the status and the API message (fall back to the raw text when the body is not JSON). Hints by status: 401 ⇒ auth hint (`set DEEPSEEK_API_KEY` / check the key); 429 ⇒ rate-limit hint; 5xx ⇒ "DeepSeek server error, retry" hint. The category for every non-2xx stays `http`; the 401/429 hints appear in the message (they do not change the category). - Network rejection (DNS/TLS/connection reset) ⇒ `ToolError` category `network`. - Deadline fired ⇒ `ToolError` category `timeout`, message naming `VISION_TIMEOUT` and its current value. - pi's signal fired ⇒ `ToolError` category `aborted`. - Classify by inspecting the combined signal / its reason, not by matching error message text (Node throws a `DOMException` or an `AbortError` whose wording is not stable). - `JSON.parse` failure ⇒ `ToolError` category `response`. - Missing/empty `choices[0].message` ⇒ `ToolError` category `response`. - Never let a raw fetch/parse exception escape; every path throws `ToolError`. Test file `extensions/vision/src/client.test.ts` (TDD, injected fake `fetch`, no network): - `buildRequestBody` asserts method/URL/headers and the **exact** JSON body: thinking enabled vs disabled, `detail` present vs absent, text block first, image order preserved. - Fake `fetch` returning 200 with a valid envelope ⇒ parsed response returned. - 400/401/429/500 with DeepSeek error bodies ⇒ `ToolError` with `category === "http"` and a message containing the status and API message; assert the 401/429/5xx hints appear. - A non-2xx with a non-JSON body ⇒ `http` error, no crash. - `fetch` rejecting (simulated network failure) ⇒ `network`. - A never-resolving `fetch` plus a tiny `timeoutMs` ⇒ `timeout`. - Aborting the passed-in signal ⇒ `aborted`. - Malformed JSON body on a 200 ⇒ `response`; missing `choices` ⇒ `response`. ## Acceptance Criteria - [ ] `buildRequestBody` produces exactly the JSON shape above: `thinking` always explicit, `detail` only when supplied, text block before image blocks, image order preserved. - [ ] The request goes to `${baseUrl}/chat/completions` as a `POST` with `Content-Type` and `Bearer` auth headers. - [ ] The deadline applies even when the caller passes `signal: undefined`, and combines with pi's signal when present. - [ ] Every failure path throws `ToolError` with the documented category: `http` for every non-2xx (with the 401/429/5xx hints in the message), `network`, `timeout`, `aborted`, and `response` for parse/shape failures — each with an actionable message. - [ ] No raw fetch/parse exception ever escapes `complete()`. - [ ] `extensions/vision/src/client.test.ts` covers the exact body, the 2xx parse, each error branch, and the timeout/abort distinction, using an injected fake `fetch`. - [ ] `node --test extensions/vision/src/client.test.ts` passes and no test touches the network. ## Test Plan ```bash cd /Users/david/Projects/pi-extensions-and-skills node --test extensions/vision/src/client.test.ts ``` Expected: all tests pass with zero network access (an injected fake `fetch` is the only transport). Optional live sanity check once a key exists in `.env` (not required for this issue and not part of the automated suite): ```bash node --experimental-strip-types -e " import('./extensions/vision/src/client.ts').then(async (c) => { const client = c.createHttpVisionClient({ apiKey: process.env.DEEPSEEK_API_KEY, model: 'deepseek-flash', baseUrl: 'https://api.deepseek.com', maxTokens: 16, timeoutMs: 20000, defaultThinking: false }); const body = c.buildRequestBody({ prompt: 'Say hi', images: [], thinking: false }, { apiKey: 'x', model: 'deepseek-flash', baseUrl: 'https://api.deepseek.com', maxTokens: 16, timeoutMs: 20000, defaultThinking: false }); console.log(await client.complete(body, undefined)); }) " ``` Expected: a response envelope with `choices[0].message.content`; a missing key yields a `ToolError` with an auth hint rather than a stack trace.

david added this to the Vision: M4 - HTTP client milestone

2026-09-17 03:33:38 +00:00

david added the

feature

label

2026-09-17 03:33:38 +00:00

david self-assigned this

2026-09-17 03:33:38 +00:00

david referenced this issue

2026-09-17 03:34:09 +00:00

Implement extensions/vision/src/tool.ts (the vision ToolDefinition) #258

david referenced this issue