Verify the vision tool live against the DeepSeek API #264

New issue

Closed

opened 2026-09-17 03:35:29 +00:00 by david · 1 comment

david commented

2026-09-17 03:35:29 +00:00

Owner

Summary

Run the four live checks from the plan's §15 against the real DeepSeek API with a real DEEPSEEK_API_KEY and real images, and record what actually happened. This is manually triggered and opt-in — it is never part of CI, because it needs the network, a paid key, and real image files.

Background

Depends on: #257, #258, #260, #261

All the unit tests fake fetch, so four behaviours have never been observed against the provider: whether DeepSeek accepts the explicit {"thinking":{"type":"disabled"}} toggle for vision requests, whether thinking mode plus a non-trivial image fits inside VISION_MAX_TOKENS, whether the deepseek-flash model id is accepted as documented, and whether the computed cost is in the right ballpark. The plan's risks section names the first of these as a ship-blocker.

The value of this step is honesty: if a check fails, the issue is not "done" until either the code is fixed or the divergence is documented in extensions/vision/README.md and on this issue.

Do not run this in CI and do not commit a real API key. Use the project .env (which is gitignored) or an exported env var.

Documentation Required

A separate process downloads these into the listed folders before this issue is implemented. Check the folder for the actual reference material before starting.

docs/reference/deepseek-api/

https://api-docs.deepseek.com/guides/vision/ — the documented deepseek-flash model name, the supported formats/limits, and the token accounting for images (auto-resize, ~1024 tokens per image upper bound) used to sanity-check cost.
https://api-docs.deepseek.com/guides/thinking_mode/ — the toggle shapes, that thinking defaults to on, that reasoning_content is returned alongside content, and that reasoning tokens count toward the output cap.
https://api-docs.deepseek.com/quick_start/pricing/ — the current deepseek-flash rates and peak windows, to compare against a logged usage.cost.total.
https://api-docs.deepseek.com/quick_start/error_codes/ — what a rejected parameter or bad key looks like, so a failure is interpreted correctly.

Implementation Details

Prepare a real key (DEEPSEEK_API_KEY in .env or the environment) and at least one real image — a screenshot with text is the most useful, since it exercises OCR and gives a falsifiable prompt.

Run each check and record the exact command, the observed finish_reason, content, usage, and details:

Vision with thinking disabled. Send a real image with thinking: false. Confirm content is populated and finish_reason === "stop".
- If DeepSeek rejects the explicit disabled toggle for vision, fall back to omitting thinking when the caller's effective value is false, update src/client.ts accordingly, adjust its tests, and document the change here and in extensions/vision/README.md.
VISION_MAX_TOKENS with thinking enabled. Call with thinking: true and a large image. Confirm finish_reason !== "length" and completion_tokens_details.reasoning_tokens > 0. If length is hit, raise VISION_MAX_TOKENS and note the value that worked.
Model id. Confirm deepseek-flash is accepted (the documented name; the retired deepseek-v4-flash-vision-exp alias also routes to it). If the id is rejected, capture the error body verbatim.
Cost sanity. Compare usage.cost.total from one request against DeepSeek's usage page / published rates for the same request's token counts. Note the peak/off-peak instant of the call and whether the off-peak half-rate matched.

Also worth observing while live (not required): a multi-image call (2–3 images) returns a coherent comparison, and a 401 with a deliberately wrong key produces the auth hint rather than a stack trace.

Record the results in extensions/vision/README.md under the development/verification section — or in a short extensions/vision/findings.md if the notes are too long for the README — and summarise them in a comment on this issue. State plainly which checks passed, which diverged, and what changed as a result.

Acceptance Criteria

A live call with thinking: false returns populated content and finish_reason === "stop" — or, if the toggle is rejected, src/client.ts, its tests, and the README are updated to omit thinking when false.
A live call with thinking: true and a large image returns finish_reason !== "length" and reasoning_tokens > 0 — or the working VISION_MAX_TOKENS value and the required documentation change are recorded.
deepseek-flash is confirmed accepted, or the rejection error body is captured verbatim.
usage.cost.total is compared against the published rates for the call's token counts, with the peak/off-peak instant noted.
The results (commands, observed finish reasons/usage, divergences) are recorded in extensions/vision/README.md (or findings.md) and summarised on this issue.
No real API key is committed, and nothing about this step runs in CI.

Test Plan

cd /Users/david/Projects/pi-extensions-and-skills
# real key required in .env (gitignored) or the environment
node --experimental-strip-types -e "
import('./extensions/vision/src/config.ts').then(async (cfg) => {
  const { createHttpVisionClient, buildRequestBody } = await import('./extensions/vision/src/client.ts');
  const { loadAndValidateImages } = await import('./extensions/vision/src/images.ts');
  const { mapUsage } = await import('./extensions/vision/src/usage.ts');
  const config = cfg.resolveConfig({ ...cfg.loadEnvFile(), ...process.env });
  if (!config) { console.error('no key'); process.exit(1); }
  const images = await loadAndValidateImages([process.argv[1]], process.cwd());
  const client = createHttpVisionClient(config);
  for (const thinking of [false, true]) {
    const body = buildRequestBody({ prompt: 'Transcribe the text in this image.', images, thinking }, config);
    const res = await client.complete(body, undefined);
    console.log(thinking, res.choices[0].finish_reason, JSON.stringify(mapUsage(res.usage)));
    console.log(res.choices[0].message.content?.slice(0, 200));
  }
})
" /absolute/path/to/image.png

Expected: both calls return text; the thinking: true call reports non-zero reasoning in the mapped usage; the printed finish_reason for both is stop (not length). Time it and note the peak/off-peak hour for the cost comparison.

## Summary Run the four live checks from the plan's §15 against the real DeepSeek API with a real `DEEPSEEK_API_KEY` and real images, and record what actually happened. This is manually triggered and opt-in — it is never part of CI, because it needs the network, a paid key, and real image files. ## Background **Depends on:** #257, #258, #260, #261 All the unit tests fake `fetch`, so four behaviours have never been observed against the provider: whether DeepSeek accepts the explicit `{"thinking":{"type":"disabled"}}` toggle for vision requests, whether thinking mode plus a non-trivial image fits inside `VISION_MAX_TOKENS`, whether the `deepseek-flash` model id is accepted as documented, and whether the computed cost is in the right ballpark. The plan's risks section names the first of these as a ship-blocker. The value of this step is honesty: if a check fails, the issue is not "done" until either the code is fixed or the divergence is documented in `extensions/vision/README.md` and on this issue. Do not run this in CI and do not commit a real API key. Use the project `.env` (which is gitignored) or an exported env var. ## Documentation Required A separate process downloads these into the listed folders before this issue is implemented. Check the folder for the actual reference material before starting. **`docs/reference/deepseek-api/`** - https://api-docs.deepseek.com/guides/vision/ — the documented `deepseek-flash` model name, the supported formats/limits, and the token accounting for images (auto-resize, ~1024 tokens per image upper bound) used to sanity-check cost. - https://api-docs.deepseek.com/guides/thinking_mode/ — the toggle shapes, that thinking defaults to on, that `reasoning_content` is returned alongside `content`, and that reasoning tokens count toward the output cap. - https://api-docs.deepseek.com/quick_start/pricing/ — the current `deepseek-flash` rates and peak windows, to compare against a logged `usage.cost.total`. - https://api-docs.deepseek.com/quick_start/error_codes/ — what a rejected parameter or bad key looks like, so a failure is interpreted correctly. ## Implementation Details Prepare a real key (`DEEPSEEK_API_KEY` in `.env` or the environment) and at least one real image — a screenshot with text is the most useful, since it exercises OCR and gives a falsifiable prompt. Run each check and record the exact command, the observed `finish_reason`, `content`, `usage`, and `details`: 1. **Vision with thinking disabled.** Send a real image with `thinking: false`. Confirm `content` is populated and `finish_reason === "stop"`. - *If DeepSeek rejects the explicit disabled toggle for vision*, fall back to omitting `thinking` when the caller's effective value is `false`, update `src/client.ts` accordingly, adjust its tests, and document the change here and in `extensions/vision/README.md`. 2. **`VISION_MAX_TOKENS` with thinking enabled.** Call with `thinking: true` and a large image. Confirm `finish_reason !== "length"` and `completion_tokens_details.reasoning_tokens > 0`. If `length` is hit, raise `VISION_MAX_TOKENS` and note the value that worked. 3. **Model id.** Confirm `deepseek-flash` is accepted (the documented name; the retired `deepseek-v4-flash-vision-exp` alias also routes to it). If the id is rejected, capture the error body verbatim. 4. **Cost sanity.** Compare `usage.cost.total` from one request against DeepSeek's usage page / published rates for the same request's token counts. Note the peak/off-peak instant of the call and whether the off-peak half-rate matched. Also worth observing while live (not required): a multi-image call (2–3 images) returns a coherent comparison, and a 401 with a deliberately wrong key produces the auth hint rather than a stack trace. Record the results in `extensions/vision/README.md` under the development/verification section — or in a short `extensions/vision/findings.md` if the notes are too long for the README — and summarise them in a comment on this issue. State plainly which checks passed, which diverged, and what changed as a result. ## Acceptance Criteria - [ ] A live call with `thinking: false` returns populated `content` and `finish_reason === "stop"` — or, if the toggle is rejected, `src/client.ts`, its tests, and the README are updated to omit `thinking` when false. - [ ] A live call with `thinking: true` and a large image returns `finish_reason !== "length"` and `reasoning_tokens > 0` — or the working `VISION_MAX_TOKENS` value and the required documentation change are recorded. - [ ] `deepseek-flash` is confirmed accepted, or the rejection error body is captured verbatim. - [ ] `usage.cost.total` is compared against the published rates for the call's token counts, with the peak/off-peak instant noted. - [ ] The results (commands, observed finish reasons/usage, divergences) are recorded in `extensions/vision/README.md` (or `findings.md`) and summarised on this issue. - [ ] No real API key is committed, and nothing about this step runs in CI. ## Test Plan ```bash cd /Users/david/Projects/pi-extensions-and-skills # real key required in .env (gitignored) or the environment node --experimental-strip-types -e " import('./extensions/vision/src/config.ts').then(async (cfg) => { const { createHttpVisionClient, buildRequestBody } = await import('./extensions/vision/src/client.ts'); const { loadAndValidateImages } = await import('./extensions/vision/src/images.ts'); const { mapUsage } = await import('./extensions/vision/src/usage.ts'); const config = cfg.resolveConfig({ ...cfg.loadEnvFile(), ...process.env }); if (!config) { console.error('no key'); process.exit(1); } const images = await loadAndValidateImages([process.argv[1]], process.cwd()); const client = createHttpVisionClient(config); for (const thinking of [false, true]) { const body = buildRequestBody({ prompt: 'Transcribe the text in this image.', images, thinking }, config); const res = await client.complete(body, undefined); console.log(thinking, res.choices[0].finish_reason, JSON.stringify(mapUsage(res.usage))); console.log(res.choices[0].message.content?.slice(0, 200)); } }) " /absolute/path/to/image.png ``` Expected: both calls return text; the `thinking: true` call reports non-zero `reasoning` in the mapped usage; the printed `finish_reason` for both is `stop` (not `length`). Time it and note the peak/off-peak hour for the cost comparison.

david added this to the Vision: M8 - Live verification milestone

2026-09-17 03:35:29 +00:00

david added the

feature

label

2026-09-17 03:35:29 +00:00

david self-assigned this

2026-09-17 03:35:30 +00:00

david referenced this issue from a pull request that will close it,

2026-09-17 12:13:38 +00:00

issue-264: Verify the vision tool live against the DeepSeek API #277

david closed this issue