Support Claude on Vertex AI (vendor + register vertex-anthropic provider) — Option B1 #27

New issue

david · 2026-07-27T06:37:01Z

david commented

2026-07-27 06:37:01 +00:00

Summary

Add support for running pi-loop's stage sessions on Claude models served via Google Cloud Vertex AI ("Claude on Vertex"), by vendoring the community pi-provider-vertex-anthropic logic into pi-loop and registering it directly on the SDK ModelRuntime. This is opt-in via an environment flag; all existing providers (Gemini via the builtin google-vertex, direct Anthropic, etc.) are unaffected when the flag is off.

This is Option B / Approach B1 from the LLM-provider investigation. See in-repo docs:

docs/plan-B1-vertex-anthropic-provider.md — the detailed implementation plan (source of truth for tasks).
docs/scope-option-B-claude-on-vertex.md — scoping + why B3 (config-only) was ruled out.

Background / motivation

pi-loop uses its own bundled SDK (@earendil-works/pi-coding-agent@0.81.1) and calls ModelRuntime.create() directly in src/stage/services/createPiStageSession.ts. It does not load the extension packages listed in ~/.pi/agent/settings.json, so the custom vertex-anthropic provider used by interactive pi is not available to pi-loop.

Investigation findings (all verified live):

The builtin google-vertex provider works for Gemini (gemini-2.5-pro returns OK) but not Claude: it uses @google/genai generateContentStream (the Gemini generateContent protocol, hardwired to publishers/google). Claude on Vertex needs publishers/anthropic/...:streamRawPredict with the Anthropic Messages body.
The builtin anthropic-messages provider wraps the official @anthropic-ai/sdk calling the direct /v1/messages API with x-api-key; it has no Vertex mode and cannot be config-pointed at the Vertex URL/auth. → B3 (models.json only) is impossible.
The raw Vertex Anthropic endpoint works with our GCP creds:
POST https://aiplatform.googleapis.com/v1/projects/<project>/locations/global/publishers/anthropic/models/claude-opus-4-8:streamRawPredict
body {"anthropic_version":"vertex-2023-10-16","messages":[...],"max_tokens":N,"stream":true} → HTTP 200, standard Anthropic Messages SSE.
Auth: our creds are ADC (~/.config/gcloud/application_default_credentials.json, authorized_user refresh-token flow). The vendored auth.ts chain (service-account JSON → ADC → gcloud CLI) resolves a bearer token.
Working model/region for our project today: location=global + claude-opus-4-8. (us-east5 → 429 quota; us-central1 → 400; pinned @date ids and claude-sonnet-4-5* → 404.)
The SDK allows programmatic registration: ModelRuntime.registerProvider(id, config) accepts a ProviderConfigInput that includes a custom streamSimple — this is exactly why B1 is feasible.
The streaming helpers the vendored code needs (createAssistantMessageEventStream, calculateCost) are exported by @earendil-works/pi-ai@0.81.1 (currently only present nested under the SDK; B1 adds it as a direct, version-pinned dependency).

Goal / acceptance criteria

With PILOOP_VERTEX_ANTHROPIC=1 and PILOOP_MODEL="vertex-anthropic/claude-opus-4-8", a pi-loop stage session runs against Claude on Vertex and returns a non-empty streamed response using existing GCP/ADC credentials.
With the flag unset (default), behaviour is unchanged: no provider is registered, no new required config, all existing tests pass.
The provider is registered via ModelRuntime.registerProvider(...) in createPiStageSession.ts, gated by the env flag.
Deterministic seams (config resolution, auth strategy selection, message/tool conversion, registration) are unit-tested; the LLM stream is treated as a black box.
A gated (PILOOP_E2E=1) live smoke test proves streaming + auth end-to-end and asserts only on exit conditions (non-empty text, tool invoked), never exact text.
No secret/token value is ever logged.
npm run lint, npm test, and npm run build all pass.
MIT attribution recorded for the vendored code.

Source to vendor

From ~/.pi/agent/npm/node_modules/pi-provider-vertex-anthropic@0.1.3/src/ (MIT). Target dir: src/model/vertexAnthropic/.

File	Role	Import change
`config.ts`	Resolve project/region; build endpoint host + `streamRawPredict` URL	none (pure Node)
`auth.ts`	`getAccessToken()`: service-account JWT → ADC refresh → `gcloud` CLI; 55-min cache	none
`shell.ts`	`spawn`/`exec` helpers used by `auth.ts`	none
`models.ts`	`VERTEX_MODELS`, `PROVIDER_NAME='vertex-anthropic'`, `API_NAME='vertex-anthropic-api'`	none
`messages.ts`	Pi↔Anthropic message/tool conversion, stop-reason map, surrogate sanitising	swap `@mariozechner/pi-ai` → `@earendil-works/pi-ai`
`vertex-api.ts`	`streamVertexAnthropic` (custom `streamSimple`): SSE parse → SDK stream events	swap `@mariozechner/pi-ai` (types + `calculateCost`, `createAssistantMessageEventStream`) → `@earendil-works/pi-ai`

pre-register.ts and index.ts (the ExtensionAPI entry) are not vendored — pi-loop registers the provider itself.

Tasks

Step 0 — De-risk: streaming-type compatibility spike (do first)

Add @earendil-works/pi-ai@0.81.1 (exact) to package.json.
Typecheck a copy of vertex-api.ts + messages.ts against @earendil-works/pi-ai types (AssistantMessage, Model, Context, SimpleStreamOptions, AssistantMessageEventStream, ThinkingContent, ToolCall) and runtime helpers (createAssistantMessageEventStream, calculateCost).
Confirm the event push shapes (start, text_start/_delta/_end, thinking_*, toolcall_*, done, error) and field names (contentIndex, partial, delta, content, toolCall, reason) match.
Exit criterion: both files typecheck with only import-path changes. Capture any shape deltas to apply in Step 3.

Step 1 — Dependency

Add "@earendil-works/pi-ai": "0.81.1" (exact) to dependencies; npm install; confirm npm run build passes.

Step 2 — Vendor pure modules

Copy config.ts, auth.ts, shell.ts, models.ts into src/model/vertexAnthropic/ (no logic changes).
Add a provenance/attribution header (source repo + MIT) to each vendored file.
Ensure no token value is ever logged (route console.error through pi-loop's logger if trivial, else leave and confirm no secrets printed).

Step 3 — Vendor streaming + messages (import re-point)

Copy messages.ts and vertex-api.ts, changing only imports @mariozechner/pi-ai → @earendil-works/pi-ai; apply Step 0 shape adjustments if any.

Step 4 — Registration entry point

Add src/model/vertexAnthropic/registerVertexAnthropic.ts exporting registerVertexAnthropic(modelRuntime) that calls modelRuntime.registerProvider('vertex-anthropic', { name, baseUrl, api: 'vertex-anthropic-api', apiKey: 'vertex-anthropic', models: VERTEX_MODELS, streamSimple: streamVertexAnthropic }). Omit the interactive oauth block (unattended).

Step 5 — Wire into stage session (env-gated)

In createPiStageSession.ts, after ModelRuntime.create(), add: if (process.env.PILOOP_VERTEX_ANTHROPIC === '1') registerVertexAnthropic(modelRuntime);

Step 6 — Config surface

Document env vars (no change to the typed config unless needed): PILOOP_VERTEX_ANTHROPIC, PILOOP_MODEL, GOOGLE_CLOUD_PROJECT/ANTHROPIC_VERTEX_PROJECT_ID, region vars (CLOUD_ML_REGION/VERTEX_LOCATION/VERTEXAI_LOCATION; decide whether to also honour GOOGLE_CLOUD_LOCATION for consistency with the builtin), and GOOGLE_APPLICATION_CREDENTIALS.
Default region in vendored config.ts is us-east5; document that this project needs global.

Step 7 — Unit tests (deterministic seams only)

config.test.ts — project/region precedence, buildEndpointHost (global vs regional), buildStreamUrl shape.
auth.test.ts — strategy selection with mocked fs/fetch/child_process (SA JSON, ADC, gcloud fallback, cache hit/expiry, no-creds error); assert no token logged.
messages.test.ts — convertMessages (text-only vs image, tool_result batching, cache_control on last user block), convertTools, mapStopReason, normalizeToolCallId, sanitizeSurrogates, orphaned-tool-call synthesis.
registerVertexAnthropic.test.ts — fake ModelRuntime; assert one registerProvider call with the expected id/baseUrl/api/models.
createPiStageSession — env flag gates registration (flag on → registered, off → not); existing tests stay green.
Optional: parseSSE unit test with a canned SSE byte stream.

Step 8 — Live smoke (gated)

Add a PILOOP_E2E=1 e2e mirroring runStage.e2e.test.ts, run with PILOOP_VERTEX_ANTHROPIC=1 PILOOP_MODEL="vertex-anthropic/claude-opus-4-8" GOOGLE_CLOUD_PROJECT=teg-gemini-ent-prod VERTEX_LOCATION=global. Assert on exit conditions only.

Step 9 — Docs

README.md + AGENTS.md: add a "Claude on Vertex (optional)" note (env vars + working model/region caveat).
Mark B1 implemented in docs/scope-option-B-claude-on-vertex.md.
Add MIT attribution (NOTICE or per-file header + README mention).

Step 10 — Verify

npm run lint && npm test && npm run build, then the gated live smoke.

Risks & mitigations

Risk	Likelihood	Mitigation
Stream event shapes differ between `@mariozechner/pi-ai` and `@earendil-works/pi-ai`	Medium	Step 0 spike gates the build; adjust vendored streamer
0.81.1 rejects custom `api` name with `streamSimple`	Low	Verified live in Step 8; fallback: register under a known api + custom stream
`location=global` unsupported for some models/projects	Medium	Documented working combo (`global`+`claude-opus-4-8`); make region configurable
Vendored copy drifts from upstream / Vertex API changes	Low-Med	Provenance + version note; single stable streaming fn; re-sync manually
Token accidentally logged	Low	Explicit test asserts no token in logs
Dep version skew (`pi-ai` vs bundled SDK)	Low	Pin `@earendil-works/pi-ai` to exact `0.81.1`; bump with the SDK

Out of scope

Interactive /login OAuth flow (unattended pipeline; env/ADC only).
Per-stage model selection (still one shared model for all stages).
The separate skills-degraded cleanup (tracked elsewhere).
Anthropic-direct or other Vertex publishers.

Effort estimate

~2 days: Step 0 spike ~0.5d; vendor + register + wire + unit tests ~1d; live smoke + docs + verify ~0.5d.

## Summary Add support for running pi-loop's stage sessions on **Claude models served via Google Cloud Vertex AI** ("Claude on Vertex"), by vendoring the community `pi-provider-vertex-anthropic` logic into pi-loop and registering it directly on the SDK `ModelRuntime`. This is **opt-in** via an environment flag; all existing providers (Gemini via the builtin `google-vertex`, direct Anthropic, etc.) are unaffected when the flag is off. This is **Option B / Approach B1** from the LLM-provider investigation. See in-repo docs: - `docs/plan-B1-vertex-anthropic-provider.md` — the detailed implementation plan (source of truth for tasks). - `docs/scope-option-B-claude-on-vertex.md` — scoping + why B3 (config-only) was ruled out. ## Background / motivation pi-loop uses its **own bundled SDK** (`@earendil-works/pi-coding-agent@0.81.1`) and calls `ModelRuntime.create()` directly in `src/stage/services/createPiStageSession.ts`. It does **not** load the extension packages listed in `~/.pi/agent/settings.json`, so the custom `vertex-anthropic` provider used by interactive pi is not available to pi-loop. Investigation findings (all verified live): - The **builtin `google-vertex` provider works for Gemini** (`gemini-2.5-pro` returns OK) but **not Claude**: it uses `@google/genai` `generateContentStream` (the Gemini `generateContent` protocol, hardwired to `publishers/google`). Claude on Vertex needs `publishers/anthropic/...:streamRawPredict` with the Anthropic Messages body. - The **builtin `anthropic-messages` provider** wraps the official `@anthropic-ai/sdk` calling the direct `/v1/messages` API with `x-api-key`; it has no Vertex mode and cannot be config-pointed at the Vertex URL/auth. → **B3 (models.json only) is impossible.** - The **raw Vertex Anthropic endpoint works** with our GCP creds: `POST https://aiplatform.googleapis.com/v1/projects/<project>/locations/global/publishers/anthropic/models/claude-opus-4-8:streamRawPredict` body `{"anthropic_version":"vertex-2023-10-16","messages":[...],"max_tokens":N,"stream":true}` → **HTTP 200**, standard Anthropic Messages SSE. - **Auth**: our creds are ADC (`~/.config/gcloud/application_default_credentials.json`, `authorized_user` refresh-token flow). The vendored `auth.ts` chain (service-account JSON → ADC → `gcloud` CLI) resolves a bearer token. - **Working model/region for our project today**: `location=global` + `claude-opus-4-8`. (`us-east5` → 429 quota; `us-central1` → 400; pinned `@date` ids and `claude-sonnet-4-5*` → 404.) - **The SDK allows programmatic registration**: `ModelRuntime.registerProvider(id, config)` accepts a `ProviderConfigInput` that includes a custom `streamSimple` — this is exactly why B1 is feasible. - The streaming helpers the vendored code needs (`createAssistantMessageEventStream`, `calculateCost`) are exported by `@earendil-works/pi-ai@0.81.1` (currently only present nested under the SDK; B1 adds it as a direct, version-pinned dependency). ## Goal / acceptance criteria - [ ] With `PILOOP_VERTEX_ANTHROPIC=1` and `PILOOP_MODEL="vertex-anthropic/claude-opus-4-8"`, a pi-loop stage session runs against Claude on Vertex and returns a non-empty streamed response using existing GCP/ADC credentials. - [ ] With the flag unset (default), behaviour is **unchanged**: no provider is registered, no new required config, all existing tests pass. - [ ] The provider is registered via `ModelRuntime.registerProvider(...)` in `createPiStageSession.ts`, gated by the env flag. - [ ] Deterministic seams (config resolution, auth strategy selection, message/tool conversion, registration) are unit-tested; the LLM stream is treated as a black box. - [ ] A gated (`PILOOP_E2E=1`) live smoke test proves streaming + auth end-to-end and asserts only on exit conditions (non-empty text, tool invoked), never exact text. - [ ] No secret/token value is ever logged. - [ ] `npm run lint`, `npm test`, and `npm run build` all pass. - [ ] MIT attribution recorded for the vendored code. ## Source to vendor From `~/.pi/agent/npm/node_modules/pi-provider-vertex-anthropic@0.1.3/src/` (MIT). Target dir: `src/model/vertexAnthropic/`. | File | Role | Import change | | ---- | ---- | ------------- | | `config.ts` | Resolve project/region; build endpoint host + `streamRawPredict` URL | none (pure Node) | | `auth.ts` | `getAccessToken()`: service-account JWT → ADC refresh → `gcloud` CLI; 55-min cache | none | | `shell.ts` | `spawn`/`exec` helpers used by `auth.ts` | none | | `models.ts` | `VERTEX_MODELS`, `PROVIDER_NAME='vertex-anthropic'`, `API_NAME='vertex-anthropic-api'` | none | | `messages.ts` | Pi↔Anthropic message/tool conversion, stop-reason map, surrogate sanitising | swap `@mariozechner/pi-ai` → `@earendil-works/pi-ai` | | `vertex-api.ts` | `streamVertexAnthropic` (custom `streamSimple`): SSE parse → SDK stream events | swap `@mariozechner/pi-ai` (types + `calculateCost`, `createAssistantMessageEventStream`) → `@earendil-works/pi-ai` | `pre-register.ts` and `index.ts` (the `ExtensionAPI` entry) are **not** vendored — pi-loop registers the provider itself. ## Tasks ### Step 0 — De-risk: streaming-type compatibility spike (do first) - [ ] Add `@earendil-works/pi-ai@0.81.1` (exact) to `package.json`. - [ ] Typecheck a copy of `vertex-api.ts` + `messages.ts` against `@earendil-works/pi-ai` types (`AssistantMessage`, `Model`, `Context`, `SimpleStreamOptions`, `AssistantMessageEventStream`, `ThinkingContent`, `ToolCall`) and runtime helpers (`createAssistantMessageEventStream`, `calculateCost`). - [ ] Confirm the event push shapes (`start`, `text_start/_delta/_end`, `thinking_*`, `toolcall_*`, `done`, `error`) and field names (`contentIndex`, `partial`, `delta`, `content`, `toolCall`, `reason`) match. - [ ] **Exit criterion:** both files typecheck with only import-path changes. Capture any shape deltas to apply in Step 3. ### Step 1 — Dependency - [ ] Add `"@earendil-works/pi-ai": "0.81.1"` (exact) to dependencies; `npm install`; confirm `npm run build` passes. ### Step 2 — Vendor pure modules - [ ] Copy `config.ts`, `auth.ts`, `shell.ts`, `models.ts` into `src/model/vertexAnthropic/` (no logic changes). - [ ] Add a provenance/attribution header (source repo + MIT) to each vendored file. - [ ] Ensure no token value is ever logged (route `console.error` through pi-loop's logger if trivial, else leave and confirm no secrets printed). ### Step 3 — Vendor streaming + messages (import re-point) - [ ] Copy `messages.ts` and `vertex-api.ts`, changing only imports `@mariozechner/pi-ai` → `@earendil-works/pi-ai`; apply Step 0 shape adjustments if any. ### Step 4 — Registration entry point - [ ] Add `src/model/vertexAnthropic/registerVertexAnthropic.ts` exporting `registerVertexAnthropic(modelRuntime)` that calls `modelRuntime.registerProvider('vertex-anthropic', { name, baseUrl, api: 'vertex-anthropic-api', apiKey: 'vertex-anthropic', models: VERTEX_MODELS, streamSimple: streamVertexAnthropic })`. Omit the interactive `oauth` block (unattended). ### Step 5 — Wire into stage session (env-gated) - [ ] In `createPiStageSession.ts`, after `ModelRuntime.create()`, add: `if (process.env.PILOOP_VERTEX_ANTHROPIC === '1') registerVertexAnthropic(modelRuntime);` ### Step 6 — Config surface - [ ] Document env vars (no change to the typed config unless needed): `PILOOP_VERTEX_ANTHROPIC`, `PILOOP_MODEL`, `GOOGLE_CLOUD_PROJECT`/`ANTHROPIC_VERTEX_PROJECT_ID`, region vars (`CLOUD_ML_REGION`/`VERTEX_LOCATION`/`VERTEXAI_LOCATION`; decide whether to also honour `GOOGLE_CLOUD_LOCATION` for consistency with the builtin), and `GOOGLE_APPLICATION_CREDENTIALS`. - [ ] Default region in vendored `config.ts` is `us-east5`; document that this project needs `global`. ### Step 7 — Unit tests (deterministic seams only) - [ ] `config.test.ts` — project/region precedence, `buildEndpointHost` (global vs regional), `buildStreamUrl` shape. - [ ] `auth.test.ts` — strategy selection with mocked `fs`/`fetch`/`child_process` (SA JSON, ADC, gcloud fallback, cache hit/expiry, no-creds error); assert **no token logged**. - [ ] `messages.test.ts` — `convertMessages` (text-only vs image, tool_result batching, cache_control on last user block), `convertTools`, `mapStopReason`, `normalizeToolCallId`, `sanitizeSurrogates`, orphaned-tool-call synthesis. - [ ] `registerVertexAnthropic.test.ts` — fake `ModelRuntime`; assert one `registerProvider` call with the expected id/baseUrl/api/models. - [ ] `createPiStageSession` — env flag gates registration (flag on → registered, off → not); existing tests stay green. - [ ] Optional: `parseSSE` unit test with a canned SSE byte stream. ### Step 8 — Live smoke (gated) - [ ] Add a `PILOOP_E2E=1` e2e mirroring `runStage.e2e.test.ts`, run with `PILOOP_VERTEX_ANTHROPIC=1 PILOOP_MODEL="vertex-anthropic/claude-opus-4-8" GOOGLE_CLOUD_PROJECT=teg-gemini-ent-prod VERTEX_LOCATION=global`. Assert on exit conditions only. ### Step 9 — Docs - [ ] `README.md` + `AGENTS.md`: add a "Claude on Vertex (optional)" note (env vars + working model/region caveat). - [ ] Mark B1 implemented in `docs/scope-option-B-claude-on-vertex.md`. - [ ] Add MIT attribution (NOTICE or per-file header + README mention). ### Step 10 — Verify - [ ] `npm run lint && npm test && npm run build`, then the gated live smoke. ## Risks & mitigations | Risk | Likelihood | Mitigation | | ---- | ---------- | ---------- | | Stream event shapes differ between `@mariozechner/pi-ai` and `@earendil-works/pi-ai` | Medium | Step 0 spike gates the build; adjust vendored streamer | | 0.81.1 rejects custom `api` name with `streamSimple` | Low | Verified live in Step 8; fallback: register under a known api + custom stream | | `location=global` unsupported for some models/projects | Medium | Documented working combo (`global`+`claude-opus-4-8`); make region configurable | | Vendored copy drifts from upstream / Vertex API changes | Low-Med | Provenance + version note; single stable streaming fn; re-sync manually | | Token accidentally logged | Low | Explicit test asserts no token in logs | | Dep version skew (`pi-ai` vs bundled SDK) | Low | Pin `@earendil-works/pi-ai` to exact `0.81.1`; bump with the SDK | ## Out of scope - Interactive `/login` OAuth flow (unattended pipeline; env/ADC only). - Per-stage model selection (still one shared model for all stages). - The separate skills-`degraded` cleanup (tracked elsewhere). - Anthropic-direct or other Vertex publishers. ## Effort estimate ~2 days: Step 0 spike ~0.5d; vendor + register + wire + unit tests ~1d; live smoke + docs + verify ~0.5d.

david commented

2026-07-27 06:39:49 +00:00

Note on claude-opus-4-8 model availability (affects Step 2 — vendoring models.ts)

claude-opus-4-8 was manually added to the local copy of the extension's src/models.ts; it is not listed by default in the pristine upstream pi-provider-vertex-anthropic@0.1.3 package. (The local VERTEX_MODELS currently contains claude-opus-4-8, claude-opus-4-6, claude-opus-4-5@20251101, claude-sonnet-4-5@20250929, etc.)

Implications for B1:

Do not assume a fresh npm install of the upstream package would contain claude-opus-4-8. When vendoring models.ts into src/model/vertexAnthropic/, treat the local, edited file as the source of truth and explicitly include claude-opus-4-8.
Add a code comment in the vendored models.ts recording that claude-opus-4-8 (and claude-opus-4-6) were added beyond the upstream defaults, so future re-syncs don't silently drop them.
The live smoke test (Step 8) depends on this entry existing, since global + claude-opus-4-8 is the only confirmed-working model/region for this project.

**Note on `claude-opus-4-8` model availability (affects Step 2 — vendoring `models.ts`)** `claude-opus-4-8` was **manually added** to the local copy of the extension's `src/models.ts`; it is **not** listed by default in the pristine upstream `pi-provider-vertex-anthropic@0.1.3` package. (The local `VERTEX_MODELS` currently contains `claude-opus-4-8`, `claude-opus-4-6`, `claude-opus-4-5@20251101`, `claude-sonnet-4-5@20250929`, etc.) Implications for B1: - Do **not** assume a fresh `npm install` of the upstream package would contain `claude-opus-4-8`. When vendoring `models.ts` into `src/model/vertexAnthropic/`, treat the **local, edited** file as the source of truth and explicitly include `claude-opus-4-8`. - Add a code comment in the vendored `models.ts` recording that `claude-opus-4-8` (and `claude-opus-4-6`) were added beyond the upstream defaults, so future re-syncs don't silently drop them. - The live smoke test (Step 8) depends on this entry existing, since `global` + `claude-opus-4-8` is the only confirmed-working model/region for this project.

david referenced this issue from a commit

2026-08-03 02:53:11 +00:00

Add opt-in Claude-on-Vertex provider (B1, issue-27)

david referenced this issue from a pull request that will close it,

2026-08-03 02:53:28 +00:00

Support Claude on Vertex AI (vendor + register vertex-anthropic provider) — B1 #169

david closed this issue

2026-08-10 01:32:45 +00:00

david referenced this issue from a commit

2026-08-10 01:32:47 +00:00

Support Claude on Vertex AI (vendor + register vertex-anthropic provider) — B1 (#169)

Sign in to join this conversation.