Live shoppy-categories embedder has drifted to a dead private IP, breaking all category writes #12

New issue

Open

opened 2026-11-07 23:26:09 +00:00 by david · 0 comments

david commented

2026-11-07 23:26:09 +00:00

Owner

Summary

The shoppy-categories index on the shared Meilisearch instance (meilisearch.carlton.kongkers.net) has its mxbai vector embedder configured against http://10.1.5.170:1234/v1/embeddings (model text-embedding-nomic-embed-text-v2-moe) — a private IP that is unreachable from the Meilisearch host ("Connection refused"). Every document write to the index fails.

This is a drift from the canonical settings in this repo: shoppy-categories.json defines the same mxbai embedder against https://openrouter.ai/api/v1/embeddings (model qwen/qwen3-embedding-8b) — the same pattern the live products index (shoppy-test-v12) runs and works with.

Evidence (2026-11-07)

Task 4834 — upsert of a single category doc to the live index:

{
  "uid": 4834,
  "type": "documentAdditionOrUpdate",
  "indexUid": "shoppy-categories",
  "status": "failed",
  "error": {
    "message": "Index `shoppy-categories`: Error while generating embeddings: runtime error: could not determine model dimensions:\n  - test embedding failed with runtime error: could not reach embedding server:\n  - io: Connection refused",
    "code": "vector_embedding_error",
    "type": "invalid_request"
  }
}

Source	`mxbai` embedder URL	Model	Works?
Live `shoppy-categories`	`http://10.1.5.170:1234/v1/embeddings` (private IP)	`text-embedding-nomic-embed-text-v2-moe`	no — Connection refused
This repo's `shoppy-categories.json`	`https://openrouter.ai/api/v1/embeddings`	`qwen/qwen3-embedding-8b`	(intended config)
Live `shoppy-test-v12` (products)	`https://openrouter.ai/api/v1/embeddings` (public)	`qwen/qwen3-embedding-8b`	yes — a product upsert succeeded the same day

The 10.1.5.170 host was presumably a local/self-hosted embedding server that is no longer running or no longer on the instance's network.

Impact

GET-only consumers (the BFF's category lookup, filter: id=<id>) are unaffected — reads do not run the embedder. The category tree currently served to dev is intact.
Any write path on the live index is broken: product-sync's category ingestion against this instance, and the BFF repo's Bruno fixture seeding (bff bruno/ci/seed-meilisearch.sh), which blocks Bruno suite tests 16/17 (category listing specials_first sorts) locally.
Forgejo CI is unaffected: the BFF's Bruno CI job runs its own Meilisearch service container with no embedder.

Suggested fix

Re-apply the embedder settings from this repo's shoppy-categories.json to the live shoppy-categories index (i.e. revert the mxbai embedder to the openrouter config, with a real OPENROUTER_API_KEY in place of the {OPENROUTER_API_KEY} placeholder). If the openrouter route is not wanted, decide on the replacement and update shoppy-categories.json here first so the repo stays the source of truth.

Then re-apply any category documents that failed while the embedder was broken (e.g. the BFF repo's Bruno fixture breakfast category doc).

Acceptance criteria

A document write to the live shoppy-categories index succeeds (task status succeeded)
product-sync's category sync against this instance completes
Live index embedder settings match shoppy-categories.json in this repo

Verification

# With a write-capable key for the instance:
curl -s -X POST "https://meilisearch.carlton.kongkers.net/indexes/shoppy-categories/documents" \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '[{"id":"__embedder-check","name":"embedder check","level":1,"order":0,"parent_id":null,"parent_name":null,"description":null,"thumbnail":null}]'
# -> poll the returned taskUid; expect "succeeded" (then delete the check doc)

Filed in shoppy/bff as #139, moved here: this repo owns the index settings the live index has drifted from.

## Summary The `shoppy-categories` index on the shared Meilisearch instance (`meilisearch.carlton.kongkers.net`) has its `mxbai` vector embedder configured against `http://10.1.5.170:1234/v1/embeddings` (model `text-embedding-nomic-embed-text-v2-moe`) — a private IP that is unreachable from the Meilisearch host ("Connection refused"). **Every document write to the index fails.** This is a drift from the canonical settings in this repo: `shoppy-categories.json` defines the same `mxbai` embedder against `https://openrouter.ai/api/v1/embeddings` (model `qwen/qwen3-embedding-8b`) — the same pattern the live products index (`shoppy-test-v12`) runs and works with. ## Evidence (2026-11-07) Task 4834 — upsert of a single category doc to the live index: ```json { "uid": 4834, "type": "documentAdditionOrUpdate", "indexUid": "shoppy-categories", "status": "failed", "error": { "message": "Index `shoppy-categories`: Error while generating embeddings: runtime error: could not determine model dimensions:\n - test embedding failed with runtime error: could not reach embedding server:\n - io: Connection refused", "code": "vector_embedding_error", "type": "invalid_request" } } ``` | Source | `mxbai` embedder URL | Model | Works? | |---|---|---|---| | Live `shoppy-categories` | `http://10.1.5.170:1234/v1/embeddings` (private IP) | `text-embedding-nomic-embed-text-v2-moe` | no — Connection refused | | This repo's `shoppy-categories.json` | `https://openrouter.ai/api/v1/embeddings` | `qwen/qwen3-embedding-8b` | (intended config) | | Live `shoppy-test-v12` (products) | `https://openrouter.ai/api/v1/embeddings` (public) | `qwen/qwen3-embedding-8b` | yes — a product upsert succeeded the same day | The `10.1.5.170` host was presumably a local/self-hosted embedding server that is no longer running or no longer on the instance's network. ## Impact - `GET`-only consumers (the BFF's category lookup, `filter: id=<id>`) are unaffected — reads do not run the embedder. The category tree currently served to dev is intact. - Any **write** path on the live index is broken: product-sync's category ingestion against this instance, and the BFF repo's Bruno fixture seeding (`bff` `bruno/ci/seed-meilisearch.sh`), which blocks Bruno suite tests 16/17 (category listing `specials_first` sorts) locally. - Forgejo CI is unaffected: the BFF's Bruno CI job runs its own Meilisearch service container with no embedder. ## Suggested fix Re-apply the embedder settings from this repo's `shoppy-categories.json` to the live `shoppy-categories` index (i.e. revert the `mxbai` embedder to the openrouter config, with a real `OPENROUTER_API_KEY` in place of the `{OPENROUTER_API_KEY}` placeholder). If the openrouter route is not wanted, decide on the replacement and update `shoppy-categories.json` here first so the repo stays the source of truth. Then re-apply any category documents that failed while the embedder was broken (e.g. the BFF repo's Bruno fixture `breakfast` category doc). ## Acceptance criteria - [ ] A document write to the live `shoppy-categories` index succeeds (task status `succeeded`) - [ ] product-sync's category sync against this instance completes - [ ] Live index embedder settings match `shoppy-categories.json` in this repo ## Verification ```bash # With a write-capable key for the instance: curl -s -X POST "https://meilisearch.carlton.kongkers.net/indexes/shoppy-categories/documents" \ -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \ -d '[{"id":"__embedder-check","name":"embedder check","level":1,"order":0,"parent_id":null,"parent_name":null,"description":null,"thumbnail":null}]' # -> poll the returned taskUid; expect "succeeded" (then delete the check doc) ``` --- *Filed in `shoppy/bff` as #139, moved here: this repo owns the index settings the live index has drifted from.*

david added the

bug

label

2026-11-07 23:26:09 +00:00