documentation: add research brief on embedding batch size and sync throughput #11

Merged

david merged 1 commit from documentation/2026-11-02/research-brief-embedding-batch-throughput into main

2026-11-02 01:50:49 +00:00

david commented

2026-11-02 01:47:52 +00:00

Owner

Adds docs/research-brief-embedding-batch-size-throughput.md — the research write-up for the question "can we increase the number of products we embed in one request?" (working hypothesis: 2 at a time, syncs taking too long).

What it documents (all verified live, read-only, 2026-11-02):

The REST embedder already batches 10 docs/request (input: ["{{text}}", "{{..}}"], confirmed in the live shoppy-test-v12 embedder settings) — and 10 is a hard, unconfigurable Meilisearch cap (no maxBatchSize in the v1.56 API schema; upstream request meilisearch#5966 still open). The "2 at a time" impression is not supported by the config or task data.
Task/batch forensics on the live index: v12 was created without settings applied (no settingsUpdate between indexCreation and the first settings task), so the first settings apply (Oct 31) retro-embedded all 31,850 docs in one task — 5.05 hours at ~5.7s per 10-doc request — while the same data re-applied the next day took 245s. The variance is provider-side (queueing/429 backoff), not batch size.
Root cause of the missing settings: stale default settings paths in product-sync (src/config.rs:12-15) that point at a non-existent directory, with the failure swallowed.
Recommendations, in leverage order: (1) fix settings-at-creation in product-sync so retro-embeds never happen; (2) add OpenRouter provider.order pinning + check key rate-limit headroom; (3) longer term, pre-computed _vectors with a userProvided embedder so Meilisearch makes zero index-time embedding calls (bundles with the pending BFF hybrid follow-up); (4) track upstream #5966. Includes validation steps and open questions (e.g. who applied the Oct 31 hand-rolled settings payload).

Docs only — no code or settings files changed.

Adds `docs/research-brief-embedding-batch-size-throughput.md` — the research write-up for the question "can we increase the number of products we embed in one request?" (working hypothesis: 2 at a time, syncs taking too long). **What it documents** (all verified live, read-only, 2026-11-02): - The REST embedder already batches **10 docs/request** (`input: ["{{text}}", "{{..}}"]`, confirmed in the live `shoppy-test-v12` embedder settings) — and 10 is a hard, unconfigurable Meilisearch cap (no `maxBatchSize` in the v1.56 API schema; upstream request [meilisearch#5966](https://github.com/meilisearch/meilisearch/issues/5966) still open). The "2 at a time" impression is not supported by the config or task data. - Task/batch forensics on the live index: v12 was created **without settings applied** (no `settingsUpdate` between indexCreation and the first settings task), so the first settings apply (Oct 31) retro-embedded all 31,850 docs in one task — **5.05 hours at ~5.7s per 10-doc request** — while the same data re-applied the next day took **245s**. The variance is provider-side (queueing/429 backoff), not batch size. - Root cause of the missing settings: stale default settings paths in `product-sync` (`src/config.rs:12-15`) that point at a non-existent directory, with the failure swallowed. - Recommendations, in leverage order: (1) fix settings-at-creation in `product-sync` so retro-embeds never happen; (2) add OpenRouter `provider.order` pinning + check key rate-limit headroom; (3) longer term, pre-computed `_vectors` with a `userProvided` embedder so Meilisearch makes zero index-time embedding calls (bundles with the pending BFF `hybrid` follow-up); (4) track upstream #5966. Includes validation steps and open questions (e.g. who applied the Oct 31 hand-rolled settings payload). Docs only — no code or settings files changed.

david added 1 commit

2026-11-02 01:47:52 +00:00

documentation: add research brief on embedding batch size and sync throughput a0a27351a2

david merged commit d584bfb70b into main

2026-11-02 01:50:49 +00:00

david deleted branch documentation/2026-11-02/research-brief-embedding-batch-throughput

2026-11-02 01:50:49 +00:00

david referenced this pull request from a commit

2026-11-02 01:50:51 +00:00

documentation: add research brief on embedding batch size and sync throughput (#11)