issue-282: add searxng extension #283

Merged

david merged 1 commit from feature/issue-282/add-searxng-extension into main

2026-09-28 06:22:50 +00:00

yousuf commented

2026-09-26 02:55:23 +00:00

Collaborator

Closes #282

What

Adds a searxng extension registering one tool, web_search, backed by a self-hosted SearXNG metasearch instance.

Path	Purpose
`src/env.ts`	Resolves `SEARXNG_URL`, read fresh per call so `/reload` picks up a change; defaults to `http://localhost:8888`
`src/client.ts`	`searxngRequest` — always forces `format=json`, raises `HttpError` with status and body on non-2xx
`src/defaults.ts`	Response shaping: limit/page defaults, snippet truncation, dropping result entries with no URL
`src/errors.ts`	Typed `HttpError`
`src/tools/search.ts`	The `web_search` tool: params, caps, `ToolError` mapping
`docker-compose.yml`, `config/settings.yml`	Local instance
`README.md` (extension + root)	Setup, config, caveats
`package.json`	Manifest entry

Why

The other network-backed extensions here (victorialogs, mongodb, vision) each wrap one external service behind a small tool surface. Web search was the gap — there was no way to find a page before reading it with fetch/read.

Self-hosting keeps queries off third-party search APIs and avoids per-call billing. The cost is running a container, which is why the compose file ships alongside.

Three decisions worth calling out:

config/settings.yml is required, not a convenience. A stock SearXNG ships search.formats as html only and enables server.limiter bot detection. Both answer format=json with a bare 403, so an unconfigured instance fails 100% of calls. The file sets use_default_settings: true and overrides nothing else, so an image bump still inherits the upstream engine list without a merge.
Disabling the limiter is safe only because the port is on loopback. Noted in the extension README so the config isn't carried onto a public instance.
Port 8888, not 8080. SearXNG's internal port is 8080, the most contended port on a dev machine.

The response is reduced to title, URL, snippet and the engines that ranked the URL. The rest of the payload is scoring and template data the model can't act on, and it's a large fraction of the bytes. details.unresponsiveEngines is surfaced because a failed upstream engine is usually why a result set is thinner than expected.

How tested

bun test in extensions/searxng/ — 43 pass, 0 fail, across client, defaults, env, errors and tools/search. Fetch is stubbed, so no live instance is needed and the suite is hermetic.

Coverage targets the failure modes, not just the happy path:

format=json is sent regardless of caller input
Optional params (categories, engines, language, time_range) forwarded when supplied, omitted when not
Hits capped at 50 and flagged truncated, with totalBeforeLimit reporting the pre-limit count
A 403 raises a ToolError naming the settings.yml fix — the single most likely setup error
A connection failure raises a ToolError asking whether the container is running
A malformed body degrades to an empty hit list rather than throwing
answers accepted as bare strings or objects, both of which SearXNG has shipped
All three unresponsive_engines shapes parsed

Also verified live against a running local instance during development.

Out of scope notes

.env is not committed — excluded by the root .gitignore and confirmed with git add --dry-run before staging. .env.example carries the SEARXNG_SECRET placeholder and the openssl rand -hex 32 command.
The root test script is unchanged. These are bun tests and the script is node --test; victorialogs sits outside it for the same reason. Folding both in is a separate change rather than a drive-by here.
HTML-format parsing is not implemented and isn't planned — JSON is the supported path.

Closes #282 ## What Adds a `searxng` extension registering one tool, `web_search`, backed by a self-hosted [SearXNG](https://docs.searxng.org/) metasearch instance. | Path | Purpose | |---|---| | `src/env.ts` | Resolves `SEARXNG_URL`, read fresh per call so `/reload` picks up a change; defaults to `http://localhost:8888` | | `src/client.ts` | `searxngRequest` — always forces `format=json`, raises `HttpError` with status and body on non-2xx | | `src/defaults.ts` | Response shaping: limit/page defaults, snippet truncation, dropping result entries with no URL | | `src/errors.ts` | Typed `HttpError` | | `src/tools/search.ts` | The `web_search` tool: params, caps, `ToolError` mapping | | `docker-compose.yml`, `config/settings.yml` | Local instance | | `README.md` (extension + root) | Setup, config, caveats | | `package.json` | Manifest entry | ## Why The other network-backed extensions here (`victorialogs`, `mongodb`, `vision`) each wrap one external service behind a small tool surface. Web search was the gap — there was no way to *find* a page before reading it with `fetch`/`read`. Self-hosting keeps queries off third-party search APIs and avoids per-call billing. The cost is running a container, which is why the compose file ships alongside. Three decisions worth calling out: - **`config/settings.yml` is required, not a convenience.** A stock SearXNG ships `search.formats` as `html` only and enables `server.limiter` bot detection. Both answer `format=json` with a bare `403`, so an unconfigured instance fails 100% of calls. The file sets `use_default_settings: true` and overrides nothing else, so an image bump still inherits the upstream engine list without a merge. - **Disabling the limiter is safe only because the port is on loopback.** Noted in the extension README so the config isn't carried onto a public instance. - **Port 8888, not 8080.** SearXNG's internal port is 8080, the most contended port on a dev machine. The response is reduced to title, URL, snippet and the engines that ranked the URL. The rest of the payload is scoring and template data the model can't act on, and it's a large fraction of the bytes. `details.unresponsiveEngines` is surfaced because a failed upstream engine is usually why a result set is thinner than expected. ## How tested `bun test` in `extensions/searxng/` — **43 pass, 0 fail**, across `client`, `defaults`, `env`, `errors` and `tools/search`. Fetch is stubbed, so no live instance is needed and the suite is hermetic. Coverage targets the failure modes, not just the happy path: - `format=json` is sent regardless of caller input - Optional params (`categories`, `engines`, `language`, `time_range`) forwarded when supplied, omitted when not - Hits capped at 50 and flagged truncated, with `totalBeforeLimit` reporting the pre-limit count - A `403` raises a `ToolError` naming the `settings.yml` fix — the single most likely setup error - A connection failure raises a `ToolError` asking whether the container is running - A malformed body degrades to an empty hit list rather than throwing - `answers` accepted as bare strings *or* objects, both of which SearXNG has shipped - All three `unresponsive_engines` shapes parsed Also verified live against a running local instance during development. ## Out of scope notes - **`.env` is not committed** — excluded by the root `.gitignore` and confirmed with `git add --dry-run` before staging. `.env.example` carries the `SEARXNG_SECRET` placeholder and the `openssl rand -hex 32` command. - **The root `test` script is unchanged.** These are bun tests and the script is `node --test`; `victorialogs` sits outside it for the same reason. Folding both in is a separate change rather than a drive-by here. - HTML-format parsing is not implemented and isn't planned — JSON is the supported path.

yousuf added the

feature

label

2026-09-26 02:55:23 +00:00

yousuf added 1 commit

2026-09-26 02:55:23 +00:00

issue-282: add searxng extension f79a58fb85

Registers a single web_search tool backed by a self-hosted SearXNG
metasearch instance, so search does not depend on a provider's built-in
tool being available or unmetered.

The raw SearXNG payload is reduced to title, url, snippet and the
engines that ranked each URL; the remainder is scoring and template
noise the model cannot act on. Unresponsive engines are surfaced in
details, since they are usually why a result set is thinner than
expected.

config/settings.yml is required, not optional: a stock SearXNG ships
search.formats as html only and enables the bot limiter, and both
answer format=json with a bare 403.