Implement src/serialize.ts with unit tests (EJSON + truncation) #138

New issue

Closed

opened 2026-08-31 22:04:13 +00:00 by david · 1 comment

david commented

2026-08-31 22:04:13 +00:00

Owner

Summary

Implement extensions/mongodb/src/serialize.ts — ejsonSerializeWithTruncation(docs), which serializes query results as Extended JSON with a byte cap that truncates only at document boundaries — with unit tests.

Background

Depends on: #133

mongo_find (a later step) returns query documents through this module. Two concerns are solved here: (1) BSON types (ObjectId, Date, Binary, Decimal128, Long) have no plain-JSON equivalent — plain JSON.stringify flattens ObjectId to a bare hex string and mangles Binary/Long — so results must be serialized with MongoDB's Extended JSON (EJSON), which renders them losslessly ({"$oid": ...}, {"$date": ...}, {"$binary": ...}, {"$numberDecimal": ...}); (2) the LLM's context window must be protected from unbounded output, so output is byte-capped and, when truncated, a visible marker tells the agent to narrow the filter/projection rather than silently missing data. Uses OUTPUT_BYTE_CAP from src/defaults.ts (sibling step in this milestone — do it first).

Documentation Required

A separate process downloads these into the listed folder before this issue is implemented. Check the folder for the actual reference material before starting.

docs/reference/mongodb-driver/ — for the EJSON API (re-exported from the bson package via the mongodb driver):

https://www.mongodb.com/docs/drivers/node/current/data-formats/extended-json/ — guide: EJSON.stringify() (relaxed mode is the default; { relaxed: false } for canonical), EJSON.parse() (accepts both relaxed and canonical), worked examples for ObjectId/Date/Code/Binary.
https://mongodb.github.io/node-mongodb-native/7.6/variables/BSON.EJSON.html — API reference for the EJSON const: stringify, parse, serialize, deserialize signatures.
https://mongodb.github.io/node-mongodb-native/7.6/ — TypeDoc API reference index (context for the driver's type surface).

Implementation Details

ejsonSerializeWithTruncation(docs: Document[]): { text: string; truncated: boolean; docCount: number }

Serialize documents one at a time with EJSON.stringify(doc, null, 2) (relaxed — the default; EJSON is available as import { EJSON } from "mongodb" or BSON.EJSON) and accumulate into an array of strings, tracking a running byte length (Buffer.byteLength(str) or new TextEncoder().encode(str).length — both work under bun).
Before adding the next document, if its addition would push the total past OUTPUT_BYTE_CAP (100 000), stop — the array is never cut mid-document, so the output stays valid JSON.
Build the result as a JSON array of the serialized docs.
When any docs were dropped, append a visible marker line: \n… (truncated: <returned> of <total> docs, <bytes>/<cap> bytes) where <returned> = number of docs actually serialized, <total> = docs.length, <bytes> = the byte length of the accumulated text before the marker, <cap> = OUTPUT_BYTE_CAP.
Return { text, truncated, docCount } — truncated true iff at least one doc was dropped; docCount = number of docs serialized.

Write co-located tests in src/serialize.test.ts (bun test):

EJSON round-trip: EJSON.parse(EJSON.stringify(doc)) preserves ObjectId/Date/Binary/Decimal128 (assert the BSON types survive the round-trip).
Under-cap input → untruncated output, truncated: false, no marker.
Over-cap input → truncated at a document boundary; marker present with correct <returned>/<total>; the JSON before the marker parses cleanly (JSON.parse/EJSON.parse).
truncated flag and docCount correct in both cases.

Acceptance Criteria

EJSON round-trip preserves ObjectId/Date/Binary/Decimal128.
Under-cap output is untruncated, truncated: false, no marker.
Over-cap output truncates at a document boundary (never mid-document) and remains valid JSON up to the marker.
The truncation marker … (truncated: <returned> of <total> docs, <bytes>/<cap> bytes) is present and correct when truncated.
truncated flag and docCount are correct in both cases.
bun test in extensions/mongodb/ is green for src/serialize.ts.

Test Plan

cd extensions/mongodb
bun test   # serialize tests green

## Summary Implement `extensions/mongodb/src/serialize.ts` — `ejsonSerializeWithTruncation(docs)`, which serializes query results as Extended JSON with a byte cap that truncates only at document boundaries — with unit tests. ## Background **Depends on:** #133 `mongo_find` (a later step) returns query documents through this module. Two concerns are solved here: (1) BSON types (ObjectId, Date, Binary, Decimal128, Long) have no plain-JSON equivalent — plain `JSON.stringify` flattens ObjectId to a bare hex string and mangles Binary/Long — so results must be serialized with MongoDB's Extended JSON (EJSON), which renders them losslessly (`{"$oid": ...}`, `{"$date": ...}`, `{"$binary": ...}`, `{"$numberDecimal": ...}`); (2) the LLM's context window must be protected from unbounded output, so output is byte-capped and, when truncated, a visible marker tells the agent to narrow the filter/projection rather than silently missing data. Uses `OUTPUT_BYTE_CAP` from `src/defaults.ts` (sibling step in this milestone — do it first). ## Documentation Required A separate process downloads these into the listed folder before this issue is implemented. Check the folder for the actual reference material before starting. **`docs/reference/mongodb-driver/`** — for the `EJSON` API (re-exported from the `bson` package via the `mongodb` driver): - https://www.mongodb.com/docs/drivers/node/current/data-formats/extended-json/ — guide: `EJSON.stringify()` (relaxed mode is the default; `{ relaxed: false }` for canonical), `EJSON.parse()` (accepts both relaxed and canonical), worked examples for ObjectId/Date/Code/Binary. - https://mongodb.github.io/node-mongodb-native/7.6/variables/BSON.EJSON.html — API reference for the `EJSON` const: `stringify`, `parse`, `serialize`, `deserialize` signatures. - https://mongodb.github.io/node-mongodb-native/7.6/ — TypeDoc API reference index (context for the driver's type surface). ## Implementation Details `ejsonSerializeWithTruncation(docs: Document[]): { text: string; truncated: boolean; docCount: number }` 1. Serialize documents **one at a time** with `EJSON.stringify(doc, null, 2)` (relaxed — the default; `EJSON` is available as `import { EJSON } from "mongodb"` or `BSON.EJSON`) and accumulate into an array of strings, tracking a running byte length (`Buffer.byteLength(str)` or `new TextEncoder().encode(str).length` — both work under bun). 2. Before adding the next document, if its addition would push the total past `OUTPUT_BYTE_CAP` (100 000), **stop** — the array is never cut mid-document, so the output stays valid JSON. 3. Build the result as a JSON array of the serialized docs. 4. When any docs were dropped, append a visible marker line: `\n… (truncated: <returned> of <total> docs, <bytes>/<cap> bytes)` where `<returned>` = number of docs actually serialized, `<total>` = `docs.length`, `<bytes>` = the byte length of the accumulated text before the marker, `<cap>` = `OUTPUT_BYTE_CAP`. 5. Return `{ text, truncated, docCount }` — `truncated` true iff at least one doc was dropped; `docCount` = number of docs serialized. Write co-located tests in `src/serialize.test.ts` (bun test): - EJSON round-trip: `EJSON.parse(EJSON.stringify(doc))` preserves ObjectId/Date/Binary/Decimal128 (assert the BSON types survive the round-trip). - Under-cap input → untruncated output, `truncated: false`, no marker. - Over-cap input → truncated at a document boundary; marker present with correct `<returned>`/`<total>`; the JSON before the marker parses cleanly (`JSON.parse`/`EJSON.parse`). - `truncated` flag and `docCount` correct in both cases. ## Acceptance Criteria - [ ] EJSON round-trip preserves ObjectId/Date/Binary/Decimal128. - [ ] Under-cap output is untruncated, `truncated: false`, no marker. - [ ] Over-cap output truncates at a document boundary (never mid-document) and remains valid JSON up to the marker. - [ ] The truncation marker `… (truncated: <returned> of <total> docs, <bytes>/<cap> bytes)` is present and correct when truncated. - [ ] `truncated` flag and `docCount` are correct in both cases. - [ ] `bun test` in `extensions/mongodb/` is green for `src/serialize.ts`. ## Test Plan ```bash cd extensions/mongodb bun test # serialize tests green ```

david added this to the MongoDB: M3 - Defaults & Serialization milestone

2026-08-31 22:04:13 +00:00

david added the

feature

label

2026-08-31 22:04:13 +00:00

david referenced this issue

2026-08-31 22:05:50 +00:00

Implement the mongo_find tool with unit tests #140

david added the

ready

label