Implement the Transit decoder and SSE stream parser in src/binfile.ts with unit tests #202

New issue

Closed

opened 2026-09-14 23:12:05 +00:00 by david · 1 comment

david commented

2026-09-14 23:12:05 +00:00

Owner

Summary

Implement the two pieces of machinery the .penpot binfile endpoints need and nothing else does: a minimal Transit decoder (~:keyword, ~u<uuid>, ~#uri) and a server-sent-events stream parser that reads event: progress lines and extracts the final event: end payload. Both are pure and unit-tested.

Background

Depends on: #188

On Penpot 2.17 the two binfile commands are the one corner of the API that does not speak plain JSON:

import-binfile and export-binfile respond with Transit payloads and Transit error bodies, even when the request and Accept headers are JSON.
Both stream SSE: a series of event: progress frames, terminated by an event: end frame whose data: line carries the result.
import-binfile's end payload is a Transit vector of created file ids: data: ["~u<uuid>"].
export-binfile's end payload is a Transit URI: data: {"~#uri":"https://…/assets/by-id/…"}.

So a Transit reader is unavoidable, but only a tiny one: three tags. Isolating it here means the import/export tools are thin, and a malformed stream can be debugged in one place. On any parse failure, fail loudly with the raw stream included — a silently empty result would be worse than a crash.

Documentation Required

A separate process downloads these into the listed folders before this issue is implemented. Check the folders for the actual reference material before starting.

docs/reference/transit-format/

https://github.com/cognitect/transit-format — the specification: the JSON/json-verbose encodings, the ground types, and the tag forms (~:keyword, ~u for UUIDs, ~# for extension/custom tags such as URIs). Implement only what is listed above; note in a comment where the full spec differs.
https://github.com/cognitect/transit-format/blob/master/README.md — the canonical README, including the tag→representation table this decoder's subset comes from.

docs/reference/penpot-api/

<PENPOT_URL>/api/main/doc/openapi.json — the import-binfile / export-binfile parameters and response types on this instance.
https://help.penpot.app/user-guide/import-export/ — what the export/import flow produces and consumes (.penpot ZIP with manifest.json, files/<id>.json, colors/, typographies/), so the values being decoded are recognisable.
https://help.penpot.app/user-guide/export-import/export-import-files/ — the "Export shared libraries" behaviour that shapes what the streamed URI points at.

docs/reference/nodejs/

https://nodejs.org/api/stream.html and https://nodejs.org/api/webstreams.html — reading a streaming response body (response.body) incrementally.
https://nodejs.org/api/globals.html#response — Response.body and the reader API used by the stream parser.

Implementation Details

Create extensions/penpot/src/binfile.ts (with src/binfile.test.ts beside it) exporting:

Transit decoding

decodeTransit(text) → the decoded value, handling:
- JSON arrays/objects/strings/numbers/booleans/nulls as-is,
- "~:keyword" → a keyword value (decide and document the representation — a Symbol, a { keyword } wrapper, or a plain string — but keep it consistent and test it),
- "~u<uuid>" → the UUID string,
- {"~#uri": "https://…"} → the URI as a string,
- a bare Transit array whose first element is a type tag (e.g. ["~#uri", "https://…"]), since json-verbose uses that form,
- transit caching (^0, ^1, …) only if a live response turns out to use it — verify against the instance and document the decision in findings.md.
decodeTransitError(rawText) → a readable message for a Transit error body (delegating to the error decoder shape used by the rest of the extension). Unrecognised Transit must produce a message containing the raw payload, never an empty string.
On any parse failure: throw/return an explicit parse error carrying the raw input.

SSE parsing

parseSseStream(body: ReadableStream<Uint8Array> | string, options?) → an async iterator (or a collector) of frames { event, data }, handling:
- event: / data: / optional id: / retry: lines,
- multi-line data: fields concatenated with newlines (per the SSE spec),
- comment lines starting with :,
- chunk boundaries that split a line (this is the most common bug — test it explicitly).
awaitStreamEnd(frames, { command }) → resolves the event: end payload (decoded via decodeTransit) or throws a clear error on: a stream that ends with no end frame, an event: error/event: fail frame, or a transport/parse failure. Include the accumulated raw frames in the error so a failure is diagnosable.
Progress frames are surfaced via an optional callback so the tools can report progress without buffering everything.

Keep the module free of HTTP: it takes a body/string and returns values.

Acceptance Criteria

decodeTransit decodes ["~u<uuid>"] to an array containing the UUID string.
decodeTransit decodes {"~#uri":"https://…"} and ["~#uri","https://…"] to the URI string.
decodeTransit decodes ~:keyword forms consistently and the representation is documented in the code.
decodeTransit passes plain JSON through unchanged.
decodeTransitError produces a readable message that includes the raw payload when the Transit body cannot be decoded.
parseSseStream yields the correct frames when a data: line is split across two chunks.
Multi-line data: fields are concatenated, and : comment lines are ignored.
awaitStreamEnd returns the decoded end payload, and fails loudly (with the raw frames attached) when there is no end frame or an error frame appears.
No HTTP call is made by this module; tests use strings/ReadableStreams only.

Test Plan

node --test extensions/penpot/src/binfile.test.ts

Live confirmation (requires PENPOT_URL/PENPOT_TOKEN) is done in the import/export steps; here, additionally:

Capture the raw stream from a real export-binfile call with curl and check it into the test suite as a fixture if it reveals encodings not covered above.
Record in extensions/penpot/findings.md whether the real streams use Transit caching (^N) codes, and what the exact end payload looked like for both commands.

## Summary Implement the two pieces of machinery the `.penpot` binfile endpoints need and nothing else does: a minimal **Transit** decoder (`~:keyword`, `~u<uuid>`, `~#uri`) and a **server-sent-events** stream parser that reads `event: progress` lines and extracts the final `event: end` payload. Both are pure and unit-tested. ## Background **Depends on:** #188 On Penpot 2.17 the two binfile commands are the one corner of the API that does not speak plain JSON: - `import-binfile` and `export-binfile` respond with **Transit** payloads and Transit **error** bodies, even when the request and `Accept` headers are JSON. - Both stream **SSE**: a series of `event: progress` frames, terminated by an `event: end` frame whose `data:` line carries the result. - `import-binfile`'s `end` payload is a Transit **vector of created file ids**: `data: ["~u<uuid>"]`. - `export-binfile`'s `end` payload is a Transit **URI**: `data: {"~#uri":"https://…/assets/by-id/…"}`. So a Transit reader is unavoidable, but only a tiny one: three tags. Isolating it here means the import/export tools are thin, and a malformed stream can be debugged in one place. On any parse failure, fail loudly with the raw stream included — a silently empty result would be worse than a crash. ## Documentation Required A separate process downloads these into the listed folders before this issue is implemented. Check the folders for the actual reference material before starting. **`docs/reference/transit-format/`** - https://github.com/cognitect/transit-format — the specification: the JSON/`json-verbose` encodings, the ground types, and the tag forms (`~:keyword`, `~u` for UUIDs, `~#` for extension/custom tags such as URIs). Implement only what is listed above; note in a comment where the full spec differs. - https://github.com/cognitect/transit-format/blob/master/README.md — the canonical README, including the tag→representation table this decoder's subset comes from. **`docs/reference/penpot-api/`** - `<PENPOT_URL>/api/main/doc/openapi.json` — the `import-binfile` / `export-binfile` parameters and response types on this instance. - https://help.penpot.app/user-guide/import-export/ — what the export/import flow produces and consumes (`.penpot` ZIP with `manifest.json`, `files/<id>.json`, `colors/`, `typographies/`), so the values being decoded are recognisable. - https://help.penpot.app/user-guide/export-import/export-import-files/ — the "Export shared libraries" behaviour that shapes what the streamed URI points at. **`docs/reference/nodejs/`** - https://nodejs.org/api/stream.html and https://nodejs.org/api/webstreams.html — reading a streaming response body (`response.body`) incrementally. - https://nodejs.org/api/globals.html#response — `Response.body` and the reader API used by the stream parser. ## Implementation Details Create `extensions/penpot/src/binfile.ts` (with `src/binfile.test.ts` beside it) exporting: ### Transit decoding - `decodeTransit(text)` → the decoded value, handling: - JSON arrays/objects/strings/numbers/booleans/nulls as-is, - `"~:keyword"` → a keyword value (decide and document the representation — a `Symbol`, a `{ keyword }` wrapper, or a plain string — but keep it consistent and test it), - `"~u<uuid>"` → the UUID string, - `{"~#uri": "https://…"}` → the URI as a string, - a bare Transit array whose first element is a type tag (e.g. `["~#uri", "https://…"]`), since `json-verbose` uses that form, - **transit caching** (`^0`, `^1`, …) only if a live response turns out to use it — verify against the instance and document the decision in `findings.md`. - `decodeTransitError(rawText)` → a readable message for a Transit error body (delegating to the error decoder shape used by the rest of the extension). Unrecognised Transit must produce a message containing the raw payload, never an empty string. - On any parse failure: throw/return an explicit parse error carrying the raw input. ### SSE parsing - `parseSseStream(body: ReadableStream<Uint8Array> | string, options?)` → an async iterator (or a collector) of frames `{ event, data }`, handling: - `event:` / `data:` / optional `id:` / `retry:` lines, - multi-line `data:` fields concatenated with newlines (per the SSE spec), - comment lines starting with `:`, - chunk boundaries that split a line (this is the most common bug — test it explicitly). - `awaitStreamEnd(frames, { command })` → resolves the `event: end` payload (decoded via `decodeTransit`) or throws a clear error on: a stream that ends with no `end` frame, an `event: error`/`event: fail` frame, or a transport/parse failure. Include the accumulated raw frames in the error so a failure is diagnosable. - Progress frames are surfaced via an optional callback so the tools can report progress without buffering everything. Keep the module free of HTTP: it takes a body/string and returns values. ## Acceptance Criteria - [ ] `decodeTransit` decodes `["~u<uuid>"]` to an array containing the UUID string. - [ ] `decodeTransit` decodes `{"~#uri":"https://…"}` and `["~#uri","https://…"]` to the URI string. - [ ] `decodeTransit` decodes `~:keyword` forms consistently and the representation is documented in the code. - [ ] `decodeTransit` passes plain JSON through unchanged. - [ ] `decodeTransitError` produces a readable message that includes the raw payload when the Transit body cannot be decoded. - [ ] `parseSseStream` yields the correct frames when a `data:` line is split across two chunks. - [ ] Multi-line `data:` fields are concatenated, and `:` comment lines are ignored. - [ ] `awaitStreamEnd` returns the decoded `end` payload, and fails loudly (with the raw frames attached) when there is no `end` frame or an error frame appears. - [ ] No HTTP call is made by this module; tests use strings/`ReadableStream`s only. ## Test Plan ```bash node --test extensions/penpot/src/binfile.test.ts ``` Live confirmation (requires `PENPOT_URL`/`PENPOT_TOKEN`) is done in the import/export steps; here, additionally: 1. Capture the raw stream from a real `export-binfile` call with `curl` and check it into the test suite as a fixture if it reveals encodings not covered above. 2. Record in `extensions/penpot/findings.md` whether the real streams use Transit caching (`^N`) codes, and what the exact `end` payload looked like for both commands.

david added this to the Penpot: M4 - Library import/export (binfile) milestone

2026-09-14 23:12:05 +00:00

david added the

feature

label

2026-09-14 23:12:05 +00:00

david referenced this issue

2026-09-14 23:12:18 +00:00

Implement penpot_import_library (multipart upload plus SSE result) #203

david referenced this issue

2026-09-14 23:12:29 +00:00

Implement penpot_export_library (SSE URI plus artifact download) #204