MSSQL spike: scoring runner for both candidates #317

New issue

Closed

opened 2026-11-10 08:18:50 +00:00 by david · 1 comment

david commented

2026-11-10 08:18:50 +00:00

Owner

Summary

Write a runner that scores each candidate guard: false accepts on the reject corpus and accepts on the accept corpus.

Background

The spike gate is: a candidate passes only with zero false accepts on the must-reject corpus; among passing candidates, the one with the most accepts wins; there is no minimum accept rate. If neither passes, Stage 2 does not start. The runner output feeds spike/RESULT.md. It must run from a clean checkout.

Depends on: #316 (and, through it, the corpora and candidate A)

Implementation Details

Create extensions/mssql/spike/run.mjs. It loads corpus-reject and corpus-accept, imports validateSql from candidate-a.mjs and candidate-b.mjs, and for each candidate prints: total reject cases, false accepts (list each by label), total accept cases, accepts, rejected-accepts (list by label), and any candidate load error (e.g. the grammar failed to build) shown as "not viable". Use Node's built-in modules only (no new dependencies). Exit code 0 when the script ran; print a final line naming the winner (zero false accepts, then highest accepts; tie noted) or GATE: STOP — no candidate has zero false accepts. Optionally emit JSON with --json. Document in a short spike/README.md how to install the spike dependencies (npm install --no-save node-sql-parser antlr4) and run it.

Acceptance Criteria

node extensions/mssql/spike/run.mjs runs from a clean checkout after the documented install.
It reports false accepts (with labels) and accepts for each candidate.
A candidate that fails to load is reported as not viable, not a crash.
The final line names the winner or the STOP gate.
No new root dependencies and no package.json change.

Test Plan

Run the runner and check the printed counts match the corpus sizes. Temporarily add a write statement to a copy of a candidate's allow path (or a fake candidate that allows everything) and confirm the runner reports false accepts and the STOP line.

## Summary Write a runner that scores each candidate guard: false accepts on the reject corpus and accepts on the accept corpus. ## Background The spike gate is: a candidate passes only with zero false accepts on the must-reject corpus; among passing candidates, the one with the most accepts wins; there is no minimum accept rate. If neither passes, Stage 2 does not start. The runner output feeds `spike/RESULT.md`. It must run from a clean checkout. **Depends on:** #316 (and, through it, the corpora and candidate A) ## Implementation Details Create `extensions/mssql/spike/run.mjs`. It loads `corpus-reject` and `corpus-accept`, imports `validateSql` from `candidate-a.mjs` and `candidate-b.mjs`, and for each candidate prints: total reject cases, false accepts (list each by label), total accept cases, accepts, rejected-accepts (list by label), and any candidate load error (e.g. the grammar failed to build) shown as "not viable". Use Node's built-in modules only (no new dependencies). Exit code 0 when the script ran; print a final line naming the winner (zero false accepts, then highest accepts; tie noted) or `GATE: STOP — no candidate has zero false accepts`. Optionally emit JSON with `--json`. Document in a short `spike/README.md` how to install the spike dependencies (`npm install --no-save node-sql-parser antlr4`) and run it. ## Acceptance Criteria - [ ] `node extensions/mssql/spike/run.mjs` runs from a clean checkout after the documented install. - [ ] It reports false accepts (with labels) and accepts for each candidate. - [ ] A candidate that fails to load is reported as not viable, not a crash. - [ ] The final line names the winner or the STOP gate. - [ ] No new root dependencies and no `package.json` change. ## Test Plan Run the runner and check the printed counts match the corpus sizes. Temporarily add a write statement to a copy of a candidate's allow path (or a fake candidate that allows everything) and confirm the runner reports false accepts and the STOP line.

david added this to the MSSQL: M2 - Spike corpora and candidates milestone

2026-11-10 08:18:50 +00:00

david added the

feature

label

2026-11-10 08:18:50 +00:00

david referenced this issue

2026-11-10 08:19:06 +00:00

MSSQL spike: write RESULT.md, apply the gate, open the Stage 1 PR #318

david referenced this issue

2026-11-10 23:10:19 +00:00

317: MSSQL spike: scoring runner for both candidates #339