Verify the STE style live in a short /grill session and record the observations #299

New issue

Closed

opened 2026-10-09 21:20:58 +00:00 by david · 1 comment

david commented

2026-10-09 21:20:58 +00:00

Owner

Summary

Run one short /grill session and check the assistant's replies and the checkpoint against the rule categories. Record every rule the model broke, and any exemption leakage (plain prose written as a probing question). Do not change the rule text in this step.

Background

Depends on: #291, #292, #293, #294, #295, #296, #297, #298

The style is enforced by a prompt rule plus a stated self-check — no runtime hook and no message rewriter. The test suite proves the guidance is well formed; it cannot prove the model follows it. This step is the only check that the style actually holds in a live session.

Exemption leakage is the specific risk to watch. The exemption covers probing questions and their alternatives. If that exemption is read as "anything written as a question", ordinary prose turns into questions and the style disappears. The verification pass must look for this.

Implementation Details

Start pi in this repo and run /grill <a small but real topic> — for example a decision you can answer quickly. Do not use a throwaway topic with no substance; the style is most likely to break in a reply that explains or summarises.
Answer three questions so the session produces at least three assistant replies.
Run /checkpoint or /checkpoint chat and read the checkpoint.
Check each reply and the checkpoint against every rule category:
- sentence length: about 20 words or fewer for an instruction, about 25 words or fewer for a description;
- one idea per sentence;
- active voice;
- simple tenses only (past, present, future, imperative), no -ing form where a simple tense works;
- one word for one meaning, no synonym trading;
- no phrasal verb where one verb does the work;
- noun groups of three words or fewer;
- short, common words;
- articles kept.
Mark each sentence as exempt when it is one of: a probing question or one of its alternatives; code, a file path, an identifier, a tool name, or a command; text quoted from the user. Exempt text is expected to break the rules and is not a finding.
Record the findings. Append a "Live STE verification" section to extensions/grillme/findings.md (create the file if it does not exist) or add a new short note file next to it. For each finding, record:
- the rule broken;
- the offending sentence;
- which reply or which checkpoint section it came from;
- whether it is a genuine break or a borderline case.
  For each exemption leak, record the sentence, which rule the leak worked around, and whether the sentence was a probing question or ordinary prose.
Add a short verdict: which rules held, which did not, and whether the style is usable as shipped. Do not edit ste-style.ts or any rule text here — the notes are input for a later review.

Acceptance Criteria

A /grill session was run with at least three assistant replies.
The checkpoint was read.
Every rule category was checked against the observed text, and each finding names the rule, the sentence, and its source.
Every exemption leak (plain prose written as a probing question) is recorded, with the sentence and a verdict.
The findings are written into the repo, in extensions/grillme/findings.md or a note file next to it.
The notes carry a verdict: which rules held, which did not, and whether the style is usable as shipped.
No rule text, prompt text, or test was changed in this step.

Test Plan

The session itself is the test. Run:

cd extensions/grillme && npm test

Expected: green, and unchanged by this step.

Then, after the notes are written, verify the notes exist and name the rules:

grep -n "Live STE verification" extensions/grillme/findings.md

Expected: the new section heading.

git diff --stat

Expected: only the notes file changed — no source or test file.

## Summary Run one short `/grill` session and check the assistant's replies and the checkpoint against the rule categories. Record every rule the model broke, and any exemption leakage (plain prose written as a probing question). Do not change the rule text in this step. ## Background **Depends on:** #291, #292, #293, #294, #295, #296, #297, #298 The style is enforced by a prompt rule plus a stated self-check — no runtime hook and no message rewriter. The test suite proves the guidance is well formed; it cannot prove the model follows it. This step is the only check that the style actually holds in a live session. Exemption leakage is the specific risk to watch. The exemption covers **probing** questions and their alternatives. If that exemption is read as "anything written as a question", ordinary prose turns into questions and the style disappears. The verification pass must look for this. ## Implementation Details 1. Start pi in this repo and run `/grill <a small but real topic>` — for example a decision you can answer quickly. Do not use a throwaway topic with no substance; the style is most likely to break in a reply that explains or summarises. 2. Answer three questions so the session produces at least three assistant replies. 3. Run `/checkpoint` or `/checkpoint chat` and read the checkpoint. 4. Check each reply and the checkpoint against every rule category: - sentence length: about 20 words or fewer for an instruction, about 25 words or fewer for a description; - one idea per sentence; - active voice; - simple tenses only (past, present, future, imperative), no `-ing` form where a simple tense works; - one word for one meaning, no synonym trading; - no phrasal verb where one verb does the work; - noun groups of three words or fewer; - short, common words; - articles kept. 5. Mark each sentence as **exempt** when it is one of: a probing question or one of its alternatives; code, a file path, an identifier, a tool name, or a command; text quoted from the user. Exempt text is expected to break the rules and is not a finding. 6. Record the findings. Append a "Live STE verification" section to `extensions/grillme/findings.md` (create the file if it does not exist) or add a new short note file next to it. For each finding, record: - the rule broken; - the offending sentence; - which reply or which checkpoint section it came from; - whether it is a genuine break or a borderline case. For each exemption leak, record the sentence, which rule the leak worked around, and whether the sentence was a probing question or ordinary prose. 7. Add a short verdict: which rules held, which did not, and whether the style is usable as shipped. Do not edit `ste-style.ts` or any rule text here — the notes are input for a later review. ## Acceptance Criteria - [ ] A `/grill` session was run with at least three assistant replies. - [ ] The checkpoint was read. - [ ] Every rule category was checked against the observed text, and each finding names the rule, the sentence, and its source. - [ ] Every exemption leak (plain prose written as a probing question) is recorded, with the sentence and a verdict. - [ ] The findings are written into the repo, in `extensions/grillme/findings.md` or a note file next to it. - [ ] The notes carry a verdict: which rules held, which did not, and whether the style is usable as shipped. - [ ] No rule text, prompt text, or test was changed in this step. ## Test Plan The session itself is the test. Run: ```bash cd extensions/grillme && npm test ``` Expected: green, and unchanged by this step. Then, after the notes are written, verify the notes exist and name the rules: ```bash grep -n "Live STE verification" extensions/grillme/findings.md ``` Expected: the new section heading. ```bash git diff --stat ``` Expected: only the notes file changed — no source or test file.

david added this to the Grill Me STE: M5 - Verification milestone

2026-10-09 21:20:58 +00:00

david added the

enhancement

label

2026-10-09 21:20:58 +00:00

david self-assigned this

2026-10-09 21:20:58 +00:00

david referenced this issue from a pull request that will close it,

2026-10-09 22:40:43 +00:00

issue-299: Verify the STE style live in a short /grill session and record the observations #309

david closed this issue