Latest reported comparison completed September 13, 2026. These experiments measure whether Gemma selects the essay with the higher human score. They examine the judgments used to train RankNet.
Parser-Stabilized Text — September 13, 2026
Gemma compared the same 504 pairs from 396 essays across 18 writing prompts. The panel covers human scores from 2.0 to 4.0 and deliberately balances score regions. It does not represent their natural frequency in the corpus.
| Arm | Text shown to Gemma | Agreement with the higher human score |
|---|---|---|
| Full baseline | Original essay text with the full judging instructions | 76.69% |
| Parser-stabilized | Text prepared for grammatical analysis, with the same judging instructions | 75.96% |
Parser-stabilized text is the exact text stream supplied to the grammatical parser after the accepted preparation procedure. That procedure applies eligible LanguageTool repairs and validated lexical repairs. Both essays in each pair receive the treatment. The experiment tests whether changing the text in this way changes the model judge's choices.
Each arm contains three repetitions in both A/B orders: 3,024 judgments per arm. All planned observations were usable. The model was Gemma 4 31B through OpenRouter's DeepInfra FP8 provider profile.
Difference and Uncertainty
The parser-stabilized arm was 0.73 percentage points lower than the baseline. A bootstrap that resampled complete writing-prompt groups gave a 95% interval from −2.15 to +0.63 percentage points. The interval includes zero.
There were 334 changed outcomes: 156 gains and 178 losses relative to the baseline. Results across adjacent score regions were mixed:
| Human-score comparison | Full baseline | Parser-stabilized |
|---|---|---|
| 2.0 versus 2.5 | 71.67% | 69.26% |
| 2.5 versus 3.0 | 79.07% | 80.37% |
| 3.0 versus 3.5 | 72.78% | 71.48% |
| 3.5 versus 4.0 | 60.93% | 59.81% |
The observed difference does not establish an improvement from this text treatment. These results concern the model judge; they do not measure a change in the final essay scorer.
Experiment Scope
This page reports the completed comparison between full-baseline and
full-baseline-parser-stabilized. The broader study includes other instruction
and rubric conditions and separate mechanism probes. That experiment task has
been verified and closed; this page covers only the comparison above.
Sources
Retained record: TASK-HULE-23-10-06, parser-stabilized experiment closeout of
September 13, 2026, and the corresponding ELLIPSE CJ experiment synthesis.
The uncertainty estimate uses 2,000 bootstrap resamples, with a complete prompt
group as the resampling unit.
Scorer evaluation reports agreement with human essay scores. RankNet acquisition and performance reports the existing pair pool and the effect of different pair budgets.