Unrestricted-length ELLIPSE experiment

Research Purpose And Boundary

This experiment asked whether the retained raw-inclusive ELLIPSE scorer should also be trained on essays excluded solely by the former 200-to-1,000-word preparation rule. The control contains 3,479 base essays and all 1,563 attributed candidates. The experimental arm adds every otherwise-admissible length extreme: 182 essays below 200 words and 74 above 1,000 words, producing a 5,298-row training population.

The comparison changed only the training population. Both no-L2 feature lanes, the five seeds, prompt-held-out folds, grade-band weighting, early stopping, calibration, prediction mapping, fixed 3,479-row OOF population, and official 2,259-row test population remained unchanged. The 256 added rows never selected a model or calibrated its predictions. Their prompt-aligned expanded OOF result is secondary diagnostic evidence.

The experiment does not change an accepted scorer. It determines whether the two unrestricted-length scorers deserve later selection over the retained 5,042-row inclusive controls.

Evidence And Sources

The retained machine-readable result set is task-hule-20-43-05-unrestricted-length-results-v1/. It contains no essay text. The decisive bindings are:

The official-test headline values below come from the frozen evaluator metrics bound by the manifests. The larger paired comparison artifacts provide the prompt, band, tail, raw-error, and greater-than-one-grade evidence. Their QWK projection uses the comparison framework's observed test-score bounds; the headline QWK uses the fixed ELLIPSE score scale. MAE, adjacency, and severe-error counts are identical under both views.

Findings And Interpretation

Population result

The raw census found 257 essays excluded solely by length: 183 short and 74 long. One short essay has an invalid source Overall category and was excluded by the unchanged scorer-cohort rule. All remaining 256 rows materialized in both feature lanes under parser policy v3. No row was excluded for rater disagreement, unresolved rater identity, identifying-information flags, genre, predicted error, or perceived quality.

Primary fixed-OOF comparison

Negative MAE change is an improvement. Positive QWK and adjacent-accuracy changes are improvements.

On the combined primary comparison, 248 rows improved, 243 worsened, and 2,988 were unchanged in absolute error. The small mean-error improvement and four fewer severe errors are real but do not produce a general performance gain: QWK and adjacent accuracy both decline slightly. The handcrafted lane is also mixed and worsens on MAE and QWK.

The combined fixed-OOF tail view explains the cancellation. The high group improves by 0.02184 MAE and 0.01560 adjacent accuracy, while the low group worsens by 0.02982 MAE and 0.02807 adjacent accuracy. The middle changes little (MAE +0.00118). Handcrafted shows the opposite tail pattern: low MAE improves 0.00351, high MAE worsens 0.01950, and the middle worsens 0.00255.

Secondary evidence on the 256 added rows

The combined scorer learns useful signal for the newly admitted short essays. The long-essay result is neutral to mixed. The handcrafted scorer reduces three severe errors but worsens its average discrimination and error. These rows are diagnostic only; their favorable combined result cannot override the fixed-OOF or official-test comparison.

Unchanged official test

The combined scorer is the primary lane, and it becomes worse on every overall official-test measure. It also creates two additional errors larger than one grade. This outweighs its small fixed-OOF MAE improvement and its good result on the 256 added rows.

The combined official-test tail result is also mixed. The exact-high group (66 essays) improves by 0.01515 mapped MAE and 0.03030 adjacent accuracy. The broader high group (399 essays) improves in raw prediction error but worsens by 0.01629 after mapping. The low group (185 essays) worsens by 0.03243 mapped MAE, and the middle worsens by 0.00418. The added training material therefore helps one narrow high-tail view while harming the low tail and overall calibrated predictions.

Prompt effects are heterogeneous rather than a general improvement. On the combined fixed OOF rows, the largest MAE improvements occur for Benefits of a problem (-0.0645), Imagination (-0.0370), and Career commitment (-0.0305). The largest deteriorations occur for Talents and skills (+0.0875), Creative arts requirement (+0.0652), and Lessons with elementary school students (+0.0303). On the official test, Summer projects improves most (-0.0833), while Praising student work worsens most (+0.0758). Complete prompt tables are retained in the linked JSON artifacts.

Decision implication

The unrestricted-length experiment does not support replacing the retained raw-inclusive combined scorer. The added rows contain learnable information, especially among short essays and some high-tail and prompt strata, but the benefit does not generalize to the unchanged official test. The unrestricted scorers remain separately retained research packages for later comparison; no accepted scorer or registry changes.

Evidence Gaps And Follow-Up

No additional run is required to answer Task 43-05. A later experiment may use the retained packages to test whether a bounded length-aware weighting or mixture can preserve the short/high-tail benefit without the low-tail and official-test regression. That would be a new arm with a new governed task; it is not a reason to alter or prolong this completed unrestricted-inclusion test.