I am developing HuleEdu to investigate how AI-supported assessment can help teachers assess English essays in Swedish upper-secondary education. The instrument produces score estimates and comparisons that teachers can inspect and question.
The rater panel will select anchor essays and discuss their quality. Its decisions, reasons, and disagreements will provide a reference for examining how well the instrument preserves the qualities teachers value in writing.
The scorers learn from human-scored essays in the public ELLIPSE corpus. Scores range from 1 to 5 in half-point steps. The prepared data contain 5,470 training essays and 2,567 official test essays. Two declared training exclusions leave 5,468 essays for the latest scorer evaluation. Test essays are excluded from training and model selection.
A separate Swedish collection contains 281 scored records representing 224 distinct essay texts from English 5, 6, and 7, now called English 1, 2, and 3. Duplicated registrations are excluded from the classroom comparison. These essays provide evidence about transfer to Swedish learners. They are excluded from scorer training and lack the comparable double-rated reference scores used for ELLIPSE.
Each scorer produces a separate estimate. The combined scorer takes a weighted average of those estimates, a method called late fusion.
| Scorer | How it learns |
|---|---|
| Whitebox | Learns from human scores and 30 named linguistic measures of accuracy, sentence structure, vocabulary, relevance, and organization. |
| Embeddings | Learns from human scores and a numerical representation of each essay, produced by DeBERTa-v3 on the research server. |
| Linear RankNet | Learns from pairwise judgments made by Gemma, a language model used as a judge. It estimates relative quality for new essays. |
Evaluation measures agreement with human scores. Quadratic weighted kappa gives greater weight to larger score differences; mean absolute error is the average distance from the human score. Predicting Gemma's choices is measured separately from agreement with human scores.
I evaluated the combined scorer on all 2,567 official test essays after training and selection on 5,468 essays. It reached weighted kappa 0.751 and mean absolute error 0.310. Adding linear RankNet to embeddings and whitebox slightly improved overall results, but increased severe errors among high-scoring essays. The five-arm evaluation reports the complete comparison and error rates.
I tested whether the language corrections used before grammatical analysis change Gemma's judgments of essay quality. Correcting an essay may also change the evidence on which its quality is judged.
Gemma judged 504 pairs under both conditions: original and corrected text. I selected the pairs across 18 writing prompts and several score levels, emphasizing close score differences to examine where the treatment affects comparisons.
Agreement with the higher recorded score was 76.69% for original text and 75.96% for corrected text. The difference was uncertain. The experiment results explain the treatment and results across score levels.
I collected judgments for 50,000 unique essay pairs in two acquisition stages. Gemma judged each pair in both orders, producing 100,000 judgments. The acquisition and performance graphs show when pairs were collected and how results changed with the number used.
I am preparing the comparative-judgment and administration interfaces for testing and the rater panel's work. A stable first iteration supports the main workflows. The current alpha includes the rater interface and the administration interface.
I am refining how essay sets are loaded, how anchor essays are handled, and what information is recorded during assessment. These choices will shape both the panel's experience and the evidence available for studying its assessment work. The interface overview shows the current layout.
The panel will establish the Study 1 anchor set by comparing candidate anchor essays and then discussing the resulting scale. Comparative judgment takes about the same or more teacher time than absolute grading at equal reliability, and it gives a scale that depends less on individual raters.
Most essay processing runs on Hemma, a research server under my control. Some experiments send public ELLIPSE essays or de-identified Swedish essays to external model providers through their APIs. The private records connecting student identities to essay identifiers are stored separately and are excluded from those requests.
The data-handling account identifies the providers, explains what a request contains, and links to the relevant data policies.