The system behind my research has grown past the point where a description in passing works. This document explains what the system is made of, how a score is produced, how I test whether the scores mean anything, and which systems process student text when the instrument runs. It assumes no background in machine learning or programming, and the glossary at the end collects the technical terms. The results it mentions are current at the last-updated date shown above, and none of them is final.
The system is a research instrument that produces estimates of essay quality. It produces these estimates through several independent methods, so that the methods can check each other. I designed it, built it, and run it myself. It is a tool for studying assessment, and specifically for studying what happens when different ways of judging writing agree or disagree. It is not a product that grades students. No student receives a grade or feedback from it, and no teacher uses it in live assessment work.
A scoring method must learn from essays that already have trusted human scores, and it needs many of them. The system works with two collections of essays.
The first collection is ELLIPSE, a public research corpus. A corpus is a collection of texts assembled for study. ELLIPSE contains roughly 6,500 essays written by school-age learners of English in the United States. Trained human raters scored each essay on a scale from 1 to 5, in half-point increments. Two trained raters scored every essay. The corpus was released for research use. It is anonymized, and it contains no identifiable students.
Because two trained raters scored every essay independently, the corpus also shows how often and how far trained raters disagree with each other. That record of human disagreement becomes the benchmark against which the system's agreement must be read, and I return to it in the section on evaluation.
The second collection is Swedish. The current scored research set contains 281 prompt-resolvable essays written by my own upper-secondary students across the three course levels referred to here by their familiar names: English 5, English 6, and English 7. The current course names are English 1, English 2, and English 3, respectively. Both naming systems occur in the retained data because the essays span the reform. The stored essays use anonymous identifiers. Only my separate private records associate those identifiers with students.
ELLIPSE provides thousands of essays, each rated twice by trained raters, so it is the material the methods learn from. The Swedish essays do not have comparable double-rated reference scores, and 281 essays are far too few to learn from. The Swedish essays therefore serve a different purpose. They provide transfer evidence, which means evidence about whether a method learned from one population also applies to another. The system learns nothing from the Swedish essays.
The system estimates essay quality through three separate methods. I call them lanes. The lanes are kept apart on purpose. Each lane uses different evidence about writing quality, so their agreement is informative and their disagreement is evidence. Figure 1 shows the three lanes and how their outputs are compared and combined.
Figure 1. How a score is produced. One essay enters the transparent lane and the hybrid lane, and essay pairs enter the comparative-judgment lane. Each lane produces its own output, and the outputs are compared and combined only after every lane has finished. The comparative-judgment lane is never trained on the human scores.
The first is the transparent lane. My working name for it is the white-box lane. Here the computer counts and measures properties of the essay text. The properties are ones that writing research has long linked to proficiency. It uses 31 such measures, organized into several groups. One family counts spelling, grammar, and punctuation errors and expresses them as rates per hundred words, using an error-checking program. Another family measures sentence structure. It relies on a parser, which is a program that works out the grammatical structure of each sentence, so that depth and complexity of that structure can be measured. A third family measures variety of vocabulary, for example how much of the essay repeats the same words and how much of it draws on rarer words rather than the most common ones. Related measures draw on published word-norm studies, which record how familiar and how concrete individual words are to readers, so that the essay's vocabulary can be characterized beyond simple variety. A fourth family measures whether the essay stays on the assigned task, by comparing the content of the essay with the content of the task description. A fifth family measures how the essay is organized as a whole, for example how its parts connect, how it holds together across paragraph boundaries, and in what order its argumentative functions appear. Every measure can be inspected on its own, and every measure has a stated rationale in writing research.
The 31 measures must then become a single score. This is done by a model. A model, in this document, is a computer program that has learned a rule from examples instead of being given the rule directly. The learning works as follows. The program is shown the 31 measures for each ELLIPSE essay together with the human score for that essay. Across thousands of essays, it adjusts how much weight each measure receives to minimize the difference between its estimates and the human scores. A model that learns to predict a number in this way is called a regression model. The regression model used here is small. It could run on a phone. It contains no language ability of its own. It only weighs the 31 measurements it is given.
The second adds a different kind of evidence. It uses a language model. A language model is a program trained on very large amounts of text until it can process and produce language. Here, a language model reads the whole essay and condenses its reading into a list of 768 numbers. Such a list is called an embedding. An individual number is not meaningful to a person, but together the 768 numbers represent patterns of meaning, phrasing, and fluency that are absent from the 31 counted measures. A regression model of the same kind as in the transparent lane then learns, again from the human-scored ELLIPSE essays, how to combine the 768 numbers with 38 scalar measures into one score.
The hybrid lane agrees with human raters more closely than the transparent lane does. It is also less inspectable, because no one of the 768 numbers can be identified and explained. For a transparent-lane estimate, I can state which measures contributed most. For a hybrid-lane estimate, I can state only what the complete language-model representation contributed. I keep the lanes paired because their difference in agreement estimates how much human scoring behavior the named measures fail to represent.
The third uses comparative judgment, the method established in the assessment literature, with a language model as the judge. Instead of scoring one essay on a scale, a large language model is shown two essays and asked which one is better. This is repeated many times over many pairs. A long-established statistical method then turns the wins and losses into a ranking of all the essays. The method is the Bradley-Terry model, which comes from psychometrics and is the same machinery that underlies human comparative judgment studies. It uses no counted measures and no embeddings, and it is never trained on the ELLIPSE scores. It arrives at its ranking from pairwise decisions alone.
The comparative-judgment literature argues that deciding which of two texts is better is an easier and more consistent act than placing one text on an absolute scale, and that reliability comes from accumulating many such decisions. The practical obstacle for human panels is the number of comparisons required. A model judge removes that obstacle, because it can make thousands of comparisons at negligible cost. A model judge must itself be validated, and validating it is part of what the project studies.
Two essays that receive the same grade can differ clearly in quality. A scale score cannot state that difference, so absolute scoring collapses the differences within a grade band. A ranking built from many pairwise decisions preserves the order of essays within a band. That within-band information survives even after the ranking is mapped onto grade thresholds.
Writing proficiency is the underlying construct. A test represents the construct only in part, because the test domain is a limited expression of it. A rubric then puts into words what raters should look for. The scale compresses those judgments into a small set of levels. Raters interpret the scale through their experience and norms. Grades finally formalize those interpretations, for example when universities use them in admission or when they serve as evidence of proficiency. The test, rubric, scale, and grade each reduce how much of the construct remains visible. Comparative judgment compares two texts directly within the same test domain, so it represents relative quality without first assigning every essay to one of the scale's few levels.
The three lanes disagree with each other in informative ways, and the pattern of agreement and disagreement is a research object in its own right. The research proposal calls this a hybrid evidence structure. The lanes are also combined. In experiments run on one ELLIPSE writing task as of mid-2026, adding the comparative-judgment ranking to either scorer raised agreement with the human raters, and a simple average of all three lanes agreed with the human raters more closely than any single lane. The combination happens only after each lane has produced its output. The comparative-judgment results never enter the training of the scorers. The purpose of the combination is triangulation, meaning that partly independent methods constrain each other's errors. Independence is what makes the combination informative. The transparent lane uses counted properties, the hybrid lane uses a language model's representation, and the comparative-judgment lane uses pairwise preferences. Similar estimates from all three methods therefore depend on three different forms of evidence. The combination experiments have so far covered only one ELLIPSE writing task, and the improvement must be replicated across tasks before it can be treated as a settled property of the instrument.
A scoring method must never be judged on the essays it learned from. A model tested on its own training material looks better than it is, in the same way that a student who has seen the answer key looks better than they are. The basic safeguard is held-out testing. Some essays are set aside before training begins, and the model never sees them until it is evaluated.
The primary test goes further than holding out essays. Essays are written in response to writing tasks, and a model can quietly learn habits of one task rather than writing quality in general. It can learn, for example, that essays on a particular topic tend to use particular words, and then reward those words instead of writing quality. The test therefore holds out whole tasks: the model is evaluated on essays written for tasks it has never seen, the closest available rehearsal of meeting new material in practice. A portion of ELLIPSE is also kept as a sealed test set. It is opened only at declared checkpoints, so that repeated peeking cannot gradually inflate the results. Between checkpoints, all development runs against the ordinary held-out material, and the sealed portion answers only the question of whether a declared stage of the work holds up.
Agreement with human raters is measured with a statistic called quadratic weighted kappa. In plain terms, it rewards close agreement and penalizes distant disagreement more heavily. Being half a point away from the human score costs little. Being two points away costs a great deal. The statistic runs from 0, meaning agreement no better than chance, to 1, meaning perfect agreement. A second, simpler measure is also tracked, namely the average distance between the model's score and the human score, because one measure can improve while the other worsens. I place more confidence in a change when both improve.
The two trained ELLIPSE raters agree with each other only moderately. Measured over thousands of double-rated essays, their agreement is about 0.58 on that scale. This does not set a hard ceiling for a model, because a model is compared with the raters' combined score rather than with a single rater. When trained human raters agree at 0.58, no model's agreement figure can be read as a measure of truth. Instead, it indicates how closely the model matches human judgments that are themselves variable.
Very few ELLIPSE essays have scores near 1 or 5, so the models have the least material to learn from at the ends of the scale. As a result, the models are most reliable in the middle of the scale and compress their estimates toward the middle at the ends. Diagnostics on the sealed material, as of mid-2026, show that this compression is real but bounded. It is a known and quantified property of the instrument.
ELLIPSE is released with an official division into training essays and test essays. Published systems report their results on the official test essays, so results computed on that material can be compared across research groups. The sealed portion described above is drawn from this official test material. In August 2026 I evaluated the scorers on the official test partition in one declared pass. The scorers were frozen, which means that their training and settings were fixed before the evaluation and nothing was changed afterwards. The evaluation admitted 2,567 of the partition's 2,571 essays. On the agreement statistic described above, the hybrid scorer achieved 0.756 with the broader training set described later in this document and 0.744 without it. Its average distance from the human score was 0.30 scale points.
The official partition also provides a point of comparison with published work. Choi et al. (2026) report an agreement of 0.726 on that partition. My evaluation admitted 2,567 of its 2,571 essays. The systems also differ in their training data and in what they model, so I read the comparison as descriptive rather than as proof that one system is better. This result comes from the regression scorer alone and uses no comparative judgment. The hypothesis for comparative judgment therefore concerns more than accuracy. The hypothesis is that the within-band quality information it preserves can make the instrument one that teachers, and not only researchers, can trust and use.
Five lines of work are open, and each runs from a gap in the current evidence to a hypothesis and a method that tests it.
Ordinary spelling and grammar checkers are built for native-like text. They miss errors that are typical of second-language writers, and they sometimes correct a misspelled word into a real word that is wrong in context. The hypothesis has two parts: error counts are themselves informative measures of proficiency, and a repaired version of the essay should improve every downstream measure that depends on parsing garbled text, because the parser and the embedding both read a repaired sentence better than a broken one. The method separates proposal from acceptance. One language model proposes a correction. A second, independent model sees only the surrounding sentence, the original word, and the proposed correction, and accepts or rejects it. A proposer reviewing its own suggestion would accept its own mistakes. Only clean one-word-to-one-word spelling corrections are promoted into a durable dictionary of verified second-language corrections. Once a correction pair is in the dictionary, applying it is a local lookup that involves no model at all. Ongoing work tests repaired against unrepaired text across both scorers, ends with an evaluation on the sealed material, and includes a human usefulness decision before any of it enters routine use.
An earlier training generation used 3,479 essays. A later, broader training set recovered 1,563 additional eligible essays while keeping the official test partition separate. In the August benchmark, the hybrid scorer's agreement increased from 0.744 to 0.756. Subgroup results were mixed, so the broader training set helped overall but did not solve every weakness across the scale.
The scoring models learn only from absolute scores. The information that absolute scoring collapses, including the differences between essays within a grade band, therefore is absent from their training data. The hypothesis is that the comparative-judgment ranking contains information about essay quality that absolute scores cannot provide, and that providing this relational information to the system should improve both its accuracy and what it can express about a cohort. The method starts from the combination experiments described earlier, which join the ranking with the scorers only after training. The open work studies how relative-quality information can be represented and provided to the system, and what it adds beyond the absolute scores. The ranking should also span writing tasks rather than depend on one, because a signal about writing quality within the same construct should preferably not depend on the task. The evaluation of candidate model judges therefore uses judging instructions that are not tuned to any single model, so that it measures each model's raw ability to discriminate writing quality and what that ability costs.
ELLIPSE scores are defined for one population of learners in the United States. Whether they order Swedish upper-secondary essays sensibly is an open question, and it is the question the 281 scored Swedish essays exist to probe. The retained experiment places the anonymous classroom essays described earlier on ELLIPSE's 1-to-5 scale. The ELLIPSE midpoint is 3. English 5 cohort medians are roughly 3.2 to 3.5, English 6 medians are 3.9 to 4.1, and the usable English 7 cohort median is about 4.3. The difference is largest between English 5 and English 6 and smaller between English 6 and English 7. These are classroom cohorts, not the separate official anchor essays. This public comparison uses English 7 VT2017; the later sample is too small.
The English 7 distance is substantial on this scale. Only about one in twenty ELLIPSE predictions are 4 or higher, compared with about two in three to three in four predictions for English 7 VT2017. This is why a difference of about 1.3 scale points matters: the English 7 cohort is concentrated in a region that is only the thin upper tail of the ELLIPSE predictions.
| Course level and cohort | Essays | Transparent median | Hybrid median | Transparent at 4+ | Hybrid at 4+ |
|---|---|---|---|---|---|
| English 5 level, VT2026 (registered as English 1) | 22 | 3.246 | 3.379 | 9.1% | 18.2% |
| English 5 level, VT2017 | 46 | 3.523 | 3.512 | 4.3% | 23.9% |
| English 6 level, VT2023 A | 60 | 3.917 | 3.960 | 38.3% | 48.3% |
| English 6 level, VT2023 B | 62 | 4.125 | 4.141 | 54.8% | 59.7% |
| English 7 level, VT2017 | 24 | 4.264 | 4.335 | 66.7% | 75.0% |
| English 7 level, VT2018 | 7 (low n) | 4.147 | 4.124 | 85.7% | 57.1% |
Figure 3. Comparison of Swedish course cohorts with ELLIPSE. The ELLIPSE curve is schematic. The course positions summarize the same classroom-cohort comparison described above. The English 7 position uses VT2017; the figure does not use the separate official anchor essays or the very small VT2018 sample.
The two populations differ in a way that may explain the ordered course differences. ELLIPSE is a corpus of designated English learners, and students who become proficient leave that population. The corpus therefore has a proficiency cap. Swedish school English has no such cutoff, because every student takes the courses regardless of proficiency. If this reading is correct, ELLIPSE may overlap most with Swedish Year 9 and the start of English 5, while the higher courses produce increasingly higher score distributions. That possibility connects directly to the supervisory group's work on using automated essay scoring and language technology to support Year 9 assessment. I do not yet have ninth-grade national-test essays, so this is a research hypothesis to test, not a finding. I do not claim that ELLIPSE and Swedish school English share one construct. The work measures the overlap and differences between the scales and identifies the score regions in which the models remain reliable enough to support assessment. The planned analysis adds roughly ten years of national assessment exemplar essays. The exemplars matter because they have official grade designations, so they can anchor the comparison to the Swedish grading standard in a way my own course cohorts cannot.
The comparisons so far measure models against rater scores. The reference that matters for the project is teacher judgment. The planned teacher panel produces a bounded account of the quality construct that Swedish teachers of English apply when they assess these essays. That account is then compared against the exemplar essays, the comparative-judgment ranking, and the ELLIPSE-trained scorers. In this design the instrument is the thing being checked, and teacher judgment is the thing it is checked against. That ordering is a standing commitment of the project. However well the lanes agree with each other or with the ELLIPSE raters, the question that decides whether the instrument can support Swedish assessment work is whether its outputs hold up against the judgment of the teachers who do that work.
Where student text goes depends on how the model is used.
A chat interface is a consumer product in which a person types or uploads text into a conversation window. Whether that content may be used to improve models depends on the service and the user's data controls. OpenAI describes those controls here.
An API differs from a chat interface in the way that decides where text goes. It is an arrangement in which one program answers requests from another program. A request contains exactly the text the sending program puts in it, the answer comes back to the sending program, and there is no conversation window and no chat history. API use is governed by business terms rather than consumer product terms. OpenAI states that API inputs and outputs are not used to train its models by default. The sending program controls what is transmitted, and the external requests described here use this route.
A self-hosted model runs on my home server. The text is processed there and never leaves the building.
The system runs almost entirely in the self-hosted mode. I built a server at home for this project. Its name is Hemma. Hemma runs the language model that produces embeddings, the parser, the error-checking programs, and the local language models, and all large-scale processing of essay text happens on it. The two scoring models themselves are the small regression models described earlier. They run locally on ordinary hardware and could run on a phone.
The retained research records document two categories of text sent to external providers. One is essay text from the public, anonymized ELLIPSE corpus. The pairwise comparative-judgment experiment sent ELLIPSE essay pairs to a provider-hosted OpenAI model in mid-2026. The correction work sends ELLIPSE text to provider-hosted models that propose and adjudicate corrections. It has run on ELLIPSE text only.
The other is de-identified Swedish essay text. The comparative-judgment lane has also judged Swedish student essays against anchor essays. An anchor essay is an essay selected as a fixed reference point, so that other essays can be compared against it. In how it is produced, an anchor essay is an assessment exemplar. Its score does not come from one rater. Several raters assess it and mediate their judgments until they converge, so its score has low measurement error and small rater effects. I use the term anchor because of what the essay does. It anchors the comparisons, and through them the model, in the shared understanding of the test domain that the teachers who assess in it hold. For that work, Swedish student essays and anchor essays have been sent to the same provider-hosted OpenAI model through the same pipeline as the ELLIPSE experiment. Each request contains the judging instructions, the writing task, and essay text under anonymous identifiers. The identifier-to-student record is kept separately and is not part of the request.
Figure 2 shows that retained data-processing contract.
Figure 2. Processing locations and external transmissions. Hemma, my home server, performs all local essay processing. The comparative-judgment and correction requests pass through a single gateway that checks every request in code. These requests contain essay text under anonymous identifiers. Comparative-judgment requests contain ELLIPSE pairs or de-identified Swedish student and anchor essays. Correction requests contain ELLIPSE candidates only. The identifier-to-student record remains offline on my home server and is never transmitted, so no request can connect an essay to a student.
I want to reduce external use further. Open-weight models now exist that could serve as the comparative judge on Hemma. An open-weight model is one whose full internals are published, so that anyone can run it on their own hardware. Models small enough to run on hardware like Hemma trade capability for that locality. Whether such a model can serve as a durable judge on my own hardware is therefore an open question, and I do not present it as a settled plan. The ranking signal does not need to be perfect. It needs to be strong enough to provide the information the instrument uses, and where that threshold lies is not yet known. One of my main current tasks is an evaluation of open-weight models as comparative judges, and it will answer how much judging capability each model offers against its cost.
The system can produce essay-quality estimates from three partly independent methods, and combining them has agreed with trained human raters more closely than any single method in held-out experiments as of mid-2026. On the official ELLIPSE test partition, the hybrid scorer has also agreed with the human raters more closely than a published comparison system (Choi et al., 2026), under the descriptive comparison stated in the section on evaluation. The system's estimates are most reliable in the middle of the 1 to 5 scale, and the weakening at the extremes has been measured and quantified. Its evidence about Swedish student writing is transfer evidence from an American learner corpus, ordered plausibly by course level but not yet anchored to Swedish grading standards. The instrument can therefore support research on assessment, and it cannot yet support assessment decisions, which is the boundary the teacher panel and the exemplar work are designed to test.
Choi, H., Kim, S., Liermann, W., Seong, J., and Huang, J.-X. (2026). Enhancing Automated Essay Scoring With Three Techniques: Two-Stage Fine-Tuning, Score Alignment, and Self-Training. IEEE Access, 14, 117193-117208.
Kinnear, G., Jones, I., and Davies, B. (2025). Comparative judgment as a research tool: A meta-analysis of application and reliability. Behavior Research Methods, 57, 222.
Pollitt, A. (2012). The method of adaptive comparative judgment. Assessment in Education: Principles, Policy & Practice, 19(3), 281-300.