A common objection to comparative judgment (CJ) is that it is impractical and takes too much teacher time. A single comparison is quick, but a reliable scale needs many comparisons per essay. At matched reliability, CJ usually takes about the same teacher time as rubric marking, or more. This page describes what a CJ scale records and what it does not, and then reports the time and reliability evidence.
CJ produces a measurement scale: a statistical model of the pairwise decisions places every essay at an estimated position on one continuous scale, with a standard error for each position. The scale shows the order of the essays within and across grade bands and how far apart the essays lie. It is also the continuum on which a panel then locates grade boundaries, using graded reference essays, and measures how wide each band is and where each essay sits within its band. A raw absolute grade records which band an essay belongs to and leaves the essay's place inside the band and the size of the step to the next band unobserved.
For Study 1, this evidence supports a bounded design. A panel of teachers judges a set of candidate anchor essays through CJ. The resulting scale becomes material for mediated panel discussion, through which the teachers establish the anchor set, and the anchor set becomes a reusable reference for placing later essays.
Information in a Comparative Judgment Scale
From Pairwise Decisions to One Scale
In CJ, a judge reads two essays written to the same task and decides which one better demonstrates the intended quality. In an Austrian national EFL test, a subset of judges decided quickly without reading the scripts in detail, and the authors conclude that participants "could judge both fast and reliably" (Sickinger et al., 2025, p. 156).
The decisions are analysed with the Bradley–Terry model, which is the logistic form of Thurstone's law of comparative judgment and a form of the Rasch model. The model gives each essay a position so that the difference between two positions predicts how often the first essay is judged better than the second (Pollitt, 2012, p. 282; Heldsinger & Humphry, 2010, pp. 5–6). Each position comes with a standard error, which states how precisely the decisions available locate that essay; every further comparison of the essay adds information and shrinks the standard error (Pollitt, 2012, p. 283; Kinnear et al., 2025, p. 2). Scale separation reliability (SSR) is the share of the spread in positions that reflects real differences between essays rather than estimation error, and it is read like Cronbach's alpha (Wheadon et al., 2020, p. 53).
Order, Distance, and Grade-Band Boundaries
Compared with raw absolute grades, the scale also records the following.
- Order within and across bands. Every essay receives its own position, whereas a rating scale gives the same score to several essays (Kinnear et al., 2025, p. 2). Two essays that would receive the same grade are still ordered, and their standard errors show whether the difference between them exceeds the uncertainty.
- Distance between essays. The positions lie on an interval scale whose unit and origin are arbitrary: distances between essays can be compared, and the numbers can be rescaled for readability without changing those distances (Heldsinger & Humphry, 2013, p. 226; Humphry & Heldsinger, 2019, p. 511). Humphry and Heldsinger (2019, p. 513) used this property to select exemplars at equal intervals along the scale. On a scale of 55,599 pupils' writing, Wheadon et al. (2020, p. 55) found that the gaps between the mean scores of successive year groups varied in size.
- Band boundaries and band widths. The scale does not locate grade boundaries by itself. Once a panel locates them on the scale with graded reference essays, each band has a measurable width, and each essay's distance from the nearest boundary can be read off. The Curriculum Council of Western Australia set grade boundaries on such a scale to derive grade descriptors (Heldsinger & Humphry, 2010, p. 14). Steedle and Ferrara (2016, pp. 220–221) fixed rubric anchor papers at scores of 1, 2, 3, and 4 on a CJ scale and then estimated the anchors freely. The gap between the anchor 3s and 4s was notably larger than the gaps between the 1s and 2s and between the 2s and 3s, which the authors describe as inconsistent with treating the 1–4 rubric as an equal-interval scale. Some anchors also fell out of the order their expert labels implied, such as a "middle 1" above a "low 2", although the authors note that some of these differences would not be statistically significant. The violated assumptions did not change the results: CJ measures from the anchored and the freely estimated calibrations correlated .998 and .997 (p. 221).
An absolute grade places an essay in an ordered category. Essays in the same band share one grade, a raw grade leaves the size of the step between bands unobserved, and any averaging of grades or conversion to points fixes the steps by assumption. Scaling grades with a Rasch model estimates the steps as category thresholds on the logit scale (Barkaoui, 2008, pp. 32–33), but these thresholds are common to all essays and do not place each essay within its band. Where each band begins rests on the rater's internal standard: a rater who awards a grade compares the essay with imagined essays, which opens the way to differences in personal standard, in how finely the rater differentiates, and in how the rater interprets the scale (Pollitt, 2012, p. 293). Rater severity, central tendency, restriction of range, and halo effects are well documented in rating data (Eckes, 2017, pp. 444–445). In a pairwise decision, the judge applies one standard to both essays at once, so differences in severity and in spread cancel out of the decision (Pollitt, 2012, p. 287; Humphry & Heldsinger, 2019, p. 505). The source of error that remains is how judges conceive quality, which is a question of validity (Pollitt, 2012, p. 293).
Swedish national-test studies show the same pattern in practice. When three external raters and the students' teachers graded 100 English national-test texts, their rank orders correlated .86–.93, while only 8 of the 100 texts received the same grade from all four (Erickson, 2009, pp. 14–16). In a national re-marking, 57 percent of the re-marked scripts received a different grade from the re-marker than from the school's teacher (Skolinspektionen, 2021, pp. 4, 9). In Coertjens et al. (2017), the share of score variance attributable to raters was 21.9 percent for rubrics and 14.6 percent for CJ. In Sickinger et al. (2025), the teachers' judgments were more consistent under CJ than under analytic rating in every group except one teacher without experience of the marking scheme.
Rubric Scaling
Rubric scores can also be placed on an interval scale. Many-facet Rasch measurement (MFRM) models each rubric score as a function of essay quality, task difficulty, and rater severity, and reports all three on a common logit scale (Sugita, 2012, p. 65; Barkaoui, 2008, p. 32). In Sugita's (2012, p. 69) study of ten raters, the most severe and the most lenient rater differed by 3.11 logits, or 0.86 of one grade on the rating scale, and MFRM measures adjust for such differences. Interval-level measures and severity adjustment are therefore available for rubric data once it is scaled. Compared with MFRM-scaled rubric scores, CJ differs in three respects:
- Severity within each decision. In a pairwise decision the judge's severity applies to both essays and cancels out (Pollitt, 2012, p. 287), and pairwise decisions leave little room for rater harshness (McGrane et al., 2018, accepted manuscript p. 5); MFRM instead estimates severity and adjusts for it. Earlier research, as Sugita (2012, p. 61) summarises it, finds that rater training reduces extreme severity but does not bring raters to the same severity.
- Separation within bands under holistic grading. CJ separates essays within a band as finely as the number of comparisons allows. In a Rasch analysis, including MFRM, raw scores are sufficient statistics for an essay's measure (Barkaoui, 2008, p. 32). By my inference, essays that receive the same holistic grade from the same raters therefore receive the same MFRM measure, so MFRM of holistic grades separates essays within a band only where raters disagree. With different raters or several analytic criteria, MFRM also separates essays within a band, so this advantage applies to holistic grading. No available study measures within-band separation directly, and at typical comparison counts CJ separation is limited too: at an SSR of .80, each essay's standard error is about 0.45 of the standard deviation of positions (my calculation from the definition of SSR), so the standard errors of essays in the same band often overlap.
- Placement of exemplars on the scale. Holistic comparisons place candidate exemplars on one scale without a rubric developed beforehand (Heldsinger & Humphry, 2010, p. 6), and the calibrated positions then serve as reference points for assessing new essays (Heldsinger & Humphry, 2013, p. 226; Humphry & Heldsinger, 2019, pp. 505, 511).
Three matters remain outside the CJ scale. The scale has an arbitrary origin, so positions become grades only after boundaries are located with graded reference essays, as in the New Zealand studies (Marshall et al., 2020, pp. 55, 57–58). Judges still differ in what they value (Pollitt, 2012, p. 293; Lesterhuis et al., 2022, pp. 7–8). When teachers later assess new essays against calibrated exemplars, rater harshness returns as a source of variation: in Humphry and Heldsinger (2020, p. 8), variation in harshness was generally modest relative to the spread of performances, but some teachers gave average scores about 30 scale points below the other raters.
Limits of the Scale
Positions and distances are model estimates with standard errors, not observed quantities. They hold as far as the decisions fit the model, and the residuals of each decision can be summed per judge and per essay to find judges and essays that the model fits poorly (Pollitt, 2012, pp. 289–290). Under adaptive pairing, where each new pair is chosen from interim results, the reported SSR can be inflated (Verhavert et al., 2019, p. 543; Kinnear et al., 2025, p. 4).
The validity of the scale depends on whether judges attend to the intended construct. Among 64 assessors of argumentative texts, all assessor types based their decisions mainly on argumentation and organisation, and they differed in how often they considered language use, sources, and layout; the authors argue that combining several assessor types broadens the construct the scale represents (Lesterhuis et al., 2022, pp. 7–8). For 25 L2 English argumentative essays judged by three groups, 32 of 6,792 decision comments concerned construct-irrelevant features (Thwaites et al., 2025, accepted manuscript p. 8). The groups still differed in the breadth and depth of the qualities they commented on, and the trained raters' CJ scale reached an SSR of .86 and correlated ρ = .67 with an MFRM-scaled rubric scale of the same essays (Thwaites et al., 2025, accepted manuscript pp. 10–11). For L2 essays of this kind, a CJ scale and a scaled rubric scale order the essays in related but different ways, and the correlation alone does not show which order is more valid.
Time per Comparison Compared With Absolute Grading
Studies that timed both methods on the same or similar material report the following. A comparison involves two texts; the absolute-grading column refers to one text.
| Study and material | CJ, time per comparison | Absolute grading, time per text |
|---|---|---|
| Coertjens et al. (2017): Dutch argumentative texts, upper-secondary, at most one A4 page | about 2.6 min (1:17 per text in the first comparison) | 7:08 (first rubric rating) |
| Benton & Gallacher (2018): GCSE English essays, about 500 words each | 3.5 min (mean) | about 7 min |
| Steedle & Ferrara (2016): US grade-11 state essays, trained teachers | 62 s (median) | 119 s (mean) |
| Marshall et al. (2020): New Zealand NCEA English reports, at least 350 words | about 84 s (derived from 7 min judge time per script) | about 7.2 min |
| Bouwer et al. (2024): Flemish primary descriptive texts | about 64 s | about 79 s (rating with benchmarks) |
| Sickinger et al. (2025): Austrian Year 8 national EFL test, A2 and B1 tasks | 34 s and 51 s (medians) | 240 s and 420 s (self-reported) |
| McMahon & Jones (2015): Irish chemistry test responses, students aged 14–15 | 33 s (mean) | about 51 s |
| Pollitt (2012): English primary writing, ages 9–11, two pieces per script | 195 s (mean of the faster half of judges; may include idle time) | 15–20 min (reported as normal) |
Coertjens et al. (2017) summarize the first judgment of a text as 5.5 times faster through pairwise comparison than through a rubric. Comparison time follows reading time: Benton and Gallacher (2018) note that 3.5 minutes for a pair of 500-word essays is close to normal on-screen reading speed. For the 250–600-word English essays in Study 1, this reading rate gives an estimate of 1.75–4.2 minutes per comparison. The estimate transfers an L1 rate to L2 essays and has to be checked in a pilot with the participating teachers; the comparison application records how long each pair stays in view, so the pilot supplies these timings. For paragraphs of about 150 words written by learners of English, Şahin (2021) reports 4.17 minutes per comparison for instructors and expects full essays to take two to three times as long, which would give about 8.3–12.5 minutes per comparison.
Total Time for a Reliable Scale
The time advantage of a single comparison does not carry over to total cost, because each text must appear in many comparisons. Across 49 CJ assessments, a scale separation reliability of .70 required 10–14 comparisons per text and .90 required 26–37 (Verhavert et al., 2019).
When both methods are compared at equal reliability, CJ takes about the same time or more. In Coertjens et al. (2017), 10 comparisons per text reached a reliability of .69 in 10:36 minutes of judge time per text, while two rubric ratings reached .67 in 11:34. At higher levels the two methods also followed nearly the same time curve, and the authors conclude that both methods require almost the same time investment to reach equally stable rank orders. Steedle and Ferrara (2016) reached a reliability above .80 at about nine judgments per essay; the authors put this at 279 seconds per essay with the median comparison time, against 239 seconds for double rubric scoring at the mean rubric time, or 17 percent more time. The mean comparison time of 93.5 seconds gives about 421 seconds per essay, or roughly 76 percent more. Both figures hold only at a reliability of .80 (pp. 220–222). With all judgments, about 18 per essay, CJ reached .89 and .90 (p. 220), a level comparable to double rubric scoring, whose two scorers correlated .81 and .85 (p. 216; the authors note that such correlations tend to be inflated), or about .90 and .92 for two scores combined by the Spearman–Brown formula. At that level the same calculation gives 558 seconds per essay with the median comparison time (18 × 62.0 ÷ 2) and about 842 seconds with the mean (18 × 93.5 ÷ 2), or about 2.3 and 3.5 times the 239 seconds of double rubric scoring. The combined rubric reliabilities and these multiples are my calculations from the paper's numbers; the authors do not report them. Bouwer et al. (2024) warn that the total time needed for reliable CJ scores can exceed the time needed for reliable absolute ratings.
A study that compared CJ with single marking, not at equal reliability, found a larger gap. In McMahon and Jones (2015), five teachers made 1,550 judgments of 154 science test responses, about 20 per response, and reached a reliability of .87. The judging took about 14 hours, while marking every response once took about 3 hours; the authors note that the judging took more than twice as long as even double marking would have, about 6 hours.
At matched reliability, the extra time for CJ ranges from none to large, and the result is the scale described above: a position and distance for every essay on one continuum where band boundaries can then be located, with rater severity removed from each decision. The number of comparisons remains the main practical cost at scale, and recruiting enough judges can itself be difficult, time-consuming, and expensive (Kinnear et al., 2025, p. 4). In this project, a language model makes the same pairwise comparisons as a judge (an LLM-as-judge); the project's hypothesis is that this approach addresses the cost. In the Study 1 design, human comparisons are limited to the candidate anchor essays; they are not repeated for every new set of student essays.
Use in High-Stakes and Low-Stakes Assessment
CJ is not confined to research settings. It has been piloted in examination awarding and grade-boundary setting, evaluated on national and state test material, and used in moderation and school-level assessment of writing. The high-stakes uses reported here are pilots and evaluations; the school-level uses include operational assessment at scale.
| Setting | Use | Result |
|---|---|---|
| Examinations and national or state tests | ||
| England, examination awarding (Ofqual, 2019) | Paired CJ piloted for maintaining grade standards across exam years | At about ten judgments per script, estimated awarding workload similar to current practice (6–16 judge-days); roughly double at higher judgment counts |
| England, OCR examinations (Benton et al., 2022) | CJ pilots for setting grade boundaries | About 30 judge-hours per exercise; 13–139 hours across studies |
| Austria, Year 8 national EFL test (Sickinger et al., 2025) | 27 teachers judged national-test scripts without prior CJ training | Reliability .89–.92; correlation .77–.83 with expert analytic scores |
| United States, grade-11 state essays (Steedle & Ferrara, 2016) | Trained teachers judged state-assessment essays in an evaluation of CJ as a scoring method | Reliability .89 and .90; above .80 at about nine judgments per essay |
| New Zealand, NCEA English (Marshall et al., 2020) | Teachers and moderators judged students' summative reports; grade boundaries were located afterward by grading six scripts near each boundary traditionally and finding their positions in the ranking | Reliability .85; judge time about 7 min per script, similar to traditional marking |
| England, moderation of non-exam assessment (Vidal Rodeiro & Chambers, 2022) | Six moderators compared candidate portfolios | Reliability .76; correlation .85 with marks |
| School and classroom assessment | ||
| England, primary writing (Wheadon et al., 2020) | Teachers in participating schools judged the writing of 55,599 pupils on the No More Marking platform | Median 11–41 s per decision; 88–330 decisions per hour |
| Australia, primary writing (Heldsinger & Humphry, 2010) | 20 raters, 17 of them classroom teachers, at one primary school judged 30 narrative scripts in about three hours | Reliability .98; correlation .92 with independent marking against the state testing program's rubric |
| Australia, classroom essay assessment (Humphry & Heldsinger, 2019, 2020) | Teachers first calibrate exemplars through pairwise comparison, then assess new essays against the calibrated exemplars | Reliability .96 in the first, pairwise stage, with 23 teachers after 30 minutes of training |
| Research comparison | ||
| Flanders, primary writing (Bouwer et al., 2024) | 14 undergraduate students rated primary-school texts by benchmark rating or by CJ, with or without CJ training | Reliability .82 with training and .79 without, at 15 comparisons per text |
The two-stage method of Humphry and Heldsinger (2019, 2020) and the statistics study of Marshall et al. (2020), in which six official boundary scripts were judged alongside the student scripts and fell in the correct order, are the closest precedents for Study 1: a scale produced through pairwise comparison becomes the reference against which later essays are placed.
Implications for Study 1
Study 1 examines how teachers collectively mediate essay quality into an anchor set. CJ gives this mediation a common starting point: one scale built from every teacher's relative judgments. On that scale the panel can see which candidate essays lie near the top, middle, and bottom of each grade band, where it places the boundaries between bands, whether the bands are of similar width, and which positions are too uncertain to decide on. The scale also shows where the panel's judgments converge and where they do not. Disagreement is expected: in van Daal et al. (2017), 18–22 percent of individual decisions went against the final rank order, while the scales still reached reliabilities of .74–.86. Judges found pairs harder when the two texts lay close on the final scale, and decisions that went against the final order felt hard however far apart the texts lay. The essays that fit the scale poorly, including discordant texts, are therefore likely to be where the panel discussion does most of its work. The discussion then settles which essays represent which grade levels and records the teachers' rationales. The resulting anchor set carries its scale positions with it, so later essays can be placed against calibrated reference points in the way the two-stage method describes.
The panel workload can be estimated in advance. For illustration, the table assumes 40 candidate anchor essays, a panel of six teachers, random pairing, and 1.75–4.2 minutes per comparison. Panel discussion time is not included. The comparison counts come from assessments with random pairing; the comparison application stops at a target reliability only with its non-adaptive pair selection, which spreads comparisons across the essays instead of targeting them from interim results.
| Target reliability | Comparisons per essay | Comparisons in total | Panel hours | Hours per teacher |
|---|---|---|---|---|
| .70 | 10–14 | 200–280 | 5.8–19.6 | 1.0–3.3 |
| .80 | about 17 | 340 | 9.9–23.8 | 1.7–4.0 |
| .90 | 26–37 | 520–740 | 15.2–51.8 | 2.5–8.6 |
At the slower rate suggested by Şahin (2021), the .80 level would require 47–71 panel hours, or 8–12 hours per teacher. For comparison, double absolute grading of the same 40 essays at about 7 minutes per essay takes about 9 hours, and in Coertjens et al. (2017) two ratings per text reached a reliability of .67. The time cost of the CJ stage is therefore of the same order as careful double grading of the candidate essays, and it is paid once, when the anchor set is established. The double grading would yield two grades per essay; the CJ stage yields positions and distances on one scale, on which the panel then locates band boundaries that it can discuss and reuse.
Limits
- No available study times the comparison of 250–600-word essays written by Swedish upper-secondary students in L2 English. The estimates above transfer L1 reading rates and must be replaced by timings from a pilot with the participating teachers.
- Estimates of total cost range from parity to substantially more time for CJ: Coertjens et al. (2017) report parity, Steedle and Ferrara (2016) 17 percent more time for CJ at a reliability of .80, when median comparison time is set against mean rubric time (about 76 percent more with mean times on both sides, and about 2.3 to 3.5 times as much at the .89–.90 reached with all judgments, by my calculation from the paper's numbers), McMahon and Jones (2015) about 14 hours of judging against about 3 hours of single marking, and Bouwer et al. (2024) warn that CJ can take longer.
- Scale reliability is not agreement between raters, and it does not by itself establish validity. Adaptive pairing can also inflate reported reliability (Verhavert et al., 2019).
- Distances and band widths are model estimates for one panel and one set of essays. No available study shows whether a Swedish teacher panel's scale reproduces the national grade levels, or whether band widths found by one panel recur with another.
- Rubric scores scaled with MFRM also yield interval-level measures with rater severity adjusted. The advantage claimed here for CJ is severity removed from each decision, finer within-band separation than MFRM of holistic grades, and direct placement of exemplars, not interval measurement as such.
- The comparison application estimates positions and standard errors but does not yet compute judge or essay fit statistics, so poorly fitting essays are identified in panel discussion rather than by the model.
References
Barkaoui, K. (2008). Effects of scoring method and rater experience on ESL essay rating processes and outcomes [Doctoral dissertation, University of Toronto].
Benton, T., & Gallacher, T. (2018). Is comparative judgement just a quick form of multiple marking? Research Matters, 26, 22–28. https://www.cambridgeassessment.org.uk/Images/514987-is-comparative-judgement-just-a-quick-form-of-multiple-marking-.pdf
Benton, T., Gill, T., Hughes, S., & Leech, T. (2022). A summary of OCR's pilots of the use of comparative judgement in setting grade boundaries. Research Matters, 33. https://doi.org/10.17863/CAM.100424
Bouwer, R., Lesterhuis, M., De Smedt, F., Van Keer, H., & De Maeyer, S. (2024). Comparative approaches to the assessment of writing: Reliability and validity of benchmark rating and comparative judgement. Journal of Writing Research, 15(3), 497–518. https://doi.org/10.17239/jowr-2024.15.03.03
Coertjens, L., Lesterhuis, M., Verhavert, S., Van Gasse, R., & De Maeyer, S. (2017). Teksten beoordelen met criterialijsten of via paarsgewijze vergelijking: Een afweging van betrouwbaarheid en tijdsinvestering. Pedagogische Studiën, 94(4), 283–303. https://hdl.handle.net/2078.1/192419
Eckes, T. (2017). Rater effects: Advances in item response modeling of human ratings – Part I. Psychological Test and Assessment Modeling, 59(4), 443–452.
Erickson, G. (2009). Nationella prov i engelska – en studie av bedömarsamstämmighet. Göteborgs universitet. https://gupea.ub.gu.se/bitstream/handle/2077/21781/gupea_2077_21781_1.pdf
Heldsinger, S., & Humphry, S. (2010). Using the method of pairwise comparison to obtain reliable teacher assessments. The Australian Educational Researcher, 37(2), 1–19. https://doi.org/10.1007/BF03216919
Heldsinger, S. A., & Humphry, S. M. (2013). Using calibrated exemplars in the teacher-assessment of writing: An empirical study. Educational Research, 55(3), 219–235. https://doi.org/10.1080/00131881.2013.825159
Humphry, S. M., & Heldsinger, S. (2019). A two-stage method for classroom assessments of essay writing. Journal of Educational Measurement, 56(4), 505–520. https://doi.org/10.1111/jedm.12223
Humphry, S. M., & Heldsinger, S. (2020). A two-stage method for obtaining reliable teacher assessments of writing. Frontiers in Education, 5, 6. https://doi.org/10.3389/feduc.2020.00006
Kinnear, G., Jones, I., & Davies, B. (2025). Comparative judgement as a research tool: A meta-analysis of application and reliability. Behavior Research Methods, 57(8), Article 222. https://doi.org/10.3758/s13428-025-02744-w
Lesterhuis, M., Bouwer, R., van Daal, T., Donche, V., & De Maeyer, S. (2022). Validity of comparative judgment scores: How assessors evaluate aspects of text quality when comparing argumentative texts. Frontiers in Education, 7, 823895. https://doi.org/10.3389/feduc.2022.823895
Marshall, N., Shaw, K., Hunter, J., & Jones, I. (2020). Assessment by comparative judgement: An application to secondary statistics and English in New Zealand. New Zealand Journal of Educational Studies, 55(1), 49–71. https://doi.org/10.1007/s40841-020-00163-3
McGrane, J., Humphry, S., & Heldsinger, S. (2018). Applying a Thurstonian, two-stage method in the standardized assessment of writing. Applied Measurement in Education, 31(4), 297–311. https://doi.org/10.1080/08957347.2018.1495216
McMahon, S., & Jones, I. (2015). A comparative judgement approach to teacher assessment. Assessment in Education: Principles, Policy & Practice, 22(3), 368–389. https://doi.org/10.1080/0969594X.2014.978839
Ofqual. (2019). Improving awarding: 2018/2019 pilots (Ofqual/19/6575). Office of Qualifications and Examinations Regulation. https://assets.publishing.service.gov.uk/media/5df2419340f0b6094e25ac37/Improving_awarding_-_FINAL196575.pdf
Pollitt, A. (2012). The method of adaptive comparative judgement. Assessment in Education: Principles, Policy & Practice, 19(3), 281–300. https://doi.org/10.1080/0969594X.2012.665354
Şahin, M. G. (2021). Feasibility of using comparative judgement and student judges to assess writing performance of English language learners. Journal of Pedagogical Research, 5(4), 140–154. https://doi.org/10.33902/JPR.2021474154
Sickinger, R., Brunfaut, T., & Pill, J. (2025). Comparative judgement for evaluating young learners' EFL writing performances: Reliability and teacher perceptions of holistic and dimension-based judgements. Language Testing, 42(2), 137–166. https://doi.org/10.1177/02655322241288847
Skolinspektionen. (2021). Ombedömning av nationella prov 2019. Skolinspektionen. https://www.skolinspektionen.se/globalassets/publikationssok/granskningsrapporter/omrattning/2021/ombedomning-av-nationella-prov-2019.pdf
Steedle, J. T., & Ferrara, S. (2016). Evaluating comparative judgment as an approach to essay scoring. Applied Measurement in Education, 29(3), 211–223. https://doi.org/10.1080/08957347.2016.1171769
Sugita, Y. (2012). Effects of rater training on raters' severity, consistency, and biased interactions in a task-based writing assessment. JLTA Journal, 15, 61–80. https://doi.org/10.20622/jltajournal.15.0_61
Thwaites, P., Jadoulle, P., & Paquot, M. (2025). Comparative judgment in L2 writing assessment: Reliability and validity across crowdsourced, community-driven, and trained rater groups of judges. Assessing Writing, 65, 100937. https://doi.org/10.1016/j.asw.2025.100937
van Daal, T., Lesterhuis, M., Coertjens, L., van de Kamp, M.-T., Donche, V., & De Maeyer, S. (2017). The complexity of assessing student work using comparative judgment: The moderating role of decision accuracy. Frontiers in Education, 2, 44. https://doi.org/10.3389/feduc.2017.00044
Verhavert, S., Bouwer, R., Donche, V., & De Maeyer, S. (2019). A meta-analysis on the reliability of comparative judgement. Assessment in Education: Principles, Policy & Practice, 26(5), 541–562. https://doi.org/10.1080/0969594X.2019.1602027
Vidal Rodeiro, C. L., & Chambers, L. (2022). Moderation of non-exam assessments: Is comparative judgement a practical alternative? Research Matters, 33, 100–119. https://doi.org/10.17863/CAM.100428
Wheadon, C., Barmby, P., Christodoulou, D., & Henderson, B. (2020). A comparative judgement approach to the large-scale assessment of primary writing in England. Assessment in Education: Principles, Policy & Practice, 27(1), 46–64. https://doi.org/10.1080/0969594X.2019.1700212