Updated September 22, 2026. Current work focuses on preparing the comparative-judgment application for testing and the rater panel's work. Open the [rater interface](https://hule.education/app/cj) or the [administration interface](https://hule.education/app/cj/admin).

## Rater Interface

The comparison workspace presents two essays side by side. The rater compares them against the assessment criteria and records a judgment. Assessment information remains available from the header, alongside an option to report that the pair cannot be assessed. A report gives one of four reasons: unreadable or incomplete material, a mismatch with the task or criteria, recognition or a conflict of interest, or another reason.

[![Rater comparison workspace with Text A and Text B side by side.](/evidence/cj-interface/rater-comparison-20260912.png)](/evidence/cj-interface/rater-comparison-20260912.png)

Comparison workspace, September 12, 2026.

In published studies, [a single comparison takes considerably less time than grading the same essays on an absolute scale](/evidence/comparative-judgment-feasibility/#time-per-comparison-compared-with-absolute-grading), but no study has timed comparisons of essays like those in Study 1. The application therefore records how long each pair stays in view, when the rater first selects an essay, and whether the rater changes the selection before submitting. A pilot with the participating teachers will supply the timings.

Some pairs will feel hard. [Between 18 and 22 percent of individual decisions go against the final rank order, and pairs that lie close on the scale feel harder to judge](/evidence/comparative-judgment-feasibility/#implications-for-study-1). The scales in that research still reached reliabilities of .74–.86.

## Administration Interface

Assessment setup moves through three steps: assessment details, essay selection, and review. The review screen brings the essay text together with the selected essays, enrolled raters, settings, and assessment materials. The administrator can edit the text and review the assessment before publishing it.

[![Administration assessment review with the essay editor and panels for selection, settings, materials, and raters.](/evidence/cj-interface/admin-review-20260912.png)](/evidence/cj-interface/admin-review-20260912.png)

Assessment review and essay editing, September 12, 2026.

An assessment's settings determine how pairs are selected and when the assessment stops. Pairs are either spread across all essays or chosen adaptively where they add most information to the scale. An assessment stops after a set number of comparisons, when every essay has appeared in a minimum number of comparisons, or when the scale reaches a target reliability. [A reliability of .70 takes 10–14 comparisons per essay and .90 takes 26–37](/evidence/comparative-judgment-feasibility/#total-time-for-a-reliable-scale). Stopping at a target reliability requires the non-adaptive selection; [adaptive pairing can inflate reported reliability](/evidence/comparative-judgment-feasibility/#limits).

## Preparing for the Panel's Work

The main assessment and administration workflows are in place. Current work refines how experiments and essay sets are loaded, how anchor essays are handled, and what information is recorded during assessment. Testing will help shape these choices before the panel begins its work.

Anchor sets are planned but not yet part of the application. The plan follows [two precedents in which a scale built through pairwise comparison becomes the reference for placing later essays](/evidence/comparative-judgment-feasibility/#use-in-high-stakes-and-low-stakes-assessment): a New Zealand statistics assessment in which official boundary scripts were judged alongside the student scripts, and a two-stage classroom method in Australia. For 40 candidate anchor essays and six teachers, [a reliability of .80 would take about 1.7–4.0 hours per teacher](/evidence/comparative-judgment-feasibility/#implications-for-study-1), not counting panel discussion.