Monitoring Jury Progress

Once a round is open, you need to know whether the panel will finish on time and whether anyone is scoring in a way that will distort the result. Two pages answer those questions: the Rounds tab on the Jury page for how much work is done, and the call's Analytics page for how the panel is scoring. This guide covers both, plus reminders and what to do when a juror drops out.

How far along is the round

Open Admin → Jurying, open your call and go to the Rounds tab. Each round card shows its dates, its status, and Scores Submitted for the round as a whole. That number against your submission count and panel size tells you whether the round is on track.

Jurors can score only while a round is Open. If the round is still in Configuring or Assigning, nobody is scoring yet, and the count staying at zero is the state machine, not the panel.

How is the panel scoring

Open Admin → Calls → your call → Analytics and use the Jury Analytics tab. Four cards summarise the round:

Total Reviews, with an average per juror.
Completion Rate across the panel.
Average Score, with the scale it sits on.
Inter-Rater Reliability, how closely your jurors agree with each other.

Below them, Bias Detection raises alerts for jurors scoring consistently high or low, using an extreme spread, or scoring inconsistently. An alert is a prompt to look, not a verdict. A juror flagged as a low scorer may simply be applying the rubric strictly while the rest of the panel is generous, which is exactly what score normalization exists to handle. See Score Normalization and Results.

The Juror Performance tab gives you the per-juror table: Juror, Reviews, Avg Score, Std Dev, Consistency, Completion and Bias. This is where a juror who has not started shows up as a completion figure near zero.

Reminders

Crafted Call emails jurors who still have work outstanding as a round's end date approaches, at roughly a week, three days, one day and four hours out. Each juror gets each reminder once. The email names how many submissions they have left and links straight into the review screen.

The control is the Scoring Reminders switch on the round, on the Rounds tab, and the round's form lets you tick which times to send. To chase one person now, click Send Reminder on their row on the Members tab; Remind Everyone Who’s Behind sends it to every juror with work left. Each row shows when that juror was last emailed.

When a juror has not started

Check the Members tab first. An invitation still showing Pending means they never accepted, and no reminder about scoring will help. Use Resend on that row.

An accepted juror at zero reviews is a different problem. Contact them directly and ask whether they are stuck on access rather than on the work.

When a juror drops out

On the Members tab, use the remove control on their row.

Their scores are kept. Every score, criterion score and note they recorded stays on the record across all rounds, and the results they contributed to remain valid. A confirmation tells you how many scores were preserved. Removal offers a brief Undo if you clicked the wrong row.

Removal is the right move when someone cannot continue. It is not a way to discard scores you disagree with.

When the deadline is close and scoring is incomplete

Two days before the end date, the Rounds tab warns you if reviews are still owed. The round finishes on its own at its end date, and once it has, it can't be reopened. You have two options, and both are legitimate.

Extend the round before it ends. Edit the round's end date on the Rounds tab while the round is still open, and tell the panel.

Close it anyway. Move the round on and decide with the scores you have. Crafted Call warns you that scoring is incomplete rather than blocking you. Weigh the warning against the shape of the gap: a panel that is 90 percent done across the pool is a sound basis for a decision, while one juror who scored only the first 40 entries alphabetically is not, because the entries they missed carry fewer opinions than the rest.

A round that has been closed cannot be reopened. If you need more scoring, create a new round. Scores from the closed round are preserved and still inform your decisions.

The audit trail

The Audit tab records juror invitations and removals, round transitions, conflicts, and bulk decisions, with timestamps and who acted. It cannot be edited. Use it for post-show documentation, for funder reporting, and for answering questions about how the panel was run.