Red Shell · Common Errors

All mistakes/Section 5: Criteria a correct run can pass

5.2Criteria a correct run cannot pass, or a wrong one can

Costs pointsRubric Criteria - Overall Rubric Quality

The mistake

A criterion stricter than its source fails correct runs: it pins a figure the source leaves open, picks one answer of several, insists on one of two sources, or scopes to a step that lacks the values. A looser one passes wrong runs.

Do this instead

Grade the shape of an answer the source leaves open, accept either source, scope to where the values appear, and add bounds: exactly, only, between.

Where it happened

3 real tasks. Each shows the spot and what it should have been; click any line to open the task there.

the words at faultthe row or field at faultmissing from the taskwhat is righta task file

Example 23Visual Learning · 3 Leg A turns

What is wrongCriterion 23 requires PCR-2026-0688 and its 10 July 2026 acceptance in the change board step. The golden's change board scene shows neither; both sit in an earlier scene.

Leg A turn 3 asks for one more decision step in source_check.mp4, to show that “finding a discrepancy, accepting a change request, and releasing a changed figure are different decisions”.

Where it shows

The change-board step keeps the documented scopes separate: PCR-2026-0688 is a later record against 32-21-07 Figure 2 that was accepted on 10 July 2026…
…(e.g. 32-21-07 accepted (Jul); a 29-11-06 Fig 12 request deferred)…the change board step as the golden built it: no PCR number, no date

What it should have been

source_check.mp4 presents acceptance and release as separate states: the July acceptance of the 32-21-07 Figure 2 change request did not put the change in the manual, which happens only at the 30 September release.

How it breaks the standardThe PCR number and the 10 July date appear in source_check.mp4, but only in the earlier record scene, 26 to 39 seconds, and Leg A turn 3 never asks for them in the new step. So the golden is Not Present on criterion 23 at weight 5. This is Overfitting, which costs points.

The rule it breaksQC spec appendix · Rubric Quality Definitions · Overfitting and Underfitting they incorrectly reject a subset of valid implementations Both are Moderate.

Example 24Property & Space · 5 Leg A turns

What is wrongCriteria 13 and 28 require Aunt Rosalind as the possible guest. The voice note asks about family members, plural and unnamed, and the golden names Tamara Brown-Ellis instead, failing both.

Leg A turn 3 hands over turn-3-voice-note.mp3, which asks about “family members who might stay with us” and names no one. The records support more than one.

Where it shows

combined-home.html identifies Aunt Rosalind as a possible guest.
The trajectory's connected-record result identifies that Aunt Rosalind cannot walk far.
Tamara is the family member the records support as an overnight guest

What it should have been

combined-home.html identifies a family member from the records as a possible overnight guest and states the recorded need that shapes where they sleep.
Criterion 28Delete it
criterion 13 now carries the recorded need

How it breaks the standardThe golden's combined-home.html answers the voice note with Tamara Brown-Ellis, from Contacts, Calendar and Email, and never mentions Rosalind, so the golden is Not Present on both criteria. Pinning one person where the voice note asks for any family member the records support is Overfitting, which costs points.

The rule it breaksQC spec appendix · Rubric Quality Definitions · Overfitting and Underfitting they incorrectly reject a subset of valid implementations Both are Moderate.

Example 25Creative & Media · 5 Leg A turns

What is wrongCriterion 26 accepts the register prices only from the 5 May calendar event. The same list is attached to the 4 May mail, and the golden took it from there.

The nine price changes sit in two records, the 5 May calendar event and price list 5 may.pdf attached to the 4 May mail. Both carry the held prices too.

Where it shows

The trajectory grounds the nine old and new register prices in the 5 May 2026 07:00 PT calendar event "New prices in the register".
The trajectory retrieves the held prices $5.00 and $2.00 from the 4 May mail…
…The list you sent Marguerite the night before ("New price list, into the register tomorrow at seven", attachment price list 5 may.pdf) is the source of truth…

What it should have been

The trajectory grounds the nine old and new register prices in the 5 May register list, from either the calendar event "New prices in the register" or the price list attached to the 4 May mail.

How it breaks the standardLeg A turn 1 names neither record, and either one gives every price. Criterion 29 rewards reading the mail, and a run that takes the whole list from there, as the golden did, fails criterion 26 at weight 5. Pinning one of two valid sources is Overfitting, which costs points.

The rule it breaksQC spec appendix · Rubric Quality Definitions · Overfitting and Underfitting they incorrectly reject a subset of valid implementations Both are Moderate.

Catch it in your own task

For each criterion, name its source line and where your golden shows it, then ask what wrong run still passes.

Where the correct guidance is written

Section titles as they print, so you can search for them in the document.

Guidelines · 5.1 Rubric Fundamentals

Since these are objective rubrics, only requirements that can be evaluated objectively from the scenario should be included.

QC spec appendix · Rubric Quality Definitions · Overfitting and Underfitting

Overfitting: Criteria that are overly specific, inflexible or too rigid - they incorrectly reject a subset of valid implementations. Underfitting: Criteria that are overly broad, permissive or loose - they accept valid implementations, but also incorrectly accept invalid implementations too.

Both are Moderate.

Why it matters

The rubric exists to separate a correct run from an incorrect one. A criterion a correct run cannot satisfy marks down the runs that did the work, and it usually catches your own golden, because the golden followed the evidence while the criterion followed your memory. A criterion with no ceiling lets a run that also took the distractors pass, so the traps the task was built around cost nothing.

Every example taken apart, with the task, how it breaks the standard and what it should have been: 5.2 Criteria a correct run cannot pass, or a wrong one can, in the explanations →