Red Shell · Common Errors

All mistakes/Section 5: Criteria a correct run can pass

5.1A criterion the golden itself fails

Fails the taskRubric Criteria - Overall Rubric QualityMost common right now

The mistake

Your golden fails a criterion that is wrong about the task: a rule only the Desired Outcome states, a phrase at a different length than the golden renders, or a file the inputs archive lacks.

Do this instead

Rewrite an unsupported criterion, or fix the golden with a Leg B cleanup turn. Move an outcome only rule into a prompt and run both legs again, or drop it.

Where it happened

2 real tasks. Each shows the spot and what it should have been; click any line to open the task there.

the words at faultthe row or field at faultmissing from the taskwhat is righta task file

Example 21Creative & Media · 3 Leg A turns

What is wrongCriterion 15 wants `family heirloom` in the logo in place of FAMILY. Leg A turn 3 asks for Tamara's exact phrase, a whole sentence, and the golden logo shows it.

Leg A turn 3 asks for “the exact phrase Tamara used when she joked about Terrence’s truck”. The records give “That truck is a family heirloom at this point.”

Where it shows

Criterion 15+3
In family_photo_tree.html, the logo shows family heirloom beneath the tree in place of the original word FAMILY.
family_photo_tree.html
The golden logo carries the whole sentence.

What it should have been

Criterion 15+3
In family_photo_tree.html, the logo shows That truck is a family heirloom at this point. beneath the tree in place of the original word FAMILY.

How it breaks the standardCriterion 15 copies Desired Outcome item 5, which Model A and Model B never see, instead of Leg A turn 3. Model B, Leg B turn 3, puts Tamara's whole sentence in the logo, as asked, so the golden fails criterion 15. The golden is right; the criterion is what is broken.

The rule it breaksGuidelines · 5.7 Final Rubric Checklist Anything it fails is a broken criterion, not a broken golden.

The full write up of Example 21
Example 22Small Business & Personal Docs · 5 Leg A turns

What is wrongCriteria 24 and 26 grade invoice 21-018 as a record that cannot be cleared, but only the Desired Outcome names it and the golden never mentions it.

Leg A turn 1 asks to check every named party and the tract, and which of the jobs they touch cannot be cleared “because the record is too thin”.

Where it shows

Criterion 24+5
conflict_check.html lists exactly three records as unresolved rather than cleared: job 14-011, invoice 21-018, and the ground covered by job 26-004.
Criterion 26+3
conflict_check.html gives the reason invoice 21-018 of 2021-03-04 to "D.S. CO." for $450.00 cannot be cleared…
Desired Outcome
…Exactly three records must be listed as unresolved rather than cleared…never reaches Model A or Model B
conflict_check.html
The golden cannot clear 14-011, 26-004 and 26-028. Invoice 21-018 is nowhere on it.

What it should have been

Criterion 24+5
conflict_check.html lists job 14-011 and the ground covered by job 26-004 as unresolved rather than cleared.
Criterion 26Delete it

How it breaks the standardInvoice 21-018 has no job number, billing address or phone in QLS_invoices_2011-2025_export.csv, and no ledger line, so it matches no named party and no tract, and no Leg A prompt names it. The golden never mentions it and fails both criteria, as would any run doing exactly what the Leg A prompts ask.

The rule it breaksGuidelines · 4. Valid vs. Invalid Model Failures A requirement the model never saw. Listed under “Not valid: your defect, or the platform's”.

The full write up of Example 22

Catch it in your own task

Before you complete Leg B, grade the golden against every objective criterion, and mark every Desired Outcome rule no turn asks for.

Where the correct guidance is written

Section titles as they print, so you can search for them in the document.

Guidelines · 5.7 Final Rubric Checklist

The golden solution passes the complete rubric set. Anything it fails is a broken criterion, not a broken golden.

Guidelines · 4. Valid vs. Invalid Model Failures

A requirement the model never saw. A rubric demanding a heading that appears only in the Desired Outcome.

Listed under “Not valid: your defect, or the platform's”.

Why it matters

The golden has to pass the complete rubric set. A criterion the golden fails is a broken criterion, not a broken golden, so a correct run gets marked Not Present. Each one counts as Major, and Majors on more than 10% of your criteria fail the task outright. The commonest cause is a rule that lives only in the Desired Outcome: the agent never sees the outcome, so a criterion resting on it fails a run that did exactly what it was told, and it catches your own golden too.

Every example taken apart, with the task, how it breaks the standard and what it should have been: 5.1 A criterion the golden itself fails, in the explanations →