The mistakes that keep costing tasks in this project, each shown in real tasks where it happened. The slides teach the standard; each entry here quotes where it is written, takes the examples apart, and ends with how to catch the mistake and what to do instead.
Each issue opens with its cost: 🔴 Fails the task, 🟠 Fails a client CORE standard or 🟡 Costs points. Examples are numbered across the course, the same on the slides and in the viewer.
How to use this
Read the sections for the step you are on. They follow the part of the task, not the build order, and the earlier the step, the more a fix costs:
| Step in the Guidelines | Sections | If you get it wrong |
|---|---|---|
| 1 Planning the Agent Task, 2 Setting Up the Run | 1; 2, for the input folder and the opening prompt | Run both legs again |
| 3 Task Runner - Leg A | 2, for each follow up you send Model A | Run both legs again |
| 5 Objective Rubrics | 4; 5 | Rewrite in place, or run both legs again if the fix changes a prompt or Model A fails under half the weight |
| 6 Milestones | 3 | Rewrite in place, before Leg B |
| 7 Golden Solution - Leg B | 2, for each turn you send Model B; 5, for the golden passing every objective criterion; 7 | A Leg B cleanup turn to Model B before you complete Leg B, or Leg B again for a hint that leaks |
| 8 Subjective Rubric Block | 6 | Rewrite in place |
| Before you submit | 7, the close out | Fix in place |
One habit prevents most of what follows: open the files, then write.
1. The Foundations Every Task Must Get Right
Two decisions made before prompt 1 that cannot be repaired afterwards. Each one is cheap to check and expensive to get wrong.
1.1 The deliverable sits below the complexity barSee it in the viewer
🟠 Fails a client CORE standard
The deliverable misses its type's complexity floor. Typically the Agent Objective says dashboard, no prompt asks for interaction, and the page ships static.
Where the correct guidance is written
Guidelines · Complexity Bar
“Interactive HTML: Interaction that actually works: controls that respond, state that changes, filters or navigation. A styled static page does not qualify. Dashboard: Several linked views over data the model extracted itself. One chart is not a dashboard.”
Both are P0 floors.
Guidelines · 1.3 Defining the Draft History
“The Draft History (Agent Objective, Desired Outcome, Multimodal Upload) is not embedded in the agent by default.”
So only a prompt can ask for a dashboard.
Example 1 (🔗Open in the viewer)
What is wrong. The Agent Objective asks for "one local HTML dashboard"; the golden ada_list.html has no script, button, filter or link. Static, it clears neither the dashboard nor the interactive HTML floor.
The task. Leg A turn 1 asks Model A for ada_list.html, a shop list giving brand, price, size and a picture per item. Leg A turns 2 and 3 add rules and layout.
Where it shows. The Agent Objective: "He needs one local HTML dashboard". The golden ada_list.html has no script, button, filter or link: a list of seven items, one comparison table and nine images.
How it breaks the standard. Nothing on ada_list.html responds, so the task ships a document, the lowest priority type on the Complexity Bar, though 14 Fresh orders could feed sorting or filters. Model A and Model B never see the Agent Objective, and no Leg A prompt asks for interaction, so no criterion grades it.
What it should have been. Leg A turn 1 asks for the interaction: "Put the buy list in a table I can sort by price, and let Ada hide what's not on Fresh."
Catching it in your own task
Ask what the page does when someone clicks it. If nothing, it is a document, whatever the Agent Objective calls it.
What to do instead
Before prompt 1, check the deliverable type against its floor, and have a prompt ask for the interaction the data supports.
1.2 The task relies on records the universe does not holdSee it in the viewer
🔴 Fails the task · Trajectory - Feasibility With Tools
A rule reads off records the assigned universe does not hold, such as a catalog or a policy page, while prompts, inputs and criteria read as if they were there.
Where the correct guidance is written
Guidelines · 1.2.1 Start With the Universe
“The universe combined with the multimodal input has to sustain the opening prompt and every follow-up you intend to add”
Guidelines · 4. Valid vs. Invalid Model Failures
“Data that is not in the server, asked for anyway, and the model penalised for not finding it.”
Listed under “Not valid: your defect, or the platform's”.
QC spec · Trajectory - Feasibility With Tools
“The primary request is impractical or impossible and can't be answered by the tools available or enabled for the task.”
Fails the task when the missing records carry the main request; costs points otherwise.
Example 2 (🔗Open in the viewer)
What is wrong. pantry_rules.md prices every line from Fresh's catalog and the delivery fee from Fresh's Delivery Policy. The universe's catalog holds none of the five board items, and its policies are empty.
The task. Leg A turn 1 asks Model A to check the five listings on Tom's pantry_board_draft.png and work out what each apartment owes, under the house rules in pantry_rules.md.
Where it shows. Rule 1 prices lines "at the pack option we add to the cart from Fresh's catalog"; rule 5 takes the fee from "Fresh's Delivery Policy and nothing else". The universe's catalog has none of the five items, and its policies are empty, so Leg B turn 3 sends Model B to find the pages "another way".
How it breaks the standard. Pricing every card and what each apartment owes rests on rules 1 and 5, so neither Model A nor Model B can do the main request. The golden keeps Tom's board prices, flagged unconfirmed, and criteria 16, 17 and 26 want Fresh's $0.55 a lb bananas, $5.45 spinach and $4.99 fee, which no tool returns.
What it should have been. Rules 1 and 5 cite only records the universe returns, checked before Leg A turn 1: a catalog search for the five items and a policy search for the fee.
Catching it in your own task
Your golden shows it: an empty search, a Leg B hint sending Model B another way, figures flagged as unconfirmed.
What to do instead
Before the run, query the universe once per rule and rewrite any rule whose record is missing. If it only surfaces in Leg B, redesign the task rather than hint again.
2. What you may not tell Model A or Model B
Leg A defines the task: the opening prompt and the follow ups you send Model A. Leg B is the golden run: Model B gets the same opening prompt, word for word, then the turns you send it as you steer it through the milestones. The no leak rule and the input folder are the same in both legs; what a turn may point at is not.
2.1 A Leg B hint that hands Model B the answerSee it in the viewer
🔴 Fails the task · Multi-Turn - Simulator Answer Leak
In Leg B, a turn you send Model B hands it what it had to work out: a value, a count, a position or the conclusion. Naming the exact file to open costs points instead.
Where the correct guidance is written
QC spec · Multi-Turn - Simulator Answer Leak
“The user may say THAT something is wrong; they may not say WHAT the right answer is, WHERE exactly to find it, or HOW MANY of anything there are”
Any turn, either leg.
QC spec · Multi-Turn - Simulator Answer Leak
“No graded value is leaked, but the hints narrow the search space beyond an area-level nudge (e.g. naming the exact file to open).”
Costs points. Written about Leg B hints.
Guidelines · 7.2 Steering the Run & Hinting
“The test is simple: if the model can copy your hint into the artifact or use it directly as the reasoning needed to reach the milestone, you have given away the answer.”
Leg B only.
Guidelines · 1.2.3 Create the Initial Prompt and the Follow-ups
“Never flag the miss. If the agent omitted a total, misread a date, or ignored a source, do not tell it what went wrong or where to look. Pointing directly at the failure compromises the 50% failure threshold every task has to meet.”
Leg A only. A punctual correction that comes up naturally is still fine, as long as it does not compromise the 50% failure threshold.
QC spec · Multi-Turn - Turn Structure & Dependency
“Each follow-up must do real work: deliver a withheld asset, extend a requirement, correct an error, tighten a constraint, or answer a clarification the agent asked for.”
About the Leg A follow ups.
Example 3 (🔗Open in the viewer)
What is wrong. Leg B turn 9 names the file Model B must open, 26-014_Field_notes_2026-03-19.pdf, and opening it is what criterion 14 grades. Leg B hints may name the mail, not the file.
The task. PT 128 is described in 26-014_Field_notes_2026-03-19.pdf, a mail attachment. After Leg B turn 8 points at the day 3 notes, Model B says the figures are in an unreachable book.
Where it shows. Leg B turn 9, to Model B: “Open the day-3 field notes PDF in the mail, the file named 26-014_Field_notes_2026-03-19.pdf”.
How it breaks the standard. Criteria 14 and 2, both weight 5, grade opening that PDF and the PT 128 description taken from it. Model B's own search had missed the attachment, and the filename did that search for it, so the golden meets both on instruction. No value from the PDF is stated, so it costs points.
What it should have been. Leg B turn 9 points at the source, not the file: "Not the physical field book. The day 3 field notes came by email."
Example 4 (🔗Open in the viewer)
What is wrong. Leg B turn 7 tells Model B "dont price our order off rosas phone", the conclusion criterion 30 tests. Model B had to reach it from rule 5 and chat_5D.png.
The task. Rule 5 of pantry_rules.md: the building account is not Prime. chat_5D.png, delivered at Leg B turn 5, shows Rosa's Prime app pricing avocados at $1.89, and Model B used it.
Where it shows. Leg B turn 7, to Model B: "dont price our order off rosas phone - thats her own account, not the buildings".
How it breaks the standard. Criterion 30, a -5 negative, catches that price. Model B kept it after the Leg B turn 6 hint, and Leg B turn 7 handed it the conclusion criterion 30 tests, so the golden avoids the trap on instruction, not reasoning. No price is stated, so it is over steering and costs points. Model A got no such turn.
What it should have been. Leg B turn 7 drops the answer and keeps its pointer: "the house rules say which prices count for us". Model B still has to find rule 5.
Catching it in your own task
For each Leg B turn, ask what is left for Model B to work out. A turn that states what a criterion grades has answered it.
What to do instead
Hint as the user would: say what looks wrong and point at the area, never at the file, the value or the conclusion. If Model B still misses, hint again at the same level.
2.2 A Leg A prompt that answers its own attachmentSee it in the viewer
🔴 Fails the task · Multi-Turn - Simulator Answer Leak
A Leg A prompt that hands Model A a file also states what the file exists to make it read, such as a count or which document is current. The same words in Leg B leak it to Model B too.
Where the correct guidance is written
QC spec · Multi-Turn - Simulator Answer Leak
“One or more user turns supply a value, count, position, or conclusion (see notes for exceptions) the agent was supposed to derive from the multimodal inputs”
Any user turn, in either leg. A count is a value.
Guidelines · 7.1 What Changes from Leg A
“The same intent expressed the same way, and the same persona from first turn to last.”
Unchanged in Leg B, so the turn that hands Model B the file repeats the value.
Example 5 (🔗Open in the viewer)
What is wrong. Leg A turn 2 releases IMG_4392.jpeg and tells Model A the count it exists to supply: "five big ones and one small one". Leg B turn 4 repeats it to Model B.
The task. Leg A turn 1 holds back IMG_4392.jpeg, the plates photo. Leg A turn 2 releases it, so Model A can plan flat storage sized from it.
Where it shows. Leg A turn 2: "There are five big ones and one small one, and they need to be stored flat somewhere in the unit." Leg B turn 4 sent Model B the same words.
How it breaks the standard. IMG_4392.jpeg shows one large plate by the rulers, a stack behind it and a small one; counting the stack is the read the photo exists to force. Given the count, Model A and Model B only have to measure one plate of each size. A count is a value, so this fails the task.
What it should have been. Leg A turn 2 and Leg B turn 4 without the count: "They need to be stored flat somewhere in the unit, not stood on edge." Both legs run again.
Catching it in your own task
Read each prompt beside the file it hands over, in both legs: any value the model should read from that file is a leak.
What to do instead
State the requirement and leave the value to the file, then run both legs again.
2.3 A deferred asset that announces its own turnSee it in the viewer
🔴 Fails the task · Input Artifacts - Deferred Asset Handling
Something in the input folder, open to both models from turn 1, previews a later turn: a file named for its turn, or a document carrying a later instruction. A file named only for its contents costs points.
Where the correct guidance is written
QC spec · Input Artifacts - Deferred Asset Handling
“The file still ships in input_files/ at upload time, but neither its filename nor any manifest may preview the turn that delivers it.”
Guidelines · 1.2.2 Select the Multimodal Inputs
“All assets must be included in the input .zip folder from the start and use neutral filenames.”
Guidelines · 1.2.3 Create the Initial Prompt and the Follow-ups
“The follow-up must name the asset being introduced so the model knows exactly what new evidence has become available.”
Written for the Leg A follow ups.
Example 6 (🔗Open in the viewer)
What is wrong. The deferred files IMG_0814_calendar.jpg and IMG_0819_travel_policy.jpg sit in the input folder from the start, and their names say what Leg A turns 2 and 3 will hand over.
The task. Leg A turn 2 hands Model A Jaylen's marked school calendar, and Leg A turn 3 the Q3 travel directives.
Where it shows. They are named IMG_0814_calendar.jpg and IMG_0819_travel_policy.jpg.
How it breaks the standard. From their first turn, Model A and Model B can see in the input folder that a school calendar and a travel policy are coming, before the turn that hands each over in either leg. Neither name states a requirement or a graded value, so this costs points.
What it should have been. IMG_0814.jpg and IMG_0819.jpg. Leg A turns 2 and 3 already say what each is, "Jaylen's school calendar" and the "Q3 travel directives", and their Leg B turns repeat it.
Catching it in your own task
Read the input folder as both models will at turn 1, filenames included: price_sheet_for_turn3.pdf fails, price_sheet.pdf costs points, sheet_02.pdf is fine.
What to do instead
Name deferred files as a phone or scanner would, and let the Leg A follow up that delivers each one say what it is; the Leg B turn repeats it.
3. Milestones
Written from the Leg A conversation with Model A, used in Leg B to steer Model B. Three standards cover what sits around the intent line: Modifier and Assets Delivered, the stopping rules, and one milestone per requirement on every Leg A turn.
3.1 Modifier and Assets Delivered that do not match the Leg A turnSee it in the viewer
🔴 Fails the task · Milestones - Milestone Annotations · Milestones - Continuation Criteria & Assets · most common right now
Modifier or Assets Delivered misdescribes the milestone's Leg A turn: deferred_asset or revision on Leg A turn 1, none as a modifier, a bare extension for a filename, or a deferred file no row lists. Only the last fails the task.
Where the correct guidance is written
Guidelines · 6.1 What a Milestone is Made of?
“Modifier: What this turn does to work that already exists. One of the four in section 6.3. Assets delivered: Files handed over at this turn, or empty.”
This turn is the milestone's Leg A turn.
Guidelines · 6.3 Modifiers
“opening: The first turn and any later turn that adds a new requirement without touching work already delivered. revision: Something already delivered changes.”
Leg A turn 1 is always opening.
QC spec · Milestones - Continuation Criteria & Assets
“Also fails where a deferred asset was handed over during the conversation but is absent from assets_delivered, or is listed but missing from input_files/.”
Example 7 (🔗Open in the viewer)
What is wrong. Milestone 9, the deferred_asset row for Leg A turn 2, reads Assets Delivered: .html, the deliverable's extension. The file that turn hands over, bourgery_legend_pages.pdf, is on no milestone.
The task. Leg A turn 1 asks for deltoid_plate_study.html from eighteen images. Leg A turn 2 hands over bourgery_legend_pages.pdf, the one deferred file.
Where it shows. Milestones 1 to 12 all read Assets Delivered: .html, including milestone 9, the deferred_asset row for Leg A turn 2.
How it breaks the standard. The .html on those rows is the extension of deltoid_plate_study.html, the deliverable, not a file the user hands over. The file Leg A turn 2 hands over, bourgery_legend_pages.pdf, sits on no row, so a user simulator replaying them has no record of when it arrives. A deferred file missing from Assets Delivered fails the task.
What it should have been. Milestone 9 reads Assets Delivered: bourgery_legend_pages.pdf, which ships in input_files/ from upload. Milestones 1 to 8 name the eighteen Leg A turn 1 files; milestones 10 to 12 are empty.
Example 8 (🔗Open in the viewer)
What is wrong. Milestones 7 and 8, on Leg A turn 1, read deferred_asset and revision, with Assets Delivered: .mp4. Leg A turn 1 is always opening, and handbook.mp4 comes with it.
The task. Leg A turn 1 attaches index.html, Vantol.pdf and handbook.mp4 together, and asks Model A to reconcile “a festival message and a runtime record” and complete the runtime section.
Where it shows. Milestones 7 and 8, both on Leg A turn 1, read deferred_asset and revision, each with Assets Delivered: .mp4.
How it breaks the standard. Leg A turn 1 asks for milestone 7's work, and handbook.mp4 is in the folder before Model A starts, so nothing is deferred. Milestone 8's corrections fall within the response to that same turn, and a revision changes work delivered at an earlier Leg A turn. A bare .mp4 names no file.
What it should have been. Milestones 7 and 8 read Modifier: opening, like milestones 1 to 6. Assets Delivered names the Leg A turn 1 files in full: index.html, Vantol.pdf and handbook.mp4.
Example 9 (🔗Open in the viewer)
What is wrong. Milestones 1 and 4 read Modifier: none, which is not one of the four modifiers. Both should read opening.
The task. Leg A has four turns. Leg A turn 1 asks for the walkthrough; Leg A turn 4 asks for every check, in order, in the reply, and touches no file.
Where it shows. Milestones 1 and 4 read Modifier: none.
How it breaks the standard. The value none is not one of the four modifiers. Leg A turn 1 is always opening, and Leg A turn 4 asks for a list in the reply, changing neither walkthrough, so it is opening too. A wrong modifier costs points.
What it should have been. Milestones 1 and 4 read Modifier: opening.
Catching it in your own task
Read Modifier and Assets Delivered beside the Leg A prompts: Leg A turn 1 is opening, and each deferred file sits on the Leg A turn that hands it over.
What to do instead
Set each modifier from what its Leg A turn does to delivered work, and list each file handed over at that turn by full filename, never the deliverable.
3.2 Stopping rules that do not gate the turnSee it in the viewer
🟡 Costs points · Milestones - Continuation Criteria & Assets · Milestones - Milestone Annotations
Continuation criteria, read in Leg B to decide when Model B moves on, either miss one of their Leg A turn's asks or demand what that turn cannot produce, such as a later turn's file.
Where the correct guidance is written
Guidelines · 6.2 Writing the Milestones
“Write one clause per requirement of that turn, separated by pipes, each of them true or false on inspection.”
That turn is the milestone's Leg A turn.
QC spec · Milestones - Continuation Criteria & Assets
“Continuation criteria are present and checkable but under-specified — they would let the simulator advance while a requirement of that turn is still unmet.”
In Leg B, the simulator is you.
Example 10 (🔗Open in the viewer)
What is wrong. Milestone 8, on Leg A turn 2, requires the travel rules in IMG_0819_travel_policy.jpg, which only Leg A turn 3 hands over. Milestone 9 repeats its intent there, word for word.
The task. Leg A turn 2 hands over IMG_0814_calendar.jpg and sets the travel itself: “about three days a week”. The travel directives, IMG_0819_travel_policy.jpg, arrive at Leg A turn 3.
Where it shows. Milestone 8, on Leg A turn 2, reads “The user has sent the travel rules his employer issued”, and its first clause requires “the travel days in the model reflect only the document the user attached”.
How it breaks the standard. At Leg A turn 2 the only file handed over is the calendar, and the user sets the travel at about three days a week himself. No correct response to Leg A turn 2 can meet the clause, so in Leg B it would keep you hinting Model B toward a directive not yet handed over.
What it should have been. Delete milestone 8. Milestone 9, on Leg A turn 3, the turn that hands over IMG_0819_travel_policy.jpg, already checks that the travel days come from it.
Example 11 (🔗Open in the viewer)
What is wrong. Leg A turn 1 asks which pieces still carry a correction, and none of milestone 1's six clauses checks it, so Model B can pass milestone 1 without it.
The task. Leg A turn 1 asks, among other things, which pieces still carry a correction. None of its three exports does, and Slack holds corrections for pieces outside the drop.
Where it shows. Milestone 1's six clauses check the file, the cards, type and subject, the counts and the read or confirmed marks. The first restates Leg A turn 1's ask and stops at “every count opening the files behind it”, with no word on corrections.
How it breaks the standard. In Leg B you move Model B past milestone 1 once its six clauses hold, and a response that never checks corrections meets every one. Here the right answer is none of the three, so a response that skips the check, or pins another piece's Slack correction on one, moves on unchecked. This costs points.
What it should have been. A seventh clause beside the six: each card states whether its piece still carries a correction. Until every card does, you keep hinting Model B in Leg B.
Catching it in your own task
Could a response skip one of the Leg A turn's asks and still pass? Could a correct one fail for lack of a later file?
What to do instead
Tick every ask in each Leg A prompt against a clause, and move any clause that needs a later file or ask to the turn that brings it.
3.3 A Leg A turn no milestone covers, or the same milestone written twiceSee it in the viewer
🟡 Costs points · Milestones - Continuation Criteria & Assets
A Leg A turn no milestone covers, usually from a set stopping a turn short, or two milestones on one turn with identical intent and clauses.
Where the correct guidance is written
Guidelines · 6.2 Writing the Milestones
“One milestone per requirement, grouped by turn: A follow up carrying three requirements produces three milestones, not one.”
So every Leg A turn has at least one.
Guidelines · 6. Milestones
“you are the simulator, and the milestone set is the only script you have.”
In Leg B, steering Model B.
Guidelines · 6.1 What a Milestone is Made of?
“Turn: Which turn of the Model A conversation the requirement came from.”
Count Leg A turns.
Example 12 (🔗Open in the viewer)
What is wrong. No milestone covers Leg A turn 5, “Use her OPB address, from my Vantol Films account.” The set stops at Leg A turn 4, and every Leg A turn needs one.
The task. Leg A turn 4 asks for a note to June. Model A asks which address and which sending account to use, and Leg A turn 5, the last, answers.
Where it shows. The milestones stop at Leg A turn 4, so “Use her OPB address, from my Vantol Films account” has none.
How it breaks the standard. Criterion 7 grades both answers: the note goes to June's OPB address, from Idris's Vantol Films account. Model B did this unasked, so the gap never showed in Leg B. A user simulator replaying the set against a future run that asks, as Model A did, has no script for the reply. This costs points.
What it should have been. A milestone on Leg A turn 5: the note to June goes to her OPB address, sent from the Vantol Films account, one continuation clause for each.
Example 13 (🔗Open in the viewer)
What is wrong. Milestones 13 and 14, on Leg A turn 2, are the same milestone twice, intent and clauses word for word. Only one artwork file, loudness_card.png, is missing.
The task. Leg A turn 2 hands over asset_pack/, the artwork and a shot of Marisol's queue. One post's artwork, loudness_card.png, never came through.
Where it shows. Milestones 13 and 14, both on Leg A turn 2, are identical, word for word.
How it breaks the standard. Leg A turn 2 has one missing file to report, and its other asks, the artwork, the queue and the take downs, already have milestones 11, 12, 15 and 16. Milestone 14 repeats milestone 13's check and covers nothing new; Model B meets both with one answer, so nothing flags it. A duplicate costs points.
What it should have been. Delete milestone 14 and keep milestone 13 as the one milestone for the missing artwork file.
Catching it in your own task
List each milestone's Turn beside the Leg A prompts: every Leg A turn needs one per ask. Identical neighbours are duplicates.
What to do instead
Write a milestone for each uncovered turn from its prompt, and delete a duplicate or give it its missing requirement.
4. A rubric that separates two runs
A rubric exists to tell a strong run from a weak one. Five standards keep it doing that, and a set can break any of them while still looking complete.
4.1 A rubric that can only reward, never penaliseSee it in the viewer
🟡 Costs points · Rubric Criteria - Overall Rubric Quality · most common right now
Too few negatives for the scenario's traps, so taking the bait costs nothing extra, or a negative worded as the virtue, which marks down correct runs.
Where the correct guidance is written
Guidelines · 5.3 Positive and Negative Distribution
“Negatives are the exception: They should usually make up around 25% of the rubric block and should generally stay below 30%. Use a negative criterion only when it captures a specific, plausible failure mode that the scenario or trajectory meaningfully exposes.”
Guidelines · 5.2 Rubric Weights
“The negative slots are for attractive wrong paths the model has to resist.”
Start with the traps Leg A walked into.
Guidelines · 5.5 Rating Model A and Justifying the Failures
“For negative weight: The trajectory contains the undesirable behavior or boundary violation described by the criterion.”
Example 14 (🔗Open in the viewer)
What is wrong. The 34 criteria carry no negative, though Leg A turn 1 sets a trap: it says the clip's quiz picks are right, and basics.mp4 shows all three wrong.
The task. Leg A turn 1 asks for the page shown in 2panels_demo.webm, built as cardio.html, and calls the clip's picks right. mod3_notes.txt says they “were fast ones”: go with the slides.
Where it shows. 24 objective and 10 subjective criteria, every weight positive.
How it breaks the standard. The trap is real: Model B, Leg B turn 1, keyed the third item to the clip's ~80%. Yet a run that trusts all three picks loses only the two +3s of objective criteria 12 and 19, exactly what Model A lost without reading the clip at all. Taking the bait costs nothing extra.
What it should have been. Add an objective negative at -3, since mod3_notes.txt points the right way: “cardio.html keys the self check to the picks selected in 2panels_demo.webm.”
Example 15 (🔗Open in the viewer)
What is wrong. Subjective criterion 10 is a -3 negative worded as a virtue: the finished plate “remains legible”. The golden's plate is legible, so the golden is Present and loses three points.
The task. Leg A turns 1 to 3 ask for plate12_simplification_lesson.mp4, a narrated lesson on Plate 12. The golden video runs four and a half minutes at 1080p, its finished plate legible.
Where it shows. Subjective criterion 10, a negative at weight -3: “the finished teaching plate remains legible at normal 1080p or full-window playback”.
How it breaks the standard. A negative takes its weight when Present, so subjective criterion 10 charges every legible video three points and spares an unreadable one. The golden, the reference the subjective block is graded against, is the video it penalises. Subjective criterion 3 has the right form: it names the flaw, a black or frozen frame.
What it should have been. Subjective criterion 10 at -3, naming the flaw: the finished plate in plate12_simplification_lesson.mp4 renders its captions or labels below legible size at 1080p.
Catching it in your own task
Negatives should be about a quarter of the set. Present means failure, so a negative describing a good run is inverted.
What to do instead
After Leg A, write a negative for each worthwhile wrong turn Model A took, stating the flaw, never the virtue.
4.2 A requirement nothing grades, and a criterion nothing asked forSee it in the viewer
🔴 Fails the task · Rubric Criteria - Overall Rubric Quality
A prompt requirement no criterion checks, or a criterion no turn asked for, usually written from the golden. Both are Major; a non critical gap, or a criterion that only narrows a real ask, costs points.
Where the correct guidance is written
Guidelines · 5.7 Final Rubric Checklist
“Walk the prompt once per turn and tick off every ask. Anything you can tick needs a criterion.”
QC spec appendix · Rubric Quality Definitions · Incorrect Criteria
“Criterion is not an explicit requirement in the prompt and implementing it makes the response worse.”
QC spec appendix · Rubric Quality Definitions · Missing Criteria - Critical Requirements
“Count each missing rubric that ought to check for an explicit requirement in the prompt or a critical implicit expectation of the prompt as one issue”
A Major issue.
Example 16 (🔗Open in the viewer)
What is wrong. Objective criterion 12 requires runbook_for_cyrus.md to cite the team's cleaning threads. No Leg A turn asks for a citation: Leg A turn 3 asks for a check against them.
The task. Leg A turn 3 asks Model A to check runbook_for_cyrus.md against “what we worked out” in the team's threads, and to put the request and due date on top.
Where it shows. Objective criterion 12: runbook_for_cyrus.md names the team's own prior cleaning threads as a source of its rules.
How it breaks the standard. A correct run checks every rule against the threads and fixes the ones that diverge, as Leg A turn 3 asks, and still fails objective criterion 12 for lacking a citation nobody requested. The golden passes only because runbook_for_cyrus.md happens to end with a Sources line, so checking against the golden never exposes it.
What it should have been. Delete objective criterion 12 and milestone 11's clause “the runbook names the threads as a source”. Objective criteria 3 to 5 already grade the rules the threads settled.
Example 17 (🔗Open in the viewer)
What is wrong. Leg A turn 4 limits Keep to what survives the sheets, Yusuf's email and the calendar. No criterion grades it: objective criteria 21 and 25 exclude only IMG_2201.jpg and IMG_2105.jpg.
The task. notes.md defines Keep on canon_study.html: the live head height, the plate IMG_2104.jpg, and the conference and workshop facts from Yusuf's email and the calendar. IMG_2107.jpg is not among them.
Where it shows. Objective criteria 1 to 10 require ten facts in Keep, and objective criteria 21 and 25 penalise IMG_2201.jpg and IMG_2105.jpg there. No criterion says Keep holds nothing else.
How it breaks the standard. Model A walked into the gap: its Leg A turn 4 page keeps IMG_2107.jpg, the deep back muscles plate, under Keep, though notes.md says “Keep is only IMG_2104.jpg.” No negative names IMG_2107.jpg, so no criterion fails Model A for it, and the point of Leg A turn 4 goes ungraded.
What it should have been. Add a closure, as objective criterion 13 does for Do not mix: Keep on the final canon_study.html holds only the items criteria 1 to 10 check for, and nothing else.
Catching it in your own task
Pair each prompt requirement with its criterion, and each criterion with the sentence that asks for it.
What to do instead
Write criteria from the prompt, never from the golden. Where an instruction closes a set, write the closure: X consists only of A, B and C.
4.3 Weights that do not separate difficulty from lookupSee it in the viewer
🟡 Costs points · Rubric Criteria - Overall Rubric Quality
Weights that ignore difficulty: a cross source derivation weighted like a direct read, or one operation at 5 in one row and 3 in the next.
Where the correct guidance is written
Guidelines · 5.2 Rubric Weights
“Weight is difficulty, never importance. How hard the criterion is to satisfy, or for a negative, how tempting the failure is.”
Priced by four axes (source coordination, reasoning depth, modality, discovery effort): three or more is +5, two is +3, zero or one is +1.
QC spec appendix · Rubric Quality Definitions · Incorrect Weights - Major
“Criteria that are objectively incorrectly weighted by two levels, e.g., 1 is selected when 5 is appropriate or vice versa.”
A Moderate issue, despite its name.
Example 18 (🔗Open in the viewer)
What is wrong. Objective criterion 12, the BendFilm fee worked out from a ticket order, weighs 1, like objective criterion 17, a page count read off the inventory. A calculation from one source takes +3.
The task. Leg A turn 1 sets the BendFilm fee at four times the order processing fee of the June Timbers vs Sounders game: $5.00 on the Ticketmaster order, so $20.
Where it shows. Objective criterion 12, the BendFilm fee of $20, and objective criterion 17, SetInLead_Treatment_Final.pdf classified as Ready, both weigh 1.
How it breaks the standard. Objective criterion 12 takes a hidden field and a calculation: find that order, take its order processing fee rather than its $15.00 service fee, and multiply by four. Objective criterion 17 checks one listed page count against a limit. At 1 each, a Model A run that misses the fee loses no more than one that misreads a page count.
What it should have been. Objective criterion 12 at 3, for its two axes, a hidden field lookup and a calculation. Objective criterion 17 stays at 1.
Catching it in your own task
Count the axes each criterion exercises and read off its weight.
What to do instead
Weight by axis count, never importance. A lookup at 5, or a chained derivation at 1, is a two level miss.
4.4 More trajectory criteria than the Guidelines allowSee it in the viewer
🟡 Costs points · Rubric Criteria - Overall Rubric Quality
More than five criteria target Trajectory. Each is usually sound on its own, so the count creeps up.
Where the correct guidance is written
Guidelines · 5.1 Rubric Fundamentals
“[NEW]: Trajectory Rubrics: Maximum of 5, minimum 0. They are NOT mandatory.”
Since v3, 10 Sep 2026.
Guidelines · 5.4 Category & Evaluation Target
“The agent's reasoning path, Trajectory: An identification, a dependency order, a piece of restraint. The intent it pursued, never a named tool call.”
Example 19 (🔗Open in the viewer)
What is wrong. Objective criteria 33 to 42, ten in all, target Trajectory, where the Guidelines allow five. Several grade the route to a value a page criterion already checks.
The task. energy_audit.html audits the August electricity bill and splits it between Jayden and Percy. Its values come from photographs, an image only statement and connected records.
Where it shows. Objective criteria 33 to 42, ten in all, target Trajectory.
How it breaks the standard. Objective criterion 34 grades where the split rule's terms came from, yet objective criterion 27 already checks that rule applied on the page. Objective criteria 35, 40 and 41 do the same for objective criteria 28, 26 and 18, whose 235 to 330 kWh band already fails a whole day method. Objective criterion 33 only asks for the queries those values need.
What it should have been. Keep the five that grade what only the run shows, objective criteria 36, 37, 38, 39 and 42, such as 37, the handwritten log read from the photographs. Drop objective criteria 33, 34, 35, 40 and 41.
Catching it in your own task
Count the objective rows that target Trajectory.
What to do instead
Keep the five only the run can show, such as a load bearing media read, and move the rest's values into artifact criteria.
Note. The previous version of this course asked for more trajectory weight; Guidelines v3 capped it.
4.5 A run that does not fail enough of the rubricSee it in the viewer
🔴 Fails the task · Hard Client Requirements for Every Task
The criteria Model A fails carry under half the rubric's weight, both blocks counted. Everything looks sound; only the arithmetic shows it.
Where the correct guidance is written
Guidelines · Hard Client Requirements for Every Task
“Model A has to fail: Enough difficulty that the Model A trajectory clearly fails at least 50% of the final rubric score”
Guidelines · 8.5 Rating Against both Models and Justifications
“Rate each subjective rubric as Present or Not Present for Model A.”
So the 50% counts both blocks.
Example 20 (🔗Open in the viewer)
What is wrong. Model A fails 13 of 49 criteria, 43 of the 123 total weight with both blocks counted: 35.0%. The floor is half the weight, 62.
The task. Leg A turns 1 to 4 build oven_decision.html, a one page case for a deck oven: its cost, the landlord's allowance, the building's power and the monthly commitments.
Where it shows. Model A failed 13 of 49 criteria, carrying 43 of the 123 total weight.
How it breaks the standard. That is 35.0%; the floor needs 62, so the task fails. Nothing else flags it: the rubric is sound and the golden passes it. Weights cannot fix it: three of the four weight 5 criteria are already failed, and objective criteria 1, 26 and 34, image reads Model A passed, belong at 3, not 1.
What it should have been. A harder scenario, with sources that disagree, and both legs run again until Model A fails at least half the weight. Never heavier weights or extra rows.
Catching it in your own task
Divide the weight of the criteria Model A fails by the total, both blocks counted, signs ignored.
What to do instead
Under 50%, make the scenario harder and run both legs again. Never inflate the weights or pad the set.
5. Criteria a correct run can pass
A criterion has to pass a correct run, fail a wrong one, carry the right Category and Evaluation Target, and count once. If your own golden fails one, check the criterion first: it is usually the one that is broken.
5.1 A criterion the golden itself failsSee it in the viewer
🔴 Fails the task · Rubric Criteria - Overall Rubric Quality · most common right now
Your golden fails a criterion that is wrong about the task: a rule only the Desired Outcome states, a phrase at a different length than the golden renders, or a file the inputs archive lacks.
Where the correct guidance is written
Guidelines · 5.7 Final Rubric Checklist
“The golden solution passes the complete rubric set. Anything it fails is a broken criterion, not a broken golden.”
Guidelines · 4. Valid vs. Invalid Model Failures
“A requirement the model never saw. A rubric demanding a heading that appears only in the Desired Outcome.”
Listed under “Not valid: your defect, or the platform's”.
Example 21 (🔗Open in the viewer)
What is wrong. Criterion 15 wants family heirloom in the logo in place of FAMILY. Leg A turn 3 asks for Tamara's exact phrase, a whole sentence, and the golden logo shows it.
The task. Leg A turn 3 asks for “the exact phrase Tamara used when she joked about Terrence’s truck”. The records give “That truck is a family heirloom at this point.”
Where it shows. Criterion 15: In family_photo_tree.html, the logo shows family heirloom beneath the tree in place of the original word FAMILY.
How it breaks the standard. Criterion 15 copies Desired Outcome item 5, which Model A and Model B never see, instead of Leg A turn 3. Model B, Leg B turn 3, puts Tamara's whole sentence in the logo, as asked, so the golden fails criterion 15. The golden is right; the criterion is what is broken.
What it should have been. Criterion 15 follows Leg A turn 3: the logo shows That truck is a family heirloom at this point. beneath the tree in place of FAMILY, which the golden passes.
Example 22 (🔗Open in the viewer)
What is wrong. Criteria 24 and 26 grade invoice 21-018 as a record that cannot be cleared, but only the Desired Outcome names it and the golden never mentions it.
The task. Leg A turn 1 asks to check every named party and the tract, and which of the jobs they touch cannot be cleared “because the record is too thin”.
Where it shows. Criterion 24 requires that conflict_check.html “lists exactly three records as unresolved rather than cleared: job 14-011, invoice 21-018, and the ground covered by job 26-004.” Criterion 26 grades the reason invoice 21-018 cannot be cleared. Only the Desired Outcome asks for the count or the invoice: “Exactly three records must be listed as unresolved rather than cleared”.
How it breaks the standard. Invoice 21-018 has no job number, billing address or phone in QLS_invoices_2011-2025_export.csv, and no ledger line, so it matches no named party and no tract, and no Leg A prompt names it. The golden never mentions it and fails both criteria, as would any run doing exactly what the Leg A prompts ask.
What it should have been. Criterion 24 names only job 14-011 and the ground covered by job 26-004, with no count, so the golden's 26-028 does not fail it. Criterion 26 goes.
Catching it in your own task
Before you complete Leg B, grade the golden against every objective criterion, and mark every Desired Outcome rule no turn asks for.
What to do instead
Rewrite an unsupported criterion, or fix the golden with a Leg B cleanup turn. Move an outcome only rule into a prompt and run both legs again, or drop it.
5.2 Criteria a correct run cannot pass, or a wrong one canSee it in the viewer
🟡 Costs points · Rubric Criteria - Overall Rubric Quality
A criterion stricter than its source fails correct runs: it pins a figure the source leaves open, picks one answer of several, insists on one of two sources, or scopes to a step that lacks the values. A looser one passes wrong runs.
Where the correct guidance is written
Guidelines · 5.1 Rubric Fundamentals
“Since these are objective rubrics, only requirements that can be evaluated objectively from the scenario should be included.”
QC spec appendix · Rubric Quality Definitions · Overfitting and Underfitting
“Overfitting: Criteria that are overly specific, inflexible or too rigid - they incorrectly reject a subset of valid implementations. Underfitting: Criteria that are overly broad, permissive or loose - they accept valid implementations, but also incorrectly accept invalid implementations too.”
Both are Moderate.
Example 23 (🔗Open in the viewer)
What is wrong. Criterion 23 requires PCR-2026-0688 and its 10 July 2026 acceptance in the change board step. The golden's change board scene shows neither; both sit in an earlier scene.
The task. Leg A turn 3 asks for one more decision step in source_check.mp4, to show that “finding a discrepancy, accepting a change request, and releasing a changed figure are different decisions”.
Where it shows. Criterion 23 is scoped to “The change-board step” and requires “PCR-2026-0688”, “accepted on 10 July 2026”. The golden's change board scene, 127.8 to 138.3 seconds, names 32-21-07 as accepted in July, with no PCR number and no date.
How it breaks the standard. The PCR number and the 10 July date appear in source_check.mp4, but only in the earlier record scene, 26 to 39 seconds, and Leg A turn 3 never asks for them in the new step. So the golden is Not Present on criterion 23 at weight 5. This is Overfitting, which costs points.
What it should have been. Criterion 23 scoped to source_check.mp4: the July acceptance of the 32-21-07 Figure 2 change request did not put the change in the manual; only the 30 September release does.
Example 24 (🔗Open in the viewer)
What is wrong. Criteria 13 and 28 require Aunt Rosalind as the possible guest. The voice note asks about family members, plural and unnamed, and the golden names Tamara Brown-Ellis instead, failing both.
The task. Leg A turn 3 hands over turn-3-voice-note.mp3, which asks about “family members who might stay with us” and names no one. The records support more than one.
Where it shows. Criterion 13 requires combined-home.html to name Aunt Rosalind as a possible guest, and criterion 28 requires the trajectory to find that she cannot walk far.
How it breaks the standard. The golden's combined-home.html answers the voice note with Tamara Brown-Ellis, from Contacts, Calendar and Email, and never mentions Rosalind, so the golden is Not Present on both criteria. Pinning one person where the voice note asks for any family member the records support is Overfitting, which costs points.
What it should have been. Criterion 13, widened: “combined-home.html identifies a family member from the records as a possible overnight guest and states the recorded need that shapes where they sleep.” Criterion 28 goes.
Example 25 (🔗Open in the viewer)
What is wrong. Criterion 26 accepts the register prices only from the 5 May calendar event. The same list is attached to the 4 May mail, and the golden took it from there.
The task. The nine price changes sit in two records, the 5 May calendar event and price list 5 may.pdf attached to the 4 May mail. Both carry the held prices too.
Where it shows. Criterion 26 wants the nine old and new register prices grounded in “the 5 May 2026 07:00 PT calendar event”, and criterion 29 wants the held prices “from the 4 May mail”. Model B, Leg B turn 1, grounds the list in the mail: “The list you sent Marguerite the night before”.
How it breaks the standard. Leg A turn 1 names neither record, and either one gives every price. Criterion 29 rewards reading the mail, and a run that takes the whole list from there, as the golden did, fails criterion 26 at weight 5. Pinning one of two valid sources is Overfitting, which costs points.
What it should have been. Criterion 26 accepts either record: the nine prices from the calendar event “New prices in the register” or from the price list attached to the 4 May mail.
Catching it in your own task
For each criterion, name its source line and where your golden shows it, then ask what wrong run still passes.
What to do instead
Grade the shape of an answer the source leaves open, accept either source, scope to where the values appear, and add bounds: exactly, only, between.
5.3 Category and Evaluation Target filled with the wrong kind of valueSee it in the viewer
🟡 Costs points · Rubric Criteria - Overall Rubric Quality
Category or Evaluation Target does not fit the criterion: Agent Behavior or Tool Use on an artifact, a category in the target field, or a subjective row tagged Instruction Following.
Where the correct guidance is written
Guidelines · 5.4 Category & Evaluation Target
“Agent Behavior, Target: Trajectory, always. If you wrote Agent Behavior against an artifact, a state change or the final message, you picked the wrong category: you are grading the deliverable, which is Task Completion, Instruction Following or Factuality.”
Guidelines · 8.4 Categories and Evaluation Targets for Subjective Rubrics
“Task Completion is the only valid category for subjective rubrics.”
Example 26 (🔗Open in the viewer)
What is wrong. Criteria 17 and 30 grade what the running note says, yet their Category is Agent Behavior, which always targets Trajectory. A check on a file grades the deliverable: Task Completion.
The task. Leg A turn 1 asks for “a running note of where each album stands and what the rules are”. Criteria 17 and 30 grade what that note states.
Where it shows. Both are Category Agent Behavior, with Evaluation Target Final answer/artifact.
How it breaks the standard. In the golden, both are read off hh_migration_notes.md, its running note, so they grade the deliverable, not the trajectory. Wording and target are right; only the category is wrong. The criteria still pass and fail the right runs, so grading never exposes it, but the client receives wrong rubric data, which costs points.
What it should have been. Category Task Completion on both, keeping Evaluation Target Final answer/artifact and the wording. A retag is made in place; neither leg runs again.
Example 27 (🔗Open in the viewer)
What is wrong. Subjective criterion 2, on key markers coloured like their muscle's heading, is tagged Instruction Following. No Leg A prompt asks for colours, and the subjective block takes Task Completion only.
The task. Subjective criterion 2 grades whether the key markers in deltoid_plate_study.html use one colour per muscle, matching that muscle's section heading.
Where it shows. It is tagged Instruction Following.
How it breaks the standard. None of the four Leg A prompts mentions colour, so subjective criterion 2 grades no instruction; it grades how deltoid_plate_study.html looks, which is Task Completion. Had a Leg A prompt asked for the colours, the check would belong in the objective block instead. The wording is right; only the tag costs points.
What it should have been. Category Task Completion, with the wording and Evaluation Target Final answer/artifact unchanged.
Catching it in your own task
Read each category beside its target: Agent Behavior and Tool Use only on Trajectory, every subjective row Task Completion.
What to do instead
Retag, do not rewrite: a check on a file takes the deliverable's category, and an explicit ask moves from the subjective block to the objective one.
5.4 The same check written twiceSee it in the viewer
🟡 Costs points · Rubric Criteria - Overall Rubric Quality
Two criteria decided by the same evidence: one inside a broader one, a negative mirroring a positive, or a subjective criterion restating an objective value.
Where the correct guidance is written
QC spec appendix · Rubric Quality Definitions · Overlapping/Redundant Criteria
“Criterion that is either completely redundant because other criteria completely encompass the former or multiple criteria that check for the same thing partly. This can also apply to criteria which have direct semantic overlap with oppositely weighted criteria.”
QC spec appendix · Rubric Quality Definitions · Block Bleed
“A deterministic value check placed in the subjective block, or a presentation judgment placed in the objective block. Count each misplaced criterion. The fix is to move it, never to duplicate it across both blocks.”
A subjective restatement of an objective value is this.
Example 28 (🔗Open in the viewer)
What is wrong. Criterion 9 checks that the two inspection maps are off the dailies index. Criterion 2 already requires the index to list only four files, neither map among them.
The task. Leg A turn 2 hands over three map exports, among them inspections_map_q3.png and inspections_church_hill.png. Leg A turn 3 asks to move inspection project work off the dailies index and recount.
Where it shows. Criterion 2 requires the dailies index to list “only these four files” as dailies, and neither inspection map is among them. Criterion 9 requires that years_in_review.html “does not list inspections_map_q3.png or inspections_church_hill.png” on the index. Both carry weight 5.
How it breaks the standard. The four files criterion 2 names leave out both inspection maps, so an index that passes criterion 2 passes criterion 9. Keeping the maps off the index is worth weight 10 across two rows, and an index that keeps one map loses both. The golden passes both, so nothing flags it; the duplicate costs points.
What it should have been. Keep criterion 2, whose list of four files already keeps both inspection maps off the dailies index, and drop criterion 9.
Catching it in your own task
Name the evidence that decides each criterion: two on the same line are duplicates.
What to do instead
Keep one, the broader where one swallows the other. A subjective restatement keeps only its presentation property, or goes.
6. The subjective block
Ten or more criteria, no more than 30% of the weight, each grading a property of the render rather than your golden's wording, and each naming its file when the task ships more than one. Check all four before you submit.
6.1 Subjective criteria that never name the fileSee it in the viewer
🟡 Costs points · Rubric Criteria - Subjective Block Scope
Subjective criteria describe features without naming the file that carries them, usually after the first one or two.
Where the correct guidance is written
Guidelines · 5.1 Rubric Fundamentals
“Include the exact expected value, filename, classification, decision, or other fact needed to determine whether it is Present or Not Present.”
8.1 applies it to the subjective block.
Guidelines · 8.1 Writing Subjective Rubrics
“The element makes the criterion self-contained by telling the reviewer where to look.”
With more than one file, that includes the filename.
Example 29 (🔗Open in the viewer)
What is wrong. None of the 10 subjective criteria names a file, though Leg A asks for three, so each leaves the grader to choose which file it grades.
The task. Leg A turn 1 asks for hh_migration_manifest.csv and a running note, hh_migration_notes.md in the golden. Leg A turn 4 adds move_walkthrough.mp4, a video that shows the album table.
Where it shows. None of the 10 subjective criteria names a file, and the golden archive holds four: hh_migration_manifest.csv, hh_migration_notes.md, move_walkthrough.mp4 and voice_note_transcript.txt.
How it breaks the standard. The task ships more than one file, so each subjective criterion has to name the one it grades, and none does. Subjective criterion 2's pass and fail marks and subjective criterion 3's table columns also appear in hh_migration_notes.md, an album table with “✅ clean” rows, so the grader has to decide which file is meant. It costs points.
What it should have been. Every subjective criterion opens with the file it grades: “In move_walkthrough.mp4, passed and failed outcomes are distinguished by colour or a mark, not by wording alone.”
Catching it in your own task
With more than one file, every subjective criterion contains a filename.
What to do instead
Open each one with “In
6.2 Too few subjective criteria, or too much weight on themSee it in the viewer
🟡 Costs points · Rubric Criteria - Subjective Block Scope
The subjective block holds fewer than ten criteria, or more than 30% of the total weight. Either alone is the error.
Where the correct guidance is written
Guidelines · 8.1 Writing Subjective Rubrics
“The expected number of subjective rubrics is 10 or more, but do not optimize for the count itself.”
QC spec appendix · Multi-Turn Authoring Standards · Writing Subjective Rubrics
“Same weight set {-5, -3, -1, +1, +3, +5}; block capped at 30% of total task weight.”
Signs are ignored on both sides of the division.
Example 30 (🔗Open in the viewer)
What is wrong. The subjective block carries 32 of 103 absolute weight, 31.1%, over the 30% cap. Its twelve criteria clear the count of ten; the weight alone is the error.
The task. The objective block holds 21 criteria worth 71 absolute weight; the subjective block holds 12 worth 32, with subjective criteria 1 and 2 at +5.
Where it shows. The subjective block carries 32 of 103 absolute weight: 31.1%.
How it breaks the standard. Signs ignored, so subjective criterion 11's -3 counts as 3: 32 over 103 is 31.1%. Twelve criteria clear the count of ten, so the weight alone breaks the cap, and 1.1 points over is still over. No single row is wrong; only dividing the sums shows it. It costs points.
What it should have been. Lower subjective criterion 1 from +5 to +3: 30 of 101, 29.7%. Adding the checks Leg A turns 2 and 5 ask for and nothing grades widens the margin.
Example 31 (🔗Open in the viewer)
What is wrong. The subjective block holds five criteria, where the Guidelines expect ten or more, and none grades the cutting diagrams or the scribe plan Leg A turn 1 asks for.
The task. Leg A turn 1 asks for front_room_builtin.html: a cut list in mm, how to scribe the edges that meet the wall and ceiling, and a cutting diagram for each sheet.
Where it shows. Five subjective criteria, on heading levels, a page that does not report an unread source, table header rows, part names in the cut list, and content inside the page width.
How it breaks the standard. Objective criteria 14 and 25 check the parts in the three cutting diagrams, and objective criterion 13 that the scribe plan names the part at each edge. No subjective criterion looks at how either reads, so an unreadable diagram scores the same as a clear one. It costs points.
What it should have been. Ten or more subjective criteria, region by region, each naming front_room_builtin.html, among them “In front_room_builtin.html, each part rectangle in the cutting diagrams is labelled inside its own outline.”
Catching it in your own task
Count the rows, ten at least, and divide the subjective weight by the total, signs ignored.
What to do instead
Add a criterion for each region nothing grades. Over the cap, add the objective checks your prompts call for, then lower overstated subjective weights.
6.3 Subjective criteria that require your golden's wordingSee it in the viewer
🟡 Costs points · Rubric Criteria - Overall Rubric Quality
A subjective criterion requires the exact heading or label your golden happens to use, words no prompt contains.
Where the correct guidance is written
QC spec appendix · Multi-Turn Authoring Standards · Writing Subjective Rubrics
“Name ONE observable property of the RENDERED artifact. The judge sees the render, not the intent behind it.”
A property, not one build's wording.
Guidelines · 8.1 Writing Subjective Rubrics
“If anything related to the render is asked in the prompt, it requires an objective rubric ONLY.”
A label you chose is graded nowhere.
Example 32 (🔗Open in the viewer)
What is wrong. Nine of ten subjective criteria require the golden page's own headings and labels, such as “Claim 1 - May trip”, which none of the three Leg A prompts asks for.
The task. Leg A turns 1 to 3 ask for reimbursement_status.html: for each of two mileage claims, whether the money reached Noor, and the deadline for an overdue one.
Where it shows. Subjective criteria 1 and 4 require the golden's own card headings, "Claim 1 - May trip" and "Claim 2 - August trip", and its exact header row, "Claim, Reached your account?, Due by 28 Aug?, and Verdict".
How it breaks the standard. No Leg A prompt asks for these headings; they are the golden's choices. Each criterion copies the golden page, so nothing flags it, but a correct page with its own headings fails subjective criteria 1 and 4 and seven more. Only subjective criterion 2 names a property without quoting text.
What it should have been. Subjective criterion 1 grades the layout, not the words: “In reimbursement_status.html, the two claims are presented as two cards side by side rather than one after the other.”
Catching it in your own task
Search the prompts for every label a subjective criterion quotes. Not found: it grades your wording. Found: it belongs in the objective block.
What to do instead
Name the element by what it is, and grade how it is presented, not what it is called.
7. Closing the task
The golden is a deliverable, not a run you stopped. Check every value against its source and every revision for wording it left behind, then run the close out before you submit.
7.1 The golden states something its own sources contradictSee it in the viewer
🔴 Fails the task · Golden Solution - Artifact Completeness
The golden states a value its source, or its own page, contradicts, often one true at an earlier turn. Nothing graded catches it.
Where the correct guidance is written
QC spec · Golden Solution - Artifact Completeness
“Subjective rubrics and pairwise scoring are graded relative to it, so a defective golden corrupts every subjective score on the task.”
Guidelines · 7. Golden Solution - Leg B
“Every factual claim is grounded in the actual inputs: images opened, videos watched, PDFs read. Nothing is asserted from a filename alone.”
Including the values no criterion grades.
QC spec · Golden Solution - Artifact Completeness
“the golden asserts a finding the inputs do not support.”
One of the shapes that fail the task.
Example 33 (🔗Open in the viewer)
What is wrong. The golden gives the Friday custody handover as 13:30 to 15:30. The calendar photo and the calendar service both put it at 18:30.
The task. Leg A turn 2 hands over the school calendar, IMG_0814_calendar.jpg, and asks how many custody handovers clash with travel. Its margin notes give the handover time as 18:30.
Where it shows. Model B, Leg B turn 4, reads the margin as “13:30/15:30”, and the golden dashboard carries it into three places, among them “on-site until 17:00 Fri, can't make the 13:30-15:30 exchange”.
How it breaks the standard. The golden is the answer the subjective block is graded against, so every value in it has to match the inputs, graded or not. No criterion grades the time, and a 17:00 finish misses the handover at either time, so every criterion still passes and nothing flags it.
What it should have been. 18:30 in all three places, reached with a Leg B cleanup turn to Model B that points at the handover time, not its value, before you complete Leg B.
Example 34 (🔗Open in the viewer)
What is wrong. The golden page counts 16 label posts, in its header and in its cards, but a note further down still explains “why the count is 15, not 17”.
The task. The export holds 17 posts, and two artist account posts come out, leaving 15. Leg A turn 2's shot of Marisol's queue adds a sixteenth.
Where it shows. “Two artist-account posts (C16, C17) go out under @sunnyokafor — under P11 not ours to review, so not shown. That's why the count is 15, not 17.”
How it breaks the standard. Fifteen was right until Leg A turn 2. When the queue added a post, the header and the cards moved to 16 and the note did not, so the golden contradicts its own page. No criterion grades the note, so nothing flags it; a stale count like this costs points.
What it should have been. The note agrees with the page: “That is why 15 of the 16 cards come from the export.”, fixed with a Leg B cleanup turn to Model B before you complete Leg B.
Catching it in your own task
Point every ungraded value at its source line, and check each total or count against the same page.
What to do instead
Before you complete Leg B, send Model B a cleanup turn that points at the area, never the value, then check every place the value appears.
7.2 A revision that leaves the old wording standingSee it in the viewer
🔴 Fails the task · Multi-Turn - Revision Turn Handling
A revision changes what was asked but leaves wording that depended on it, so the artifact states the old position and the new one.
Where the correct guidance is written
QC spec · Multi-Turn - Revision Turn Handling
“FAIL: both present but contradictory: March rows gone and the flags survived, but the total at the bottom still includes the March amounts.”
Guidelines · 6.3 Modifiers
“The golden has to show the edit applied to work that already existed: earlier content kept, the change visible, nothing left stale that depended on it.”
Example 35 (🔗Open in the viewer)
What is wrong. The golden foia_register.html puts the newsletter teaser on RPD-26-2204 yet says “The newsletter teaser rides the same Records batch”, and labels a count from acknowledgment “days since filing”.
The task. Leg B turn 6, to Model B, asks to move the newsletter teaser from the records row to RPD-26-2204, the slip's request, and to count sitting time from acknowledgment.
Where it shows. The walk in section still says “The newsletter teaser rides the same Records batch”, and every row's detail still reads “days since filing”.
How it breaks the standard. Model B moved the teaser to RPD-26-2204 after Leg B turn 6, so the walk in section lists the newsletter there and then says it rides the Records batch: two answers on one page. Each row's detail calls an acknowledgment count “days since filing”. No criterion reads either sentence, so nothing flags them.
What it should have been. The walk in sentence follows the teaser or goes, and the detail reads “days since acknowledgment”: one Leg B cleanup turn to Model B before you complete Leg B.
Catching it in your own task
After each revision turn, search the artifact for every value, label and name that moved.
What to do instead
Send Model B a Leg B cleanup turn that points at the stale area, never at the new wording.
7.3 The five minute close out nobody runsSee it in the viewer
🔴 Fails the task · Golden Solution - Trajectory Exclusion · Golden Solution - Artifact Completeness · Trajectory - Golden/Preferred Run Selection
Five cheap checks, often skipped: stray files in the golden archive, a filename spelled two ways, a golden that stops early, no starred run, and clauses a grader must decode. The last two cost points.
Where the correct guidance is written
QC spec · Golden Solution - Trajectory Exclusion
“golden/ ships finished artifacts only. The external judge receives the golden ARTIFACTS and never the golden trajectory — anything left in the folder describing how the answer was reached leaks the solution path into grading.”
A caveat about the deliverable stays.
Guidelines · 7.1 What Changes from Leg A
“The preferred run (if multiple exist) must still be marked as ‘Preferred‘ with the star.”
Also in Leg B. A missing star costs points.
Guidelines · 7. Golden Solution - Leg B
“Satisfies every objective requirement, with every required file present and named exactly as requested.”
The prompt's spelling wins.
Guidelines · 1.2.3 Create the Initial Prompt and the Follow-ups
“Contains avoidable grammatical or wording issues.”
A mark of a weak prompt; in a criterion, a Minor issue.
Example 36 (🔗Open in the viewer)
What is wrong. The golden archive holds five files where the Leg A prompts name three. make_baseline_video.py names each answer's “Source of truth”, and baseline_quiz_page.jpg is the input IMG_4642.jpg renamed.
The task. The four Leg A prompts name three files: baseline_check.mp4, baseline_check_script.md, which Rosalba prints for the instrument case, and MEMORY.md.
Where it shows. The golden archive also ships make_baseline_video.py, whose docstring names each answer's “Source of truth”, and baseline_quiz_page.jpg. baseline_check_script.md, the script Rosalba prints, tells its reader to run “python3 make_baseline_video.py”.
How it breaks the standard. The golden archive ships finished artifacts only. make_baseline_video.py shows how the video was made, down to which input each answer came from, and baseline_quiz_page.jpg is an input under a new name. The script Rosalba prints even tells her to run the generator. Model B, Leg B turn 1, announced it, and no Leg B turn removed it.
What it should have been. The golden archive holds only baseline_check.mp4, baseline_check_script.md and MEMORY.md, with no rebuild line in the printable script: one Leg B cleanup turn to Model B before you complete Leg B.
Example 37 (🔗Open in the viewer)
What is wrong. Objective criterion 22 asks for a fee of “%0.00” where $0.00 is meant, objective criterion 7 omits the day, and milestone 4 ends in a clause that does not parse.
The task. Leg A turn 1 sets the order for Monday 29 June 2026, and Leg A turn 3 moves it to Monday 6 July 2026.
Where it shows. Objective criterion 22 wants a fee of “%0.00”, objective criterion 7 dates its call only “for the Mon Jul 2026 order”, and milestone 4 ends “an apartment the sheet doe snot marks keeps all the requests”.
How it breaks the standard. A grader must decode objective criterion 22's “%0.00” as $0.00, and piece objective criterion 7's date together from objective criterion 3. In Leg B you judge Model B against milestone 4, and a user simulator replays it later: its broken clause leaves each to guess the meaning. These cost points.
What it should have been. Objective criterion 22: “pantry_audit.html shows a delivery fee of $0.00 for the Mon 6 Jul 2026 order.” Milestone 4: “an apartment the sheet does not mark keeps all its requests”.
Catching it in your own task
Check the archive against the prompts' filenames, the run picker for “Preferred: 1/1 selected”, each filename character by character, each clause read straight through, and the golden reaching the last prompt.
What to do instead
Remove anything not named, clean build notes out with a Leg B cleanup turn, star the golden run, match the prompts' spelling, and rewrite any clause you read twice.
The order that keeps the cost down
- Confirm the universe returns every record your rules point at.
- Check the deliverable type against the complexity bar.
- Build the inputs, then prompts that they and the universe can support.
- Run Leg A, write the objective rubric with negatives from Leg A's wrong turns, and rate Model A.
- Write the milestones from the Model A conversation, and check them before Leg B.
- Run Leg B to the golden. Before you complete it, grade the golden against every objective criterion, check its values and revisions, and fix anything with a Leg B cleanup turn to Model B.
- Write the subjective block with the Model A render and the golden side by side, and rate both models.
- Run the close out, and confirm Model A fails at least half the weight, both blocks counted.