1. The Foundations Every Task Must Get Right
Ask what the page does when someone clicks it. If nothing, it is a document, whatever the Agent Objective calls it.
1.1 The deliverable sits below the complexity barYour golden shows it: an empty search, a Leg B hint sending Model B another way, figures flagged as unconfirmed.
1.2 The task relies on records the universe does not hold2. What you may not tell Model A or Model B
For each Leg B turn, ask what is left for Model B to work out. A turn that states what a criterion grades has answered it.
2.1 A Leg B hint that hands Model B the answerRead each prompt beside the file it hands over, in both legs: any value the model should read from that file is a leak.
2.2 A Leg A prompt that answers its own attachmentRead the input folder as both models will at turn 1, filenames included: price_sheet_for_turn3.pdf fails, price_sheet.pdf costs points, sheet_02.pdf is fine.
2.3 A deferred asset that announces its own turn3. Milestones
Read Modifier and Assets Delivered beside the Leg A prompts: Leg A turn 1 is opening, and each deferred file sits on the Leg A turn that hands it over.
3.1 Modifier and Assets Delivered that do not match the Leg A turnCould a response skip one of the Leg A turn's asks and still pass? Could a correct one fail for lack of a later file?
3.2 Stopping rules that do not gate the turnList each milestone's Turn beside the Leg A prompts: every Leg A turn needs one per ask. Identical neighbours are duplicates.
3.3 A Leg A turn no milestone covers, or the same milestone written twice4. A rubric that separates two runs
Negatives should be about a quarter of the set. Present means failure, so a negative describing a good run is inverted.
4.1 A rubric that can only reward, never penalisePair each prompt requirement with its criterion, and each criterion with the sentence that asks for it.
4.2 A requirement nothing grades, and a criterion nothing asked forCount the axes each criterion exercises and read off its weight.
4.3 Weights that do not separate difficulty from lookupCount the objective rows that target Trajectory.
4.4 More trajectory criteria than the Guidelines allowDivide the weight of the criteria Model A fails by the total, both blocks counted, signs ignored.
4.5 A run that does not fail enough of the rubric5. Criteria a correct run can pass
Before you complete Leg B, grade the golden against every objective criterion, and mark every Desired Outcome rule no turn asks for.
5.1 A criterion the golden itself failsFor each criterion, name its source line and where your golden shows it, then ask what wrong run still passes.
5.2 Criteria a correct run cannot pass, or a wrong one canRead each category beside its target: Agent Behavior and Tool Use only on Trajectory, every subjective row Task Completion.
5.3 Category and Evaluation Target filled with the wrong kind of valueName the evidence that decides each criterion: two on the same line are duplicates.
5.4 The same check written twice6. The subjective block
With more than one file, every subjective criterion contains a filename.
6.1 Subjective criteria that never name the fileCount the rows, ten at least, and divide the subjective weight by the total, signs ignored.
6.2 Too few subjective criteria, or too much weight on themSearch the prompts for every label a subjective criterion quotes. Not found: it grades your wording. Found: it belongs in the objective block.
6.3 Subjective criteria that require your golden's wording7. Closing the task
Point every ungraded value at its source line, and check each total or count against the same page.
7.1 The golden states something its own sources contradictAfter each revision turn, search the artifact for every value, label and name that moved.
7.2 A revision that leaves the old wording standingCheck the archive against the prompts' filenames, the run picker for “Preferred: 1/1 selected”, each filename character by character, each clause read straight through, and the golden reaching the last prompt.
7.3 The five minute close out nobody runs