All mistakes/Section 5: Criteria a correct run can pass

5.3Category and Evaluation Target filled with the wrong kind of value

Costs pointsRubric Criteria - Overall Rubric Quality

The mistake

Category or Evaluation Target does not fit the criterion: Agent Behavior or Tool Use on an artifact, a category in the target field, or a subjective row tagged Instruction Following.

Do this instead

Retag, do not rewrite: a check on a file takes the deliverable's category, and an explicit ask moves from the subjective block to the objective one.

Where it happened

2 real tasks. Each shows the spot and what it should have been; click any line to open the task there.

the words at faultthe row or field at faultmissing from the taskwhat is righta task file

Example 26Operations & QA · 5 Leg A turns

What is wrongCriteria 17 and 30 grade what the running note says, yet their Category is Agent Behavior, which always targets Trajectory. A check on a file grades the deliverable: Task Completion.

Leg A turn 1 asks for “a running note of where each album stands and what the rules are”. Criteria 17 and 30 grade what that note states.

Where it shows

Criterion 17+3categoryAgent Behavior

A markdown or text file in the workspace states the checklist rule that an album off on both count and size means stop and hold everything.

Criterion 30+3categoryAgent Behavior

A markdown or text file in the workspace records that bc eagles 3-21 is absent from both destinations and that finale reel FINAL is short on the T7.

What it should have been

Criterion 17+3categoryTask Completion

Criterion 30+3categoryTask Completion

How it breaks the standardIn the golden, both are read off hh_migration_notes.md, its running note, so they grade the deliverable, not the trajectory. Wording and target are right; only the category is wrong. The criteria still pass and fail the right runs, so grading never exposes it, but the client receives wrong rubric data, which costs points.

The rule it breaksGuidelines · 5.4 Category & Evaluation Target If you wrote Agent Behavior against an artifact, a state change or the final message, you picked the wrong category

The full write up of Example 26

Example 27Visual Learning · 4 Leg A turns

What is wrongSubjective criterion 2, on key markers coloured like their muscle's heading, is tagged Instruction Following. No Leg A prompt asks for colours, and the subjective block takes Task Completion only.

Subjective criterion 2 grades whether the key markers in deltoid_plate_study.html use one colour per muscle, matching that muscle's section heading.

Where it shows

Subjective criterion 2+3categoryInstruction Following

Key markers in deltoid_plate_study.html use one color per muscle, the same color as that muscle's section heading, so a marker's muscle can be told at a glance.

What it should have been

Subjective criterion 2+3categoryTask Completion

How it breaks the standardNone of the four Leg A prompts mentions colour, so subjective criterion 2 grades no instruction; it grades how deltoid_plate_study.html looks, which is Task Completion. Had a Leg A prompt asked for the colours, the check would belong in the objective block instead. The wording is right; only the tag costs points.

The rule it breaksGuidelines · 8.4 Categories and Evaluation Targets for Subjective Rubrics Task Completion is the only valid category for subjective rubrics.

The full write up of Example 27

Catch it in your own task

Read each category beside its target: Agent Behavior and Tool Use only on Trajectory, every subjective row Task Completion.

Where the correct guidance is written

Section titles as they print, so you can search for them in the document.

Guidelines · 5.4 Category & Evaluation Target

Agent Behavior, Target: Trajectory, always. If you wrote Agent Behavior against an artifact, a state change or the final message, you picked the wrong category: you are grading the deliverable, which is Task Completion, Instruction Following or Factuality.

Guidelines · 8.4 Categories and Evaluation Targets for Subjective Rubrics

Task Completion is the only valid category for subjective rubrics.

Why it matters

Category says what kind of check it is; target says where the grader looks. Both go to the client with the rubric data, so a wrong pair is wrong data even when the criterion is right. Agent Behavior and Tool Use grade how the run got there, so they sit on Trajectory: one pointed at a file, a state change or the final message is grading the deliverable and needs the deliverable's category. In the subjective block Task Completion is the only valid category, because anything the prompt explicitly asked for belongs in the objective block by definition.

Every example taken apart, with the task, how it breaks the standard and what it should have been: 5.3 Category and Evaluation Target filled with the wrong kind of value, in the explanations →