All mistakes/Section 1: The Foundations Every Task Must Get Right
1.2The task relies on records the universe does not hold
Fails the taskTrajectory - Feasibility With Tools
The mistake
A rule reads off records the assigned universe does not hold, such as a catalog or a policy page, while prompts, inputs and criteria read as if they were there.
Do this instead
Before the run, query the universe once per rule and rewrite any rule whose record is missing. If it only surfaces in Leg B, redesign the task rather than hint again.
Where it happened
One real task. Each shows the spot and what it should have been; click any line to open the task there.
the words at faultthe row or field at faultmissing from the taskwhat is righta task file
Example 2Commerce & Product · 3 Leg A turns
What is wrongpantry_rules.md prices every line from Fresh's catalog and the delivery fee from Fresh's Delivery Policy. The universe's catalog holds none of the five board items, and its policies are empty.
…the two endpoints that serve product pages and policy text (search_shop_catalog and search_shop_policies_and_faqs) return zero results for everything…
What it should have been
Before prompt 1
Search the universe's catalog for the five items, and its policies for the fee. Both come back empty, so the rules change before the run.
Rules 1 and 5 cite only records the universe returns.
How it breaks the standardPricing every card and what each apartment owes rests on rules 1 and 5, so neither Model A nor Model B can do the main request. The golden keeps Tom's board prices, flagged unconfirmed, and criteria 16, 17 and 26 want Fresh's $0.55 a lb bananas, $5.45 spinach and $4.99 fee, which no tool returns.
The rule it breaksGuidelines · 1.2.1 Start With the UniverseThe universe combined with the multimodal input has to sustain the opening prompt and every follow-up you intend to add
Your golden shows it: an empty search, a Leg B hint sending Model B another way, figures flagged as unconfirmed.
Where the correct guidance is written
Section titles as they print, so you can search for them in the document.
Guidelines · 1.2.1 Start With the Universe
The universe combined with the multimodal input has to sustain the opening prompt and every follow-up you intend to add
Guidelines · 4. Valid vs. Invalid Model Failures
Data that is not in the server, asked for anyway, and the model penalised for not finding it.
Listed under “Not valid: your defect, or the platform's”.
1 more source
QC spec · Trajectory - Feasibility With Tools
The primary request is impractical or impossible and can't be answered by the tools available or enabled for the task.
Fails the task when the missing records carry the main request; costs points otherwise.
Why it matters
A rule pointing at data no tool returns cannot be followed by any run, so every criterion downstream of it is unreachable. Not wrong. Impossible. It takes the difficulty with it too: if that data was the second system the task cross checks against, every cross check has nothing on the other side, and the task is no longer multi system.
Example 2The task relies on records the universe does not holdFails the task
Commerce & Product3 Leg A turnstask 6aac6ee4f0c334e21057f9e6
From the taskAssigned universe
openclaw_mm_thaddeus_okonjo_commerce_product
From the taskLeg A prompt, turn 1, to Model A
Leg A turn 1
hey so our building fresh order goes in monday 6/29/26. tom threw together pantry_board_draft.png for the neighbors, signup_sheet.jpg is the sheet from the laundry room and the house rules are in pantry_rules.md. can you check his listings before anyone sees it and make me pantry_audit.html - verdict+fixed card for each listing. and what each apt owes. people should be able to pick the apt and only see their part
From the taskFrom the Leg B conversation, with Model B
Model B · Leg B turn 2
Updated **`/app/outputs/pantry_audit.html`** (v2). Here's what changed and why.
## First, the honest blocker
I did go to Fresh directly on your account this time — but the two endpoints that serve product pages and policy text (`search_shop_catalog` and `search_shop_policies_and_faqs`) return **zero results for everything**, including items I can prove are in your catalog (I searched Sourdough loaf, Jasmine rice, Romano, Gala apples, 2% milk — all things you've ordered — and all came back empty). The account itself is fine (profile, history, cart, subscriptions all work), and the cart *does* return real Fresh price/pack data — but only for a variant ID that's already in your history, and **none of the five pantry items are**. So I can't open their live pages, and I won't dress Tom's numbers up as Fresh's.
## You were right about the kept claims
Checking each claim against the rule-7 standard ("Fresh's *catalog or policies* back it" — a photo label or a nutrition line doesn't count), **three claims I kept in v1 shouldn't have been**:
- **Spinach "pre-washed"** → that's off the packaging, not Fresh's catalog copy → **removed** until the listing shows it.
- **Eggs "free-range / Grade A"** → unverifiable → **removed** (not just flagged).
- **Avocado "Organic"** → I'd kept it outright; it's actually **can't-verify** from Fresh. It matters because it's the *only* thing 2A ends up receiving, so it now carries a hard "confirm on Fresh" gate.
The ones I removed in v1 (potassium, vitamin C, 30-day return, spinach "organic") were correctly removed. Net result: **once you require Fresh to back each line, the compliant cards carry no descriptive claims at all right now** — just name, pack, price/unit — and a claim only goes back on after you see it on the product's Fresh page. There's a full claim-by-claim table in section 1.
## What's solid vs. provisional
- **Final** (rules + sheet + photo + your card statement): all verdicts are **Fix**; the per-apartment *quantities*; 2A's dropped items; 5D's "any is fine" override; the S&S logic; delivery must be split proportionally, not evenly.
- **Provisional / `Pull from Fresh`**: every price, the delivery fee/threshold, the avocado organic flag and single-vs-bag pack, and therefore the dollar amounts. The owed figures are shown with Tom's numbers only as placeholders and clearly tagged, plus one new subtlety — the S&S "2 lb" depends on Fresh's real banana price ($1.02 ÷ price/lb), so that's on the checklist too.
Section 4 is a short "confirm on Fresh before you publish" list so that when those endpoints come back (or you check the app), you can drop in real values and the numbers finalize. Want me to keep retrying the catalog endpoint, or take a run at pulling the delivery/return policy specifically?
You, as the user · Leg B turn 3
the prices are still toms numbers and theres no delivery on the totals - i need the
real fresh prices and fresh's own delivery policy in there, not placeholders. the
buildings fresh account is connected and all five items are sold on fresh, so if the
search is coming back empty try getting to the product pages and the policy pages
another way. also the fixed cards have ended up with no claims left on them at all,
that cant be right either
From the taskObjective criteria 16, 17, 26
16
The corrected Bananas card in pantry_audit.html prices the 1lb pack at $0.55 per lb.
In turn 1 the model sets the Mon 29 Jun 2026 delivery fee at $4.99 on an items subtotal of $48.98
+5targetTrajectorycategoryAgent Behavior
Task filepantry_rules.md
Look at The rules file the task ships. Rules 1 and 5 send every price and the delivery fee to records the assigned universe does not hold.
pantry_rules.mdinput file
# 123 Main St shared pantry - group order rules
(what we agreed at the June building meeting)
1. We charge what Fresh charges us for what actually goes in the cart. Every line is priced at the pack option we add to the cart from Fresh's catalog. If an item has no pack options, use its catalog price.
2. For each item we buy the cheapest mix of Fresh's pack options that covers everyone's total. Never less than what people asked for.
3. "Organic only" apartments only get items that Fresh's own catalog marks as organic. Anything else they asked for is dropped, not swapped. Whatever an apartment said last is what counts.
4. 4B's Subscribe & Save: whatever it's due to bill in the week the order goes in (our weeks run Monday to Sunday) comes off 4B's request for that item.
5. The building account is not Prime, so no member prices or member deals. For the delivery fee we go by Fresh's Delivery Policy and nothing else.
6. Splitting: a line's cost is split by how many units each apartment gets on that line, each share rounded to the cent (if the rounded shares don't add up to the line cost, the difference goes on whoever has the most units on that line). An apartment's items total is the sum of its line shares. Each apartment's delivery share = delivery fee x (its items total / everyone's items totals added up), rounded to the cent only at the very end. Any leftover cent, up or down, goes on 4B.
7. Listing cards show: name, the pack option we're buying, price per unit (line cost / total units on that line, 2 decimals), and only claims that Fresh's own catalog or policies back up.
8. Verdicts: publish = card is right as is, fix = card needs changes, pull = nobody ends up ordering it.
Task filepantry_audit.html
Look at The delivered page, saying in its own words that it never reached the catalog.
pantry_audit.htmlgolden artifact
Task filepantry_board_draft.png
pantry_board_draft.pnginput file
Click the image to see it at full size, and again to fit it back.