agent brief/2026-10-29

Benchmark Crowns Get Contested

Gemini 4 Carbon claims a top index score while its own reported OSWorld row trails Opus 5.5 — and DABStep's 16% floor shows who's really being scored.

time to read23m
time saved397 min
sources566
Benchmark Crowns Get Contested
λsynopses
  • Contested Crowns Google's Gemini 4 Carbon claims 59 on Artificial Analysis v5.0, reportedly ahead of Opus 5.5 at 58 — yet its own OSWorld row lags.
  • Harness Matters DABStep's 450+ tasks topped out at 16% accuracy at release, with reasoning models at 0% under a ReAct prompt per Adyen.
  • Hold Loosely Name the harness, keep router decisions reversible.
#tags
subscribe
system operational
end :: 566 signals processed█
keep reading
→recent briefs
2026-11-13

Agents Get Safer, Faster, Harder to Grade

- **Idempotency Goes Default** Builders converge on retry-safe actions and approval gates before shipping side effects. - **Speed Over Frontier** Gemini 4 Hydrogen edges Argon 51–50 on Artificial Analysis' index while finishing tasks 2.5x faster. - **Benchmarks Reset** Hugging Face claims GAIA's top spot as Meta's Gaia2 reframes evaluation around acting, best score 42% pass@1.

2026-11-12

Approval Must Bind the Call

- **Gates Go Granular** Reddit builders converge on one lesson: approval must bind to the exact tool call, not the session or a timer. - **Breach Postmortem** Hugging Face's timeline shows an agent with legitimate tool access chaining permissions into an incident; Microsoft, OpenAI, Akka and Scalekit describe invocation-level gates. - **Memory Math** MosaicLeaks reports leakage worsening from 34% to 51.7% when agents train purely for task performance — prompting alone may not hold.

2026-11-11

Benchmarks Meet Their Limits

- **Orchestration Gap** Einsia's WorkflowArena holds the worker model fixed and varies only the planner, arguing coordination is separate from raw model power — authors' own claim, not an independent re-run. - **Scorecard Asterisks** H Company's Holo4 claims 85.2% on OSWorld for its research-only 27B, while the Apache-2.0 35B-A3B scores 80.8% — all vendor-reported. - **Failure Taxonomies** MAST's 14 failure modes and VAKRA's "structural" multi-step tool-chain failures point practitioners at harness and design, with reports of harness choice halving token usage.