Tag
Peking University
2 issues found
Nov 11, 2026
Benchmarks Meet Their Limits
Description
- Orchestration Gap Einsia's WorkflowArena holds the worker model fixed and varies only the planner, arguing coordination is separate from raw model power — authors' own claim, not an independent re-run.
- Scorecard Asterisks H Company's Holo4 claims 85.2% on OSWorld for its research-only 27B, while the Apache-2.0 35B-A3B scores 80.8% — all vendor-reported.
- Failure Taxonomies MAST's 14 failure modes and VAKRA's "structural" multi-step tool-chain failures point practitioners at harness and design, with reports of harness choice halving token usage.
Tags
AWSAgentOpsAlibabaAmazonAnthropicArize+64 more
286 time saved529 sources29 min read
Nov 9, 2026
Agents Learn to Decide
Description
- Open Weights Rise: Thinking Machines' Inkling 2 Small reportedly scored 81.7% on ARC-AGI-2 at $0.27/task, topping ARC Prize's open-weight evals.
- Cheap Decisions: Decision models like Drex Max claim 183ms answers versus Opus 5.5's 3.5s, reshaping routing economics for builders.
- Prove It Twice: Holo4 adds a third action lane, but IBM, Microsoft and authorization-drift work push repeatability and boundaries.
Tags
ARC PrizeAirbnbAmazonAndreessen HorowitzAnthropicBaidu+45 more
795 time saved5060 sources24 min read