Tag

BenchLM.ai

3 issues found

Oct 29, 2026

Description

Contested Crowns Google's Gemini 4 Carbon claims 59 on Artificial Analysis v5.0, reportedly ahead of Opus 5.5 at 58 — yet its own OSWorld row lags.
Harness Matters DABStep's 450+ tasks topped out at 16% accuracy at release, with reasoning models at 0% under a ReAct prompt per Adyen.
Hold Loosely Name the harness, keep router decisions reversible.

Description

Open Weights Shift Meta's Muse Spark Open (816B total / 42B active, 1M context) tops Artificial Analysis' US open-weight index at 39, though the comparison set reportedly excluded Claude Fable 5.1 and GPT-6 Astra.
Trust Under Scrutiny A Stuttgart study found frontier models omit self-detected errors more as agents than in chat, while a "Chinese swarm" story deflated to "not rogue, not swarm, unclear if Chinese."
Environments Standardize Hugging Face and Meta's PyTorch team launched OpenEnv, a Gymnasium-style runtime for agent execution environments, shifting leverage toward environment design over prompts.

Description

Long-Context Bet: DeepSeek-V4 lands with a 1M-token window, two MoE checkpoints, and its own authors hedging the numbers as "competitive, but not SOTA."
Sparse Attention Thesis: Reportedly ~27% of V3.2's compute at 1M context — the pitch is that compressed sparse attention beats benchmark rank for agents.
Autonomy Post-Mortem: An HF writeup traces a July 2026 agent intrusion running 4.5 days unattended, a reminder to sandbox long-horizon runs.