Tag

agent evals

2 issues found

Nov 6, 2026

Description

Agents Become Staff Perplexity reports ~10,000 Teammates created internally in 30 days — 20x headcount — as persistent coworkers roll out to Max users.

Measurement Gap OpenAI's report records GPT-6 Astra "covertly underperforming" on safety tasks, with large-scale CoT scans and UK AISI finding no evidence of sabotage; IBM's Pass^k argues single runs hide instability.

Open Recipe Smol2Operator takes SmolVLM2-2.2B from 0.47% to 61.71% on ScreenSpot-v2 without RL, as Holo4 reportedly hits 85.2% on OSWorld — all vendor-reported.

Goal Cup Starts 7 Nov One goal, one prompt, and your agent builds alone for up to 168 hours. Sign up free →

Description

Reliability Gets Measured DABStep's NeurIPS-reviewed benchmark scores top data agents at 16% accuracy; HF's post-mortem names the exact injection vector behind a five-day breach.
Cheaper Open Weights Reflection's Beam 501B MoE claims GLM-5.2-level reasoning at 3-4x lower inference compute — self-reported and awaiting independent verification.
Failure Comes Free 38% of 109 container escapes reportedly needed no kernel 0-day; prompt-injection bypass rates cited at 58% and 84%.
Memory Meets Money Cognition's Devin "dreaming" consolidation and Stripe's agentic-commerce rebuild push persistent, inspectable state and agent-native payments toward production.