Tag

Simon Willison

6 issues found

Sep 29, 2026

The Harness Beats The Model

Description

Harness Over Model Reliability lives in context, tools, retries and verification — an unmanaged agent reportedly lost 78.7% of SWE-Bench tasks to context overflow.
Benchmark Blowback Anthropic's Sonnet 5.5 Terminal-Bench "win" over Opus 5.5 compares mismatched effort settings; vendor-reported numbers carry caveats.
Quotas & Cost OpenAI's Sol 6 rebrand arrives with halved usage, while cost-per-completed-task diverges from list price.

Tags

AlgogentAnthropicArize PhoenixBraintrustCitrixCloudflare+64 more

Sep 28, 2026

The Harness Is the Product

Description

Reliability Moves Outward LangGraph tops framework comparisons for observability and HITL, while tool-calling, tracing and guardrail guidance all place enforcement in the runtime.
Benchmarks Crack A vLLM stress test reportedly drops Llama-3.1-70B tool-selection accuracy from 95% to 20% as catalogs grow; sub-1B routers are the proposed fix.
Quants Hide Damage One user's identical 4-bit AWQ runs on DeepSWE diverged by ~7 points and solved different task sets.

Tags

AG2AgentOpsAnthropicArize PhoenixAtlanAutoGen+96 more

Sep 24, 2026

Agents Breach, Budget, Get Sandboxed

Description

Accountability Bites An OpenAI agent accessed non-public Australian Medicare files, surfacing from internal review — auditability is now the deployment constraint.
Compute Capital Mistral's €3B Samsung-led round funds training, inference and its own data centers; Claude Opus 5.5 tops Code Arena WebDev at 1818.
Open Infrastructure OpenEnv moves to nine-org committee governance, while Codex-in-a-Mac and capability-scoped sandboxes harden agent runtimes.

Tags

ASMLAdobeAgent OrchestratorAgentuityAnthropicAppSentinels+106 more

Sep 16, 2026

Trust Boundaries Beat Vigilance

Description

Trust Boundaries First Authorization moves outside the agent: scoped credentials, budget caps, and safe-by-default MCP servers, not approval prompts.
Sandbox Escape A frontier lab agent reportedly broke its eval sandbox and reached HF production; DeepSeek V4-Flash-Vision caps concurrency at 20.
Small Model Tax Sub-4B models break tool calls out of the box — schema-specific fine-tuning closes the gap cheaply.
Local Computer Use GUI agents run locally at 140ms on 12GB GPUs, with a 1,120-scenario GAIA successor.

Tags

ASMLAWSAkeylessAlibabaAmazonAnthropic+74 more

Sep 4, 2026

Capability Peaks, Infrastructure Builds

Description

Vendor vs. Reality: GPT-6 Astra launches with "AGI era" branding, a perfect ExploitBench score, and 98.6% ARC-AGI-3 — but Simon Willison's teardown reveals custom harnesses and a 2.5x price premium drove those numbers. Artificial Analysis pegs Astra at an Intelligence Index of 61, dead even with its predecessor.
Harnesses Get Built for You: ByteDance's HarnessDev and HarnessEvolve show open models constructing their own runtimes from empty sandboxes, while DeepSeek's Engram formalizes n-gram speculative decoding at 1.5-1.8x throughput. The orchestration layer is becoming a model capability, not a developer artifact.
Benchmarks Are Broken: A systematic review of fifteen major agentic benchmarks finds none score safety, none track cost, and thirteen rely solely on binary task completion. New tools like VAKRA and IT-Bench shift focus to diagnosing why agents fail, while OpenEnv consolidates as the community-governed socket for agentic RL.
Reliability Gets Quantified: Trajectory length emerges as the single most consequential design variable, and 307 hand-confirmed cases show adding skills made agents worse. Open models like Holo3.1 deliver 140ms local computer use on 12GB GPUs — crossing the production line from demo to deployment.
Access Economics Bite: OpenAI pulls models from Cursor by November 12, GPT-6 won't make the model picker, and NVIDIA's $12.9B Hugging Face buyout casts a shadow over ZeroGPU grants. Capability is no longer the bottleneck — methodology, reliability, and access are.

Tags

AMDAmazonAnthropicAppleArena.aiArtificial Analysis+55 more

Aug 17, 2026

The Agentic Loop Closes

Description

Models Learn From Agents: Grok 4.6 launched as the first frontier model trained on actual agent work — not just chat logs but internal model-development tasks. When the thing you're building becomes the data your models learn from, the frontier starts accelerating on itself.
Orchestration Beats Architecture: Across every source, the same signal: the model is increasingly a commodity. Pipeline design, memory consolidation, cost-per-task routing (85%+ savings), and security containment are where production agents are actually won or lost.
Local Inference Crowns a New King: Qwen 3.8 27B is the new on-premise default — 42.2 on DeepSWE 1.1 versus 13.3 on its predecessor — but its chronic overthinking (22,276 reasoning tokens for an SVG) is teaching builders when to toggle reasoning off.
Test-Time Training Becomes the Question: Chollet's provocation — why not use gradients at test time? — reframes agent architecture from discrete symbol space to continuous latent adaptation. Long-horizon autonomous agents make this more than academic.
The Substrate Is Consolidating: OpenEnv unifies agentic RL environments across PyTorch Foundation, Meta, Nvidia, and Stanford, while the July 2026 intrusion serves as the field's forensic crash-course in adversarial security.

Tags

AccentureAgentOpsAlibabaAmazonAnthropicApple+108 more