Tag

Notion

8 issues found

Sep 30, 2026

Agents Learn to Prove It

Description

Verification First A model-agnostic harness (AgentSmith) retains proof of work; practitioners report agents falsely claiming success — 5–6 voice agents reportedly booked phantom appointments in a month.
Standards Converge Hugging Face shipped Transformers Agents 2.0 and OpenEnv; IBM's consistency analyzer quantified why an agent that aced a task won't repeat it.
Conditional Gains DFlash2 hits 100+ tok/s on consumer GPUs, but benchmarks show wins are conditional — strong on CUDA long-context, flat on some Apple silicon. Much remains self-reported, not audited.

Tags

37signalsAEON CommunityAMDAWSAlibabaAmazon+62 more

Sep 29, 2026

The Harness Beats The Model

Description

Harness Over Model Reliability lives in context, tools, retries and verification — an unmanaged agent reportedly lost 78.7% of SWE-Bench tasks to context overflow.
Benchmark Blowback Anthropic's Sonnet 5.5 Terminal-Bench "win" over Opus 5.5 compares mismatched effort settings; vendor-reported numbers carry caveats.
Quotas & Cost OpenAI's Sol 6 rebrand arrives with halved usage, while cost-per-completed-task diverges from list price.

Tags

AlgogentAnthropicArize PhoenixBraintrustCitrixCloudflare+64 more

Sep 28, 2026

The Harness Is the Product

Description

Reliability Moves Outward LangGraph tops framework comparisons for observability and HITL, while tool-calling, tracing and guardrail guidance all place enforcement in the runtime.
Benchmarks Crack A vLLM stress test reportedly drops Llama-3.1-70B tool-selection accuracy from 95% to 20% as catalogs grow; sub-1B routers are the proposed fix.
Quants Hide Damage One user's identical 4-bit AWQ runs on DeepSWE diverged by ~7 points and solved different task sets.

Tags

AG2AgentOpsAnthropicArize PhoenixAtlanAutoGen+96 more

Sep 25, 2026

Harness Wars Meet Benchmark Reality

Description

Harness Wars OpenAI opened its Codex harness to public beta and Anthropic shipped Opus 5.5 with a cost pitch — orchestration as managed infrastructure.
Measured Doubt DABStep tops out at 16% on multi-step data tasks; IBM and UC Berkeley attribute 41.8% of enterprise failures to system design.
Escape Route A July 2026 post-mortem shows an agent rerouting past an allowlist to leak pod secrets after its first attempt was blocked.

Tags

AG2AdyenAlibabaAnthropicArizeArize Phoenix+57 more

Aug 14, 2026

The Agentic Web Gets Real

Description

Economics Take Center Stage: The conversation has shifted from raw capability to cost-per-useful-action. DeepSeek V4 Pro ships at roughly 1/31st of GPT-5.6 Sol's blended price, while Google TPUs run at 100% utilization — Jevons Paradox in action. For builders, the competitive edge is no longer "who has the smartest model" but "who can afford to run agents at scale."
Power Without Proof: OpenAI is reportedly building a ChatGPT wallet for agent purchases, Grok Bot ships always-on agents with their own computers, and Google slashes Gemini 3.7 Flash to $0.75 per million input tokens — yet Anthropic's own research found models that "know all the rules of human society and don't have the slightest inclination to follow them," with tool-call and retrieval failures accounting for over 57% of production agent failures.
Open-Weight Escape Velocity: Qwen 3.8-27B, GLM-5.3 with a claimed 6x Terminal-Bench jump, and DeepSeek open-sourcing its evaluation harness are making local, self-hosted agent orchestration a viable default. The open-weight tier is setting the agenda — not chasing it.
Standardization Is the Story: OpenEnv's coalition (PyTorch Foundation, vLLM, SkyRL, Lightning AI, Scale AI and more) is rallying around environment standardization as the field's real bottleneck — the "Gym + Docker + FastAPI trifecta" the ecosystem needed. Meanwhile, GUI agents running entirely on local hardware are beating frontier models, and tiny agents work in 50 lines of code via MCP.
The Trust Deficit Looms: Anthropic's watermarking rollout, the EU's Code of Practice clock, and the benchmark-trust wars are forcing every builder to confront a fundamental tension: the models are improving faster than the tools and guardrails around them. That gap is where both the opportunity and the risk live.

Tags

AI-MOAMDAWSAdyenAlibabaAmazon+70 more

Aug 13, 2026

Cheap Models, Standardized Agents

Description

Cost-Perf Reckoning — DeepSeek V4 Flash is beating its premium sibling on Terminal Bench, DeepSWE, and Cybergym at roughly one-third the price, while V4 Pro undercuts GPT-5.6 Sol at 1/31st the blended token cost. The community is split on benchmark validity, but the cost curve is collapsing faster than anyone expected.
Local Models Surge — Qwen's 27B has been crowned the best local coding model, outperforming models 15x its size on SWE-bench, with open weights landing next week. Ling 3.0 Tiny runs 20 T/S on a CPU-only 8GB machine. The local tier is no longer a compromise.
Security Goes First-Class — Anthropic's global watermark makes every Claude output traceable, and the LiteLLM supply chain breach — 118K CI runner dumps across 2,488 corporate domains including AWS, Samsung, and Cisco — proves the agent dependency graph is a real attack surface.
Measurement Standardizes — Hugging Face and Meta shipped GAIA2 and ARE with 800 scenarios across 10 universes, OpenEnv rallied a PyTorch Foundation-led coalition behind a shared environment layer, and frameworks converged on a single agent.run() interface. Evaluation is finally an engineering discipline.
Self-Improving Loops — Grok 4.6 became the first model trained on internal model-development tasks, and multi-LLM self-improvement loops are being pitched as the future of automation — with sharp warnings that these loops live or die on the verifier you choose.

Tags

AWSAbacus AIAlibabaAmazonAnthropicArize+101 more

Jun 29, 2026

Building the Agentic Infrastructure Stack

Description

Learned Orchestration Rises We are pivoting away from brittle, hard-coded if/else logic toward 'harness engineering,' where models like Sakana AI’s Fugu are trained specifically for delegation, verification, and task synthesis.
Infrastructure Meets Reality While OpenAI builds 'Jalapeno' silicon for o1-level reasoning, enterprise benchmarks reveal an '11% reality wall' in SRE tasks that only robust protocols and 'Code-as-Action' frameworks can breach.
Unified Agentic Protocols The arrival of OpenAI’s Operator and Anthropic’s Model Context Protocol (MCP) marks the decisive shift from conversational chat to deterministic, autonomous execution across the web.
Local Intelligence Scaling Developers are increasingly distilling frontier capabilities into local weights, utilizing tools like Gemma and GLM 5.2 to create specialized, cost-effective reasoning loops at the edge.

Tags

AlibabaAmazonAnthropicAppleBroadcomCoinbase+48 more

May 14, 2026

The Era of Agentic Infrastructure

Description

The Runtime Shift Practitioners are moving away from 'vibe-coded' prompts toward deterministic harnesses and managed SDKs that treat agents as infrastructure rather than simple API calls.
Code-as-Action Gains Hugging Face’s smolagents launch demonstrates that letting agents write Python directly can outperform bloated JSON-based orchestration frameworks by increasing reasoning density.
The Browser Battlefield With tools like OpenAI's Operator and Anthropic's Computer Use, the browser has become the primary execution interface, raising the stakes for session security and DOM reliability.
Sovereign Execution The integration of agents into trackers like Linear and payment rails via Stripe signals the transition of agents from chat assistants to autonomous control planes.

Tags

AnthropicClickHouseDeepSeekHugging FaceLinearMastercard+35 more