Tag
Promptfoo
3 issues found
Nov 10, 2026
Benchmark Leads, Orchestration Bills
Description
- Cheaper Claims: Fable 5.2 ships at $8/$40 per million tokens, with Anthropic-claimed benchmark leads and up to 30% faster outputs.
- Split Verdicts: DeepSeek-V4 touts 1M context but admits scores are "competitive, not SOTA"; independent analysis flags weaker agentic execution.
- Orchestration Costs: A harness reported a 5x cost cut via local workers, while Azure docs warn orchestration multiplies invocations.
Tags
AWSAdyenAmazonAnthropicArizeAzure+50 more
313 time saved519 sources29 min read
Sep 23, 2026
Opus 5.5 Cuts Prices, Costs Bite
Description
- Price War Opens Anthropic's Opus 5.5 claims a 20% price cut and wins on all nine benchmarks shown, shifting competition to cost-per-task.
- Coordinator Pattern Cursor, OpenAI and Claude all shipped coordinator-plus-workers layouts in one week, as 847-run tests show context degrading in the middle.
- Europe's Bet Mistral's €3B Series D, reportedly Europe's largest equity raise, funds compute and data centers; MCP builders cite distribution, not protocol, as the blocker.
Tags
ASMLAWSAlibabaAnthropicArtificial AnalysisBNP Paribas CIB+49 more
328 time saved1877 sources34 min read
Mar 11, 2026
The Hardening Agentic Stack
Description
- Sovereign Infrastructure Risks Anthropic’s federal lawsuit over 'supply chain risk' signals a shift where model selection is now tied to geopolitical compliance and sovereign security.
- The Memory Wall Benchmarks like Mem2ActBench expose the 'Turn 6' problem—agents struggle to ground tool parameters in long-context interactions, moving the focus from retrieval to state management.
- Code-as-Action Evolution The industry is abandoning brittle JSON outputs for 'code-as-action' frameworks like smolagents and Agents.js, turning LLMs into verifiable logic engines.
- Production Hardening With OpenAI acquiring Promptfoo and builders deploying 'Ship Safe' protocols, the era of 'vibe coding' is ending in favor of cost-optimized, secure agentic architectures.
Tags
AMDAmazonAnthropicAppleByteDanceCrewAI+41 more
391 time saved2559 sources21 min read