All tags
Topic: "real-world-evaluation"
not much happened today
gpt-5.6-sol grok-4.5 terra-max fable-5-max opus-4.8 100b-reasoning-model prime-intellect vllm langchain threepointone factory cognition arena artificial-analysis parlance-labs agentic-reinforcement-learning rollout-traces message-dags long-horizon-reinforcement-learning multimodality harness-design cost-per-task coding-agents benchmarks model-efficiency real-world-evaluation task-specialization johannes_hage willccbb mikasenghaas xeophon omarsar0 skirano imjaredz
Prime Intellect released verifiers v1, a redesigned environment stack for agentic reinforcement learning and evaluations, improving efficiency by storing rollout traces as message DAGs to reduce complexity from O(nĀ²) to O(n). This enables practical long-horizon multimodal rollouts, demonstrated with a 100B reasoning model running 40-turn SWE agent tasks on 6 H200 nodes in under 2 days. The ecosystem support includes vLLM integration to avoid tokenization drift. Discussions highlight that harnesses are becoming critical as the product surface for coding agents, with task-specialized harnesses favored over generic wrappers. Benchmarks are shifting focus from token price to cost per task, with models like Terra Max, Fable 5 Max, and Opus 4.8 compared on efficiency and cost. Real-world agent benchmarks show GPT-5.6 Sol ranking #2 and Grok-4.5 jumping to #13 on Arena's leaderboard, emphasizing cost per task as a key metric for long-horizon knowledge work.