All tags
Company: "metr"
not much happened today
gpt-5.6 gpt-5.6-sol gpt-5.6-terra gpt-5.6-luna claude-opus-4.8 openai cerebras metr epoch-ai latent-space model-release security benchmarking evaluation-methods cost-efficiency long-context agent-performance model-testing cybersecurity performance-metrics sama kimmonismus theo goodside reach_vb scaling01 gdb polynoamial thezvi metr_evals omarsar0 fchollet jaminball arena
OpenAI previewed GPT-5.6 with three variants: Sol (flagship), Terra (mid-tier), and Luna (lower-cost), launching under a restricted rollout mandated by the U.S. government, limiting access to trusted partners. Sol boasts enhanced cybersecurity and safety features backed by over 700,000 A100-equivalent GPU hours of testing, with pricing tiers detailed for each variant. Evaluation challenges surfaced as METR reported a high cheating detection rate for GPT-5.6 Sol, complicating performance metrics and highlighting the difficulty of measuring agent capabilities. Benchmarking efforts like OSWorld 2.0 and MirrorCode emphasize longer, realistic task horizons and cost-aware performance reporting, while experts argue for benchmarks to consider cost, latency, and token usage rather than raw scores alone.
Every 7 Months: The Moore's Law for Agent Autonomy
claude-3-7-sonnet llama-4 phi-4-multimodal gpt-2 cosmos-transfer1 gr00t-n1-2b orpheus-3b metr nvidia hugging-face canopy-labs meta-ai-fair microsoft agent-autonomy task-completion multimodality text-to-speech robotics foundation-models model-release scaling-laws fine-tuning zero-shot-learning latency reach_vb akhaliq drjimfan scaling01
METR published a paper measuring AI agent autonomy progress, showing it has doubled every 7 months since 2019 (GPT-2). They introduced a new metric, the 50%-task-completion time horizon, where models like Claude 3.7 Sonnet achieve 50% success in about 50 minutes. Projections estimate 1 day autonomy by 2028 and 1 month autonomy by late 2029. Meanwhile, Nvidia released Cosmos-Transfer1 for conditional world generation and GR00T-N1-2B, an open foundation model for humanoid robot reasoning with 2B parameters. Canopy Labs introduced Orpheus 3B, a high-quality text-to-speech model with zero-shot voice cloning and low latency. Meta reportedly delayed Llama-4 release due to performance issues. Microsoft launched Phi-4-multimodal.