All tags
Topic: "data-contamination"
OpenAI reports Navier-Stokes singularity find, a contender for second ever Millenium Prize awarded, overshadowing Cognition's $48B Series E, Mistral's $24B Series D, Meta's Muse agent, and GPT Image 2.5
gpt-6-astra openai meta-ai-fair test-time-compute formal-verification parallel-computing scientific-governance personal-ai-agent linux-vm service-integration data-contamination open-science-norms sama sebastienbubeck terence_tao sam_altman
OpenAI announced a proposed Navier–Stokes proof by an internal model "significantly more capable than GPT-6 Astra" using 10,000 agents over 88 hours plus 17 hours of formal verification. The effort highlights the emergence of massive test-time compute scaling as a new axis beyond pretraining, with estimated costs of $10M–$40M and 130B output tokens. Controversy arose over priority, data contamination, and scientific norms, with key figures like Sam Altman, Sébastien Bubeck, and Terence Tao weighing in on governance and open science risks. Meanwhile, Meta launched Muse, a consumer personal AI agent featuring persistent isolated Linux VMs, browser integration, and connectors to various apps including Meta-native services like Instagram and Messenger, emphasizing security and broad service integration.
Clémentine Fourrier on LLM evals
claude-3-opus huggingface meta-ai-fair llm-evaluation automated-benchmarking human-evaluation model-bias data-contamination elo-ranking systematic-annotations preference-learning evaluation-metrics prompt-sensitivity clem_fourrier
Clémentine Fourrier from Huggingface presented at ICLR about GAIA with Meta and shared insights on LLM evaluation methods. The blog outlines three main evaluation approaches: Automated Benchmarking using sample inputs/outputs and metrics, Human Judges involving grading and ranking with methods like Vibe-checks, Arena, and systematic annotations, and Models as Judges using generalist or specialist models with noted biases. Challenges include data contamination, subjectivity, and bias in scoring. These evaluations help prevent regressions, rank models, and track progress in the field.
Evals: The Next Generation
gpt-4 gpt-5 gpt-3.5 phi-3 mistral-7b llama-3 scale-ai mistral-ai reka-ai openai moderna sanctuary-ai microsoft mit meta-ai-fair benchmarking data-contamination multimodality fine-tuning ai-regulation ai-safety ai-weapons neural-networks model-architecture model-training model-performance robotics activation-functions long-context sam-altman jim-fan
Scale AI highlighted issues with data contamination in benchmarks like MMLU and GSM8K, proposing a new benchmark where Mistral overfits and Phi-3 performs well. Reka released the VibeEval benchmark for multimodal models addressing multiple choice benchmark limitations. Sam Altman of OpenAI discussed GPT-4 as "dumb" and hinted at GPT-5 with AI agents as a major breakthrough. Researchers jailbroke GPT-3.5 via fine-tuning. Global calls emerged to ban AI-powered weapons, with US officials urging human control over nuclear arms. Ukraine launched an AI consular avatar, while Moderna partnered with OpenAI for medical AI advancements. Sanctuary AI and Microsoft collaborate on AI for general-purpose robots. MIT introduced Kolmogorov-Arnold networks with improved neural network efficiency. Meta AI is training Llama 3 models with over 400 billion parameters, featuring multimodality and longer context.
12/30/2023: Mega List of all LLMs
deita-v1.0 mixtral amazon-titan-text-express amazon-titan-text-lite nous-research hugging-face amazon mistral-ai local-attention computational-complexity benchmarking model-merging graded-modal-types function-calling data-contamination training-methods stella-biderman euclaise joey00072
Stella Biderman's tracking list of LLMs is highlighted, with resources shared for browsing. The Nous Research AI Discord discussed the Local Attention Flax module focusing on computational complexity, debating linear vs quadratic complexity and proposing chunking as a solution. Benchmark logs for various LLMs including Deita v1.0 with its SFT+DPO training method were shared. Discussions covered model merging, graded modal types, function calling in AI models, and data contamination issues in Mixtral. Community insights were sought on Amazon Titan Text Express and Amazon Titan Text Lite LLMs, including a unique training strategy involving bad datasets. Several GitHub repositories and projects like DRUGS, MathPile, CL-FoMo, and SplaTAM were referenced for performance and data quality evaluations.