All tags
Topic: "performance-benchmarks"
not much happened today
deepseek-r1 deepseek-v3 coder-v2 prover deepseek hugging-face dell openai instruction-tuning performance-benchmarks model-deployment training-costs hardware-scalability ai-safety risk-mitigation ethical-ai open-source gpu-utilization yann-lecun yoshua-bengio francois-chollet giffman
DeepSeek-R1 and DeepSeek-V3 models have made significant advancements, trained on an instruction-tuning dataset of 1.5M samples with 600,000 reasoning and 200,000 non-reasoning SFT data. The models demonstrate strong performance benchmarks and are deployed on-premise via collaborations with Dell and Hugging Face. Training costs are estimated around $5.5M to $6M, with efficient hardware utilization on 8xH100 servers. The International AI Safety Report highlights risks such as malicious use, malfunctions, and systemic risks including AI-driven cyberattacks. Industry leaders like Yann LeCun and Yoshua Bengio provide insights on market reactions, AI safety, and ethical considerations, with emphasis on AI's role in creativity and economic incentives.
OpenAI launches Operator, its first Agent
operator deepseek-r1 videollama-3 llama-4 o1 claude openai anthropic deepseek-ai google-deepmind perplexity-ai computer-using-agent reasoning multimodality performance-benchmarks open-source ai-safety benchmarking video-generation model-evaluation sam-altman swyx
OpenAI launched Operator, a premium computer-using agent for web tasks like booking and ordering, available now for Pro users in the US with an API promised. It features long horizon remote VMs up to 20 minutes and video export, showing state-of-the-art agent performance but not yet human-level. Anthropic had launched a similar agent 3 months earlier as an open source demo. DeepSeek AI unveiled DeepSeek R1, an open-source reasoning model excelling on the Humanity's Last Exam dataset, outperforming models like LLaMA 4 and OpenAI's o1. Google DeepMind open-sourced VideoLLaMA 3, a multimodal foundation model for image and video understanding. Perplexity AI released Perplexity Assistant for Android with reasoning and search capabilities. The Humanity's Last Exam dataset contains 3,000 questions testing AI reasoning, with current models scoring below 10% accuracy, indicating room for improvement. OpenAI's Computer-Using Agent (CUA) shows improved performance on OSWorld and WebArena benchmarks but still lags behind humans. Anthropic AI introduced Citations for safer AI responses. Sam Altman and Swyx commented on Operator's launch and capabilities.
Genesis: Generative Physics Engine for Robotics (o1-mini version)
o1 o1-preview gpt-4o claude-3.5-sonnet gemini-2.0-pro llama-3-3b llama-3-70b openai google-deepmind meta-ai-fair hugging-face function-calling structured-outputs vision performance-benchmarks sdk webrtc reasoning math code-generation transformer-architecture model-training humanoid-robots search model-efficiency dataset-sharing aidan_mclau sundarpichai adcock_brett
OpenAI launched the o1 model API featuring function calling, structured outputs, vision support, and developer messages, achieving 60% fewer reasoning tokens than its preview. The model excels in math and code with a 0.76 LiveBench Coding score, outperforming Sonnet 3.5. Beta SDKs for Go and Java and WebRTC support with 60% lower prices were also released. Google Gemini 2.0 Pro (Gemini Exp 1206) deployment accelerated, showing improved coding, math, and reasoning performance. Meta AI FAIR introduced research on training transformers directly on raw bytes using dynamic entropy-based patching. Commercial humanoid robots were successfully deployed by an industry player. Hugging Face researchers demonstrated that their 3B Llama model can outperform the 70B Llama model on MATH-500 accuracy using search techniques, highlighting efficiency gains with smaller models. Concerns about reproducibility and domain-specific limitations were noted.
Genesis: Generative Physics Engine for Robotics (o1-2024-12-17)
o1 gemini-2.0-pro openai google carnegie-mellon-university universal-physics-engine robotics-simulation physics-simulation photo-realistic-rendering generative-data simulation-platform open-source function-calling vision performance-benchmarks sdk realtime-api zhou-xian aidan_mclau sundar-pichai
Genesis is a newly announced universal physics engine developed by a large-scale collaboration led by CMU PhD student Zhou Xian. It integrates multiple state-of-the-art physics solvers to simulate diverse materials and physical phenomena, targeting robotics applications with features like lightweight, ultra-fast simulation, photo-realistic rendering, and generative data capabilities. The engine is open source and designed for robotics simulation beyond just video generation. Additionally, OpenAI released the o1 model to API with advanced features like function calling and vision support, showing strong math and coding performance. Google teased updates on Gemini 2.0 Pro, accelerating deployment for advanced users.
not much happened today
llama-2-70b llama-2-7b mistral-7b qwen-1.5 llava microsoft mistral-ai ollama fine-tuning synthetic-data retrieval-augmented-generation embeddings hardware-optimization performance-benchmarks model-memory multimodality
The Reddit community /r/LocalLlama discusses fine-tuning and training LLMs, including tutorials and questions on training models with specific data like dictionaries and synthetic datasets with 25B+ tokens. Users explore retrieval-augmented generation (RAG) challenges with models like mistral-7b and embedding generation for EEG brain activity. Discussions include hardware optimization for running llama-2-70b locally under budget constraints, and performance benchmarks for qwen-1.5 models. There is interest in extending LLM capabilities, such as converting llama-2-7b into a vision-capable model like llava and improving model memory for longer context retention.