All tags
Person: "alec-radford"
ModernBert: small new Retriever/Classifier workhorse, 8k context, 2T tokens,
modernbert gemini-2.0-flash-thinking o1 llama answerdotai lightonio hugging-face google-deepmind openai meta-ai-fair figure encoder-only-models long-context alternating-attention natural-language-understanding reasoning robotics-simulation physics-engine humanoid-robots model-performance model-releases jeremyphoward alec-radford philschmid drjimfan bindureddy
Answer.ai/LightOn released ModernBERT, an updated encoder-only model with 8k token context, trained on 2 trillion tokens including code, with 139M/395M parameters and state-of-the-art performance on retrieval, NLU, and code tasks. It features Alternating Attention layers mixing global and local attention. Gemini 2.0 Flash Thinking debuted as #1 in Chatbot Arena, and the O1 model scored top in reasoning benchmarks. Llama downloads surpassed 650 million, doubling in 3 months. OpenAI launched desktop app integrations with voice capabilities. Figure delivered its first humanoid robots commercially. Advances in robotics simulation and a new physics engine Genesis claiming 430,000x faster than real-time were highlighted.
Evals-based AI Engineering
jamba bamboo qwen-1.5-moe grok-1.5 llama2-7b openai mistral-ai x-ai llamaindex evaluation fine-tuning prompt-engineering voice-cloning quantization model-optimization code-generation context-windows hamel-husain alec-radford
Hamel Husain emphasizes the importance of comprehensive evals in AI product development, highlighting evaluation, debugging, and behavior change as key iterative steps. OpenAI released a voice engine demo showcasing advanced voice cloning from small samples, raising safety concerns. Reddit discussions introduced new models like Jamba (hybrid Transformer-SSM with MoE), Bamboo (7B LLM with high sparsity based on Mistral), Qwen1.5-MoE (efficient parameter activation), and Grok 1.5 (128k context length, surpassing GPT-4 in code generation). Advances in quantization include 1-bit Llama2-7B models outperforming full precision and the QLLM quantization toolbox supporting GPTQ/AWQ/HQQ methods.