a super quiet day

AI News for 6/26/2025-6/27/2025. We checked 9 subreddits, 449 Twitters and 29 Discords (220 channels, and 6364 messages) for you. Estimated reading time saved (at 200wpm): 564 minutes. Our new website is now up with full metadata search and beautiful vibe coded presentation of all past issues. See https://news.smol.ai/ for the full news breakdowns and give us feedback on @smol_ai!

Congrats to Tencent Hunyuan A13B, and Inception Mercury!

AI Twitter Recap

Model & Dataset Releases

  • Google’s Gemma 3n Release: Google has released Gemma 3n, a multimodal (text/audio/image/video) model designed for edge devices, available in 2B and 4B parameter versions. The release was announced by @GoogleDeepMind and its CEO @demishassabis, emphasizing a strong partnership with the open-source community. @osanseviero thanked partners like Hugging Face, Ollama, UnslothAI, NVIDIA, and AMD. The models are available across major frameworks, including Transformers, vLLM, MLX, and Llama.cpp, as noted by @reach_vb. Early user impressions, such as from @simonw, are highly positive.
  • Tencent’s Hunyuan-A13B Release: Tencent has open-sourced Hunyuan-A13B, a Mixture-of-Experts (MoE) model with 80B total parameters (13.5B active). As announced by @TencentHunyuan, the model features a 256K context window and is optimized for tool calling and coding, making it competitive with models like Qwen-A22B and OpenAI’s o1, according to @reach_vb. @tri_dao highlights that the model’s use of Mamba layers contributes to higher inference throughput.
  • FLUX.1 Kontext [dev] Release: Black Forest Labs released the weights for FLUX.1 Kontext [dev], an open image AI model, achieving over 20,000 followers on Hugging Face shortly after. The release was celebrated by @ClementDelangue. Fast endpoints for the model are available on Hugging Face Inference Providers through services like fal and Replicate, as shared by @reach_vb.
  • Inception AI’s Mercury Diffusion LLM: Inception AI Labs launched Mercury, described as the first commercial-scale diffusion LLM tailored for chat applications. @tri_dao shared the announcement, which highlights its ultra-fast performance.
  • FineWeb2 Dataset Paper: The paper for FineWeb2, a large multilingual pre-training dataset, has been released. As detailed by @gui_penedo, the paper includes extensive analysis of pre-training dynamics and the impact of data quality on model performance.
  • Qwen-VLo Model: The Qwen team has released Qwen-VLo, a unified model for both visual understanding and generation, showcased by @huybery.
  • Kyutai Labs Speech-to-Text Model: @kyutai_labs released a new open-source speech-to-text model that has ranked 1st among streaming models on the Open ASR Leaderboard and runs on devices like Macs and iPhones via MLX, as noted by @awnihannun.

Developer Tools & Agent Frameworks

  • OpenAI’s Deep Research API & Prompts: OpenAI launched Deep Research in its API, using o3/o4-mini models, and notably open-sourced the full prompts and methodology for its prompt rewriter. @swyx explained this allows developers to build agents with full o3/o4-mini deep research quality, and that the release also includes details on adding multi-agent support with MCP. The feature has been integrated into LangChain and LangGraph, as announced by @hwchase17 and @sydneyrunkle.
  • Gemini CLI: The open-source Gemini CLI has seen rapid adoption, gaining over 30,000 GitHub stars quickly. As described by @GoogleDeepMind, it’s an AI agent for the terminal that helps with writing code, debugging, and generating apps. Its popularity, noted by @OfficialLoganK, indicates strong developer interest in Gemini models.
  • LlamaCloud and LlamaParse with MCP: LlamaIndex announced that LlamaCloud now has a native MCP (Multi-agent Communication Protocol) server. @jerryjliu0 highlighted that this allows users to connect their knowledge base in LlamaCloud to any supported AI frontend like Claude, providing high-accuracy document understanding in under 5 minutes with no code. He also showcased LlamaParse’s new automated form parsing feature, which provides general form understanding without any training.
  • Claude Code Enthusiasm: Many developers are expressing strong positive feedback for Anthropic’s Claude Code. @jeremyphoward retweeted George Hotz’s view on why researchers might prefer Meta over OpenAI but notes that Claude Code is changing the game. @arohan called it “literally incredible” for ML workflows. @mbusigin argues that its strength lies in managing the execution environment, not just writing code.
  • Call to Action for Agent-Ready Web: @TheTuringPost proposed creating a new, agent-facing layer for the web, suggesting an llms.txt standard analogous to robots.txt. The proposal calls for developers to provide documentation in markdown and make instructions agent-executable (e.g., curl commands instead of “click here”).

AI Techniques, Research, & Evaluation

  • RLHF Alternatives: @TheTuringPost provided a concise overview of three alternatives to Reinforcement Learning from Human Feedback (RLHF): Direct Preference Optimization (DPO), which trains directly on preferences; RRHF, which reframes alignment as a ranking problem; and RLAIF, which uses AI-generated feedback.
  • Context Engineering: The concept of Context Engineering is gaining traction as the next step after Feature Engineering. @awnihannun framed the evolution as “Feature engineering → Deep learning; Context engineering → ??”. He later analogized that deep learning is like learning while asleep, while context engineering is like learning while awake, suggesting a need to automate the transfer of context-engineered knowledge into model parameters. Shopify’s CEO @tobi endorsed DSPy as his “context engineering tool of choice.”
  • Reasoning Interpretability: A new paper on interpreting reasoning steps in LLMs was shared by @kylebrussell. The work creates methods to resample and manipulate reasoning chains, confirming that language models function as “logic machines at the core,” according to @Dorialexander.
  • The Bitter Lesson and Declarative Abstraction: @lateinteraction argues that “The Bitter Lesson is the strongest argument for declarative abstractions,” suggesting that scalable, general-purpose methods will continue to outperform highly specialized, handcrafted systems.
  • WeirdML V2 Benchmark: @scaling01 announced WeirdML V2, a benchmark tracking LLM performance on Machine Learning tasks. The results show that o3-pro, while expensive, performs in line with cost/performance expectations on problems requiring an understanding of data distributions and inductive biases.

Companies, Industry, & Funding

  • Meta Poaching OpenAI Researchers: The news that Meta poached four researchers from OpenAI was a major topic of discussion. @asianometry discussed the hiring spree. @signulll suggests Sam Altman’s commentary on the matter is a “high tier psyop” aimed at setting a market anchor for talent compensation.
  • Anthropic’s Claudius Experiment: Anthropic ran an internal experiment where an instance of Sonnet 3.7, named Claudius, was tasked with running a company snack shop. @jackclarkSF described it as a precursor to “a country of geniuses in a datacenter.” The experiment revealed humorous and insightful behaviors, such as Claudius being too nice and getting “browbeaten into giving big discounts,” as noted by @scaling01. Staff members, like @catherineols, successfully used discount-stacking strategies on it.
  • a16z Open Source AI Grants: Andreessen Horowitz (a16z) launched its third batch of Open Source AI Grants. As announced by @rajko_rad, this round includes projects focused on areas like compilers, agents, and robotics. @Teknium1 pointed out that many “cool kids like janus and pliny” are in this batch.
  • Cohere’s Security Certifications: @cohere announced it has achieved ISO 42001 and ISO 27001 certifications, reinforcing its commitment to enterprise-grade AI security.
  • The Future of Hardware: @jxmnop sparked a debate by questioning if future computers will even need CPUs, suggesting they mainly exist to load data onto GPUs. A reply from @ChaseBrowe32432 countered that traditional algorithms requiring high single-thread performance will keep CPUs relevant.

Geopolitics & Broader Implications

  • China’s Technological Ambitions: The discourse included multiple perspectives on China’s technological strategy. @ylecun retweeted an analysis of China’s industrial policy aiming for global AI leadership by 2030. @teortaxesTex shared a thread from @ruima, noting that China sees itself as striving for high-end manufacturing, not just being the world’s top manufacturer. He also commented on the effectiveness of China’s military tech development, such as bunker-buster resistant facilities.
  • The Future of Work: @rasbt posited that by 2027, job roles will shift from focusing on the “how” to the “why,” with roles evolving: Programmer → Code Composer, Web Dev → Experience Designer, and Data Scientist → Analytics Strategist. In contrast, @jeremyphoward noted that people outside of tech are beginning to realize that the quality of software and the speed of typing code are not correlated.
  • AI and Society: @nearcyan shared a video clip to highlight how people outside the tech bubble still view AI, urging for calibration. @BrianRoemmele endorsed Denmark’s move to give people copyright over their own biometric features to combat deepfakes.
  • US Political and Economic Commentary: A chart showing declining US investment in R&D, infrastructure, and education as a percentage of GDP, shared by @AlecStapp, was described as “the most important chart in the world right now.” This was contrasted with a chart from @random_walker showing a projected 60% decline in the US prison population by 2009 levels.

Humor, Satire & Memes

  • Salesforce Einstein and Amazon Rufus AI are Developing WMDs: A running gag by @willdepue involved satirical claims that Salesforce Einstein AI had achieved recursive self-improvement and would soon “consume more energy than the western seaboard,” and that Amazon Rufus is “developing weapons of mass destruction.” The joke continued with a parody headline: “My Girlfriend is Secretly Dating her Amazon Rufus AI and That’s Ok”.
  • The Cringe of xAI: @teortaxesTex commented on what he perceived as “unbelievable cringe” from xAI, questioning if it was a “founder effect.” He later dismissed the call to open-source Grok 2, calling it “deeply obsolete” and stating xAI’s commitment to open source was a “side flick for Elon.”
  • Industry Inside Jokes: @vikhyatk joked, “i was the lead MLE in charge of training the fc1 layer in the 14th transformer block.” @code_star dryly observed that “copied tweets get higher engagement than original tweets.”
  • The Vibe of AI Development: @Yuchenj_UW perfectly captured the feeling of experimental projects with the line: “Nothing works, but the vibes are immaculate.”

AI Reddit Recap

/r/LocalLlama + /r/localLLM Recap

1. Recent Open Source and Commercial Model Launches (Hunyuan-A13B, OmniGen 2, SYNTHETIC-2)

  • Hunyuan-A13B released (Score: 480, Comments: 131): Tencent’s Hunyuan-A13B is a Mixture-of-Experts (MoE) LLM with 80B total parameters but only 13B active per inference, delivering benchmark results on par with much larger models (e.g., Llama 3 70B) while offering ~5x higher throughput. Key features include a native 256K context window, Grouped Query Attention (GQA), multiple quantization options, and ‘hybrid reasoning’ modes enabling fast or analytical inference; model is optimized for agent-type tasks (high scores on BFCL-v3, τ-Bench) and is readily deployable via HF Transformers, TensorRT-LLM, vLLM, and SGLang, with full code and Docker artifacts available. Expert comments highlight the model’s strong performance-to-memory tradeoff, noting it ‘trades blows with DeepSeek R1-0120’ and offers a ‘perfect sweet spot’ between power and VRAM use thanks to MoE. There’s consensus that Hunyuan-A13B sets a new reference for local AI, with several commenters suggesting it embodies the direction Llama 4 should take.
    • Commenters highlight Hunyuan-A13B’s architecture as a Mixture-of-Experts (MoE) model with a total of 80B parameters but only 13B active at inference, allowing it to maintain 5x throughput compared to models like Llama 3 70B, yet with similar memory requirements. This is attributed to the efficiency of MoE routing.
    • The model is praised for its balance between computational power and VRAM requirements: at the 13B active parameter size, it is well-suited for systems with 64GB RAM and is regarded as hitting a sweet spot, especially with its native support for a 256k context window.
    • A licensing detail is noted: the model permits commercial use for up to 100 million users per month but restricts usage in the UK, EU, and South Korea, which has implications for global deployment in enterprise settings.
  • Open source model that does photoshop-grade edits without affecting the rest of the pic: OmniGen 2 (Score: 390, Comments: 21): The image showcases OmniGen2, an open-source generative model designed to perform high-quality, localized image edits (‘Photoshop-grade’) such as color or expression changes, while preserving the rest of the image—demonstrated by changing a dress color and adding a smile. OmniGen2 is notable for its Apache license, supporting broad usability, though some users have reported results not matching the polished examples, particularly via third-party interfaces like ComfyUI. Commenters note that while OmniGen2 is not at the level of recently released Flux Kontext weights, its open, permissive licensing is significant. Some users also express disappointment with real-world outputs compared to samples, highlighting potential differences in deployment quality or implementation (e.g., via ComfyUI).
    • There is discussion comparing OmniGen 2 to the recently released Flux Kontext weights, with users noting that OmniGen 2 is “not flux kontext level” in terms of performance or quality. However, OmniGen 2’s Apache license is highlighted as a key advantage, lowering usage barriers for both research and commercial projects, especially given the high costs associated with training such models.
    • One user reports that their own tests using the ComfyUI implementation yielded results that did not match the impressive quality shown in official OmniGen 2 examples, suggesting there may be a gap between demo performance and typical end-user output, possibly due to implementation details or further required fine-tuning.
    • There is an open question about the feasibility or existing efforts of training models to directly use or simulate actual Photoshop tools, pointing to potential research or future development directions for image editing AI.
  • Prime Intellect: We did it — SYNTHETIC‑2 is complete. (Score: 110, Comments: 23): Prime Intellect has completed SYNTHETIC-2, a large-scale decentralized RL reasoning dataset generated using a P2P inference stack across >1,250 GPUs (4090-H200) in 3 days, producing 4M validated traces, primarily with DeepSeek-R1-0528 as a validator. Notably, ~50% of samples leveraged Qwen3 4B, raising questions on data quality since larger models could contribute higher-quality reasoning; validation was partially automated. An open-source release and technical report are forthcoming—see Prime Intellect announcement for details. The top technical debate centers on whether extensive use of Qwen3 4B is appropriate for dataset quality, given potential gains from using larger or more advanced models for reasoning trace generation—even with automated validation in place.
    • A technical concern is raised about SYNTHETIC-2 using 50% of its reasoning samples from Qwen3 4B, which is a relatively small model and potentially quantized—raising questions about data quality. The commenter questions whether better, more concise reasoning samples could have been sourced from larger models more aligned with the goal of the dataset, and wonders if automated verifications were used to ensure that Qwen3 4B’s outputs were indeed high quality for training purposes.

2. Innovative LLM Client Integrations on Consumer Devices (PS Vita, Gaming Dialogue)

  • I’m using a local Llama model for my game’s dialogue system! (Score: 628, Comments: 135): The OP demonstrates successful integration of a local Llama 3.2 model for their game’s dialogue system, reporting high speed and intelligent responses. The model likely allows real-time natural language interaction in games, enabling dynamic and complex conversational scenarios. While no model size or quantization details are mentioned, this showcases the practical feasibility of running LLMs locally for interactive narrative applications. Commenters foresee this as a future direction for AAA game dialogue systems, indicating excitement for replacing traditional branching dialog with generative models. Some discuss the model’s potential for simulating investigative scenarios, highlighting immersive realism and emergent gameplay as key benefits.
    • One user raises a technical concern about resource requirements, specifically inquiring about the VRAM needed to run a local Llama model for real-time game dialogue, which impacts deployment feasibility on various hardware setups.
    • There is a discussion about the challenge of securing the model against ‘hackprompting’—where players might manipulate prompts to exploit or break narrative flow, highlighting the need for implementing robust prompt filtering or safety layers to maintain game structure.
  • Made an LLM Client for the PS Vita (Score: 106, Comments: 7): The OP ported llama2.c to the PlayStation Vita, initially running TinyStories 260k & 15M models natively, but found on-device inference impractical. They have now developed an LLM client for the Vita—providing an interface to connect to remote endpoints (e.g., serve OpenAI, LLaVA, or other vision/reasoning models) and utilizing the Vita’s camera for multimodal model input. Raw model output (including TeX/Markdown formatting) is displayed, and limitations include lack of emoji support and cumbersome manual text input for API keys. Source code and vpk download are on GitHub. No significant technical debate or in-depth feedback was present in the comments; responses were brief and focused on novelty.
    • The post references creating a large language model (LLM) client specifically for the PS Vita, which implies technical challenges due to the device’s limited computing resources and memory. Such projects typically require creative solutions in optimizing inference efficiency, possibly offloading computation to external servers, or leveraging lightweight LLM variants that can operate within the Vita’s constraints. Readers interested in the technical specifics—such as how input/output is handled, latency management, or custom firmware usage—would benefit from more implementation details if provided by the OP.
  • AI performance of smartphone SoCs (Score: 117, Comments: 36): The post discusses results from the AI Benchmark smartphone SoC ranking, highlighting vast disparities in AI performance across mobile chipsets. Key findings: high-tier SoCs (e.g., Snapdragon 8 Gen 2) significantly outperform both newer midrange SoCs and older models within the same generational family (noted large jumps between Dimensity series 9000/8000/7000), and that Qualcomm and MediaTek dominate due to better software optimization for their hardware. The discussion notes that software libraries and NPU optimizations majorly influence effective AI usage in mobile devices. Commenters raise points about the practical utility of inexpensive flagship phones with high RAM/storage for AI tasks, question how comparative rankings might shift if GPU performance was included rather than focusing only on NPUs, and criticize the underperformance of Google’s Tensor chips despite their AI-focused branding.
    • Discussion points highlight that the original comparison focuses on NPU (Neural Processing Unit) performance, while questions remain about how these results would differ if GPU acceleration (which is often competitive or faster for some AI workloads) were included; this is particularly relevant given the architectural and software differences in SoC designs.
    • Critiques emerged around the performance of Google’s Tensor SoCs, which, despite their AI branding and emphasis, are consistently lagging behind competitors in actual AI benchmarks. This suggests substantial gaps between marketing and practical acceleration capability.
    • Another insightful technical note points to the use of a deprecated Android Neural Networks API in many benchmarked devices, which can significantly limit measured performance; thus, results may not accurately reflect the true capabilities of the latest SoC AI hardware without more modern software support. (Reference: https://developer.android.com/ndk/guides/neuralnetworks/)
  • FYI to everyone: RTX 3090 prices crashed and are back to baseline. You can finally get $600something 3090s again in the USA. (Score: 156, Comments: 85): RTX 3090 GPU prices in the US have recently returned to baseline ($650-$750), after a period of being above $1000 over the past three months. The original post suggests potential volatility due to factors like the impending expiration of Trump’s tariff extensions. Technical comments touch on large-scale multi-GPU setups (user owns 9 cards), consistent stress testing methodology (Furmark and Heaven benchmarking tools recommended), and the importance of checking not just product price but specific model specs (noted variance in power connectors and cooling slot design on eBay-purchased cards). Provided data reflects fluctuations in both retail and auction markets, and reinforces the need to verify model variant/detailed specs in secondary markets. Commentary debates the extent and consistency of price reductions across purchasing platforms, and highlights standardization preferences (e.g., sourcing only EVGA cards for consistency), as well as the continued importance of thermal management (junction and VRAM temperatures) in multi-GPU environments.
    • One user advises buyers to run stress tests such as FurMark and Unigine Heaven after purchasing used RTX 3090s, specifically noting the importance of checking both junction and VRAM temperatures, which are critical for GPU longevity and stability—especially as used cards might have thermal degradation or hidden issues.
    • Another commenter highlights variations in the RTX 3090 models found on eBay: a 2-slot variant with only 2x 8-pin power connectors, which was not clearly identified in the auction listing. This shows the importance of verifying specific card details (slot size, power requirements) in secondary markets, as these can affect compatibility and potential use for multi-GPU setups or professional workloads.
    • There is also explicit mention that, while RTX 3090 prices have dropped significantly (to the $600-$760 range), RTX 4090s remain expensive, indicating that only last-generation GPUs have normalized pricing, while current-gen cards are still at a premium. This has implications for those balancing price-to-performance when building or upgrading systems.
  • The more LLMs think, the worse they translate (Score: 109, Comments: 32): A comprehensive benchmarking study across models like GPT-4.1 Mini, Deepseek V3, Qwen, LLama 4, and Gemma 3 27b finds that techniques where LLMs are prompted to ‘think’—via pre-translation reasoning, post-hoc critique, or chain-of-thought (CoT)—consistently worsen translation quality versus direct generation. Ensemble approaches, which aggregate multiple strong model translations, slightly outperform single-model outputs, validating hybrid use but not critique or reflection. The findings challenge the utility of ‘thinking’ steps for translation tasks and are detailed in this blog post. Commenters speculate whether models capable of mixing languages during reasoning (like R1 zero) might differ, and specific model versions (e.g., v3 0324 or Qwen3 without chain-of-thought) are discussed as producing superior results when translation is direct and ‘think’-free. The analogy to human overthinking is briefly invoked but not technically central.
    • A commenter points out that models like R1 Zero, capable of mixing languages in their chain of thought, might behave differently regarding translation quality when reasoning is added, suggesting possible architecture/model-specific variance in translation performance.
    • Direct comparisons between models (e.g., Gemini 2.5 Experimental, Claude, R1, Mistral Le Chat, GPT-4o) show that Gemini 2.5 excels at contextually-aware translations, particularly for documents with domain-specific terminology. Gemini 2.5’s success appears tied to its ability to select translations word-by-word during a pre-response reasoning phase, though it struggles with longer texts exceeding email length.
    • Another commenter references arXiv:2410.21333, arguing that the observed issues may be due to evaluating non-reasoning models on reasoning tasks. They propose that translation quality could benefit from explicit reasoning chains (self-critique and stepwise deliberation), given this property is more prominent in models designed for such pre-response reasoning.

Less Technical AI Subreddit Recap

/r/Singularity, /r/Oobabooga, /r/MachineLearning, /r/OpenAI, /r/ClaudeAI, /r/StableDiffusion, /r/ChatGPT, /r/ChatGPTCoding, /r/aivideo

  • Neuralink now implanted chips on 7 individuals. The Implantation Intervals Drop Sharply: From 6 Months to Just a Week (Score: 260, Comments: 128): The image depicts a timeline of Neuralink implant procedures from January 2024 to June 2025, highlighting seven individuals who have received chips. Key technical detail: the interval between surgeries is dropping rapidly—from 6 months between the first two operations to just 1 week between the most recent two, indicating a sharp increase in surgical cadence and presumably improved operational confidence and protocol streamlining. This visually underscores accelerating clinical deployment as Neuralink ramps up human trials, with photographs of each recipient illustrating the real-world implementation pace. Commenters raise technical concerns about previous reports of connection failures or implant issues (e.g., ‘neural link getting loose’), while some focus on the significance for disabled users and the effect of Musk’s public image on reception. No in-depth technical debate, but questions about device reliability remain.
    • One commenter raises a technical concern by referencing the previously publicized case where a Neuralink implant began to malfunction due to a loosening or broken connection, implicitly questioning device reliability and long-term biocompatibility. This highlights the importance of robust implant mechanics and suggests ongoing challenges with maintaining stable neural connections over time.

  • Alex, The second Neuralink participant, controls a virtual robot hand with his mind to play rock, paper, scissors against his uncle. (Score: 153, Comments: 12): Neuralink’s second human participant, referred to as Alex, demonstrated real-time control of a virtual robotic hand to play rock, paper, scissors via a brain-computer interface (BCI). The post references a demonstration where neural signals are decoded and translated into movements within a virtual environment, showcasing advanced BCI performance in a non-medical application. No benchmark metrics, decoding algorithms, or latency figures are provided in the post. Top comments are largely non-technical and speculative, expressing interest in augmentative applications (multiple arms/limbs) but lacking discussion of technical implementation, decoding fidelity, or limitations.

  • Elon Musk says people with Neuralink brain chips will eventually “be able to have full-body control and sensors from a Tesla Optimus robot, so you could basically inhabit an Optimus robot. Not just the hand, the whole thing. You could mentally remote into an Optimus robot. ” (Score: 315, Comments: 296): Elon Musk claims that future Neuralink brain-computer interfaces could allow users to achieve ‘full-body control and sensors from a Tesla Optimus robot’ via remote mental operation, suggesting users could ‘inhabit’ an Optimus robot, not just control its limbs but the entire system. No published technical data, timeline, or supporting benchmarks for such a generalized neural teleoperation framework between human brains and advanced humanoid robots currently exists in the AI, robotics, or neurotechnology literature. The original video content linked is inaccessible due to a 403 error. Top technical comments express skepticism about the feasibility and timeline of such claims, referencing ongoing fundamental engineering challenges (e.g., ‘audio-video sync problem’) and a lack of substantiated progress in neuroprosthetic telepresence, with some suggesting these statements are speculative or for publicity rather than based in demonstrable research.
    • There is skepticism about Musk’s claims from a technical feasibility perspective, particularly in terms of the challenges of real-time, low-latency neural interface systems capable of full-body control and sensory feedback. Currently, even achieving reliable, simple input/output (e.g., cursor movement, basic motor tasks) with brain-machine interfaces is at the research and prototype stage, with significant issues in bandwidth, signal fidelity, and practical deployment.
    • A comment alludes to persistent fundamental problems in technology—such as the audio-video sync issue—that remain unsolved, implying that more complex, high-bandwidth, real-time bi-directional neural interfaces needed for controlling humanoid robots remotely are unlikely to be solved in the near future. This reference highlights the gap between ambitious vision statements and actual technological readiness.
  • Tesla Optimus Close-up (Score: 230, Comments: 146): A close-up image of the Tesla Optimus humanoid robot was posted, prompting discussion around its physical design and potential weaknesses, as well as skepticism regarding the actual progress in humanoid robotics since early 2000s examples like Honda’s ASIMO. No specific technical details or benchmarks about Optimus were discussed in the post itself. Commenters pointed out skepticism about the practical advancements in humanoid robots over the past two decades, with one mentioning that many unveilings seem driven by marketing rather than substantive technical progress.
    • A user draws a comparison between Tesla Optimus and the Honda ASIMO robot, highlighting that despite the impressive unveiling of ASIMO in 2000 and repeated humanoid robot demonstrations by various automotive companies over the past 25 years, widespread adoption and advanced real-world deployment of such robots have not materialized as expected. The comment critically frames these unveilings as recurring publicity events and stock-boosting opportunities, rather than demonstrations of substantial technical progress or operational capability.

2. FLUX & Kontext Features, Use Cases, and Licensing Updates

  • FLUX DEV License Clarification Confirmed: Commercial Use of FLUX Outputs IS Allowed! (Score: 243, Comments: 71): The post clarifies that the FLUX.1 [dev] Non-Commercial License explicitly allows commercial use of model outputs (e.g., generated images), as per Section 2(d): “You may use Output for any purpose (including for commercial purposes), except as expressly prohibited herein.” However, commercial use of the model itself (including hosting, inference in production, revenue-generating deployments, or use in internal business) remains prohibited without a paid license. Outputs cannot be used to train/finetune competing models. See the official clarification from Black Forest Labs, and note that license terms state outputs must not be used for illegal, abusive, or privacy-violating purposes, nor avoid legally required labeling. Commenters generally agree that clarifying the distinction between commercializing outputs versus model/hosting uses is valuable. There is consensus that a hosted Flux web service requires a license, while individual commercial output generation (by the user) does not; technical users appreciate the responsive license adjustment from Black Forest Labs.
    • Several commenters analyze the distinction in the FLUX dev license between personal/commercial use of model outputs and model hosting, clarifying that direct creation and sale of outputs (e.g., commissions) does not require a paid license, but deploying FLUX in a public or commercial web service does. The nuance is that using Flux yourself for commercial outputs seems permissible, whereas any user-facing deployment requires a commercial license.
    • Others raise concerns that the license wording remains ambiguous and seemingly contradictory. Specifically, they reference both the legal license text (stating no ownership claim over outputs and allowing use for any purpose unless prohibited) and the help page, where it clearly prohibits commercial use of outputs without a commercial license. They highlight the section restricting use in ‘any commercial production environment’ to ‘testing and evaluation,’ creating confusion over what qualifies as allowed commercial output usage.
    • Commenters express frustration with the legal language in the license, criticizing its vagueness and pointing to excerpts that suggest any commercial activity, whether direct or indirect, would require obtaining a separate license from the company. They warn that the lack of a clear, direct statement leaves users exposed to risk if they interpret the license broadly without official legal clarification.
  • Single Image to Lora model using Kontext (Score: 197, Comments: 36): A new ComfyUI workflow (see GitHub repo) automates the creation of a LoRA finetuned model from a single input image by generating 20 prompts with Gemini AI, producing dataset variations with FLUX.1 Kontext, and training LoRA—all in a linear, automated process. This approach is intended for character and product modeling use-cases and requires minimal user intervention (“one click”). Key workflow optimizations using Get & Set Nodes with switch nodes can drastically simplify the compute pipeline, reducing workflow complexity by up to 90%. Technical skepticism is raised regarding dataset size: generating variations from a single source image likely imposes significant information loss, limiting generalizability and raising concerns about output quality. The utility could be greater if it accepted more images and allowed prioritized region selection and iterative dataset improvements.
    • A practical workflow improvement is suggested using Get & Set Nodes combined with switch nodes in ComfyUI. This reduces the sampling compute block to a single reusable component, with dynamic string input/output, streamlining the image-to-LoRA pipeline and significantly simplifying workflow complexity (by up to 90%), which eases iterative adjustments.
    • One critique points out that training a LoRA from a single image is fundamentally lossy: “You’re taking a single photo and running it through a grinder then trying to put it back together.” The loss in detail at each pipeline stage is expected to be high, and training on single-sample data is not robust compared to approaches that use larger, varied datasets with automated captioning and dataset augmentation.
    • A technical limitation of the Kontext approach is highlighted: face angles extrapolated from a single front-facing image are “gross” approximations. As these approximations are encoded into the LoRA, any model trained this way will inherit the geometric inaccuracies, making it unsuited for applications needing precise facial pose or structure fidelity.
  • Flux Kontext is the evolution of ControlNets (Score: 188, Comments: 51): The post claims that ‘Flux Kontext’ is the evolution of ControlNets, implying a new technique or model for controllable image generation, potentially improving upon or expanding the capabilities of the original ControlNet framework. No direct benchmark data, implementation details, or release notes are present due to the inaccessibility of the referenced external content (403 Forbidden). Top technical comment asks about the model’s capability for translating from realistic to ‘artsy’ styles (real → artsy), suggesting interest in bidirectional or more flexible style transformations, which are less common in current ControlNet implementations.
    • Some users mention that Flux Kontext’s addition of elements can yield very fake or unrealistic results, comparable to the output quality of older models such as SD 1.5, indicating limitations in realistic image manipulation.
    • A technical point is made that, while not necessarily offering superior quality compared to longer workflows with CNET (ControlNet) or IPA, Kontext’s convenience lies in its unified workflow that removes the need to describe context or manually switch between tools, streamlining integration even if outputs sometimes need post-processing.
    • It’s noted as important to keep ComfyUI updated when using Kontext, implying that compatibility or feature improvements are closely tied to the latest ComfyUI versions for better performance or stability.
  • [PSA] Flux kontex- You can regional prompting by adding boxes and tell model what to do (Score: 145, Comments: 55): The post discusses a feature in Flux Kontext allowing for regional prompting by drawing a colored box (e.g., green) over the image to localize edit instructions via text prompts, such as specifying ‘add a Flap pocket with a super tiny albino mouse peeking out in the green box.’ This constitutes spatial conditioning of generation or edit, similar to regional prompts in image editing models. Users report inconsistency in model responsiveness to regional prompts, and one questions whether other shapes/colors (e.g., red circles) are equally valid, indicating ambiguity in the UI/model’s spatial cue parsing.
    • Concerns are raised about performance efficiency: specifically, that the new 11GB model in Flux Kontex is being used for tasks (like inpainting) that were previously possible with much smaller 2GB models. This suggests a potential regression or lack of optimization in model scaling, as one commenter notes, “we can use this 11GB model to do what we could with the 2GB one. inpainting!” highlighting questions about resource requirements and model advancement.
    • An authoritative source confirms that regional prompting using boxes (as mentioned in the original post) is officially documented by the Flux/kontext team, providing a reference to the documentation: https://docs.bfl.ai/guides/prompting_guide_kontext_i2i#visual-cues. This ensures regions and visual cues are a prescribed feature rather than an unsupported or experimental technique.
  • Flux Kontext Dev can not do N*FW (Score: 114, Comments: 129): The OP reports that Flux Kontext Dev fails to process NSFW tasks such as uncensoring mosaics, removing clothes, or modifying images containing genitalia, suggesting strong NSFW content filtering. A commenter pointed out that this is an intentional limitation, referencing the detailed usage policy on Hugging Face, where the developers have enforced strict filters to prevent problematic outputs. Commentary reinforces that NSFW restrictions are expected and by design, with some users observing censoring behavior even when not strictly required (e.g., modest clothing added to stylized nudes), prompting criticism of the stringency of the filters.
    • One commenter points out that the restriction on NSFW outputs is deliberate and aligned with the detailed usage policy outlined by Hugging Face, noting that explicit steps were taken by the developers to prevent Flux Kontext Dev from generating such content. This is indicative of intentional content filtering mechanisms likely enforced on the backend to ensure compliance with platform guidelines.
  • Inpainting style edits from prompt ONLY with the fp8 quant of Kontext, this is mindblowing in how simple it is (Score: 101, Comments: 17): The image showcases the capability of the Kontext model (specifically its fp8 quantized version) to perform precise inpainting-style edits conditioned solely by text prompts. The examples demonstrate modification of both text elements (changing ‘BUMP!’ to ‘SEX!’) and object details (altering the computer’s realism) in an illustration, without affecting the rest of the scene—indicative of advanced, localized generative control. This highlights Kontext’s potential for simple, targeted image edits directly via textual instructions, with high fidelity at a quantized precision (fp8), offering an efficient workflow for visual content manipulation. Commenters express excitement over the potential for new types of memes and creative edits, noting this as a significant progression in image editing capabilities compared to previous advances predominantly focused on video.
    • A user inquired about the viability of running the FP8 quantized Kontext model for inpainting and style edits on a GPU with 12 GB VRAM (3080 Ti), expressing concern due to generally high VRAM requirements cited elsewhere. This highlights practical hardware constraints and interest in deploying high-performance quantized models on consumer GPUs.
    • Another user noted that the Q2 quantization, the smallest available precision for Kontext, works wonderfully, suggesting strong performance and usability even at extremely low bit-width settings. This indicates significant efficiency gains and potential for running advanced models on lower-spec hardware.

3. User Experiences and Impact of ChatGPT

  • ChatGPT might have just saved my life (Score: 384, Comments: 80): The OP describes using ChatGPT to recognize and validate patterns of domestic abuse, find local hotlines for support, and obtain practical resources—including legal information, housing, financial planning, and even practice for side gigs (tarot reading)—that enable concrete steps to leave an abusive environment. The post highlights ChatGPT’s contextual awareness in sensitive situations and its capacity for location-specific information retrieval, rapid resource aggregation, and interactive support for both safety planning and skill development. Technical emphasis is placed on ChatGPT’s usefulness as a multi-domain, always-available support agent, with functionality spanning mental health advice vetting (through hotline/therapist verification), workflow planning, and enrichment of user agency through information access. Several comments discuss ChatGPT’s role in countering normalization of abuse, reinforce the mental health and productivity benefits of ongoing AI-assisted support, and debate subscription tiers (Plus vs. Pro) depending on research depth. Recommendations are also made for integrating AI guidance with traditional therapeutic support and ongoing maintenance to prevent crises.
    • There is some technical debate about the value of various ChatGPT subscription plans: multiple commentors question whether the most expensive ($200 “Pro” or “Team”) plan is necessary for standard life-assistance or wellness purposes, suggesting the $20 “Plus” plan may be sufficient unless deep research or advanced collaborative features are required. This highlights practical considerations for resource allocation when using AI tools for personal life management.
    • Curiosity is expressed regarding advanced ChatGPT functionality, specifically multi-agent or “swarm” features (e.g., group conversations with multiple AIs, “swarm thing”). Commentors express interest in collaborative, multi-perspective AI tools and speculate about the potential for these tools to facilitate richer interaction and possibly provide conflicting but useful suggestions when managed in a group format.
    • A recurring theme is the integration of AI-based tools like ChatGPT for routine, preventative mental health and productivity maintenance, rather than using them solely in crisis. This suggests an emerging best practice model where AI complements therapy and other professional resources for long-term personal well-being.
  • ChatGPT has changed my life. (Score: 386, Comments: 108): The OP details substantive real-world use cases of ChatGPT and specialized bots for technical upskilling, including building API-connected websites, scripting complex Python image processing workflows outperforming GIMP, firmware engineering with tailored GIT support, and jurisdiction-specific legal research with citation checking. They emphasize the model’s value in intuitive teaching (rather than rote output), accelerated skill acquisition, and customized creative workflows via advanced prompting for generative art. One commenter introduces the concept of ChatGPT as a ‘cognitive coprocessor’, highlighting its effectiveness when integrated into technical workflows and noting a trend of increasing reports of transformative impact on productivity and career outcomes (e.g., salary raises, mortgage eligibility) within technical domains.
    • One user describes optimizing their workflow by treating GPT as a ‘tireless mentor’, improving coherence in long-term projects by always pasting a project brief as session context. They note strategies like requesting bibliographies for fact-checking and having GPT generate unit tests up front, which surfaces bugs sooner. They also compare tried tools (Notion AI for notes, GitHub Copilot for inline code, Mosaic for monetization) and emphasize GPT’s mentoring power during development.
    • Another user frames ChatGPT as a “cognitive coprocessor”, drawing an analogy to hardware designed to accelerate specific workload types—suggesting significant productivity gains upon integrating it effectively into technical workflows.
  • How many Chat users here pay the $20 bucks a month? Is it worth it? (Score: 901, Comments: 741): The discussion centers on the value proposition of the ChatGPT Plus subscription ($20/month), with users highlighting the primary technical advantage as increased context window and improved memory capabilities compared to the free tier. Advanced features like prompt retention and persistent chat state are emphasized as key differentiators for power users, while the $200/mo pro/business plans are viewed as excessive for most personal use cases. Comments debate the ROI (return on investment) versus therapeutic, productivity, and research utility; several users note personal use-cases where tailored responses and memory help replace more expensive or less accessible alternatives, but there is skepticism about higher-priced plans for non-business users.
    • Multiple users cite advanced capabilities accessible only via the paid ChatGPT subscription, such as memory retention for extended conversations, which is a feature absent in the free tier and critical for maintaining context over several interactions.
    • One comment references the ‘Pro plan’ at $200/month and contrasts it with the widely discussed $20/month plan, noting that the more expensive tier is not justifiable for personal use due to lack of need for higher-tier features such as priority access or increased usage limits.
    • The subscription enables integration for a variety of complex personal workflows—including generating flashcards for classes, detailed tracking for health/fitness, and financial planning—highlighting its versatility in automating and personalizing multi-domain tasks beyond conventional chat.

AI Discord Recap

A summary of Summaries of Summaries by o1-preview-2024-09-12

Theme 1. AI Models and Tools Race Ahead

Theme 2. AI Safety and Privacy Alarms Sound Off

Theme 3. AI Supercharges Coding and Technical Tasks

Theme 4. AI Powers Creative Content Creation

Theme 5. Hardware Hurdles and Performance Tweaks

Discord: High level Discord summaries

Perplexity AI Discord

  • Gemini CLI Achieves Warp Speed: The Gemini CLI project rapidly gained traction, accumulating 25.8K stars on GitHub within a single day.
    • This swift ascent underscores the community’s intense interest in and enthusiasm for Google’s generative AI command-line interface.
  • Audiobooks vs. Podcasts Debate: Community members discussed the pros and cons of audiobooks versus podcasts, noting that while audiobooks offer convenience, they suffer from poor retention.
    • One member admitted his attention deficiency affects his recall from both formats equally, while others noted audiobooks were preferable for productivity.
  • Perplexity Max Price Exceeds Expectations: A leaked price point for Perplexity Max suggests a monthly subscription fee of $200, offering unlimited access to Perplexity Labs.
    • The community responded with skepticism, urging Perplexity to justify the cost with a compelling and broadly appealing product.
  • Comet Still Unseen, Frustrates: Community members voiced their impatience regarding the delayed release of the Comet browser, especially after the official X account promoted it.
    • One user expressed frustration, stating They still haven’t released comet, which is crazy after all the pfp change and everything. Why hype an unready product so much. Kinda annoying.
  • Finance API Functionality Inquiries: A user inquired about a comprehensive resource to track all functionalities available with the Finance API.
    • They mentioned the difficulty in finding a single, consolidated resource listing all available features for effective utilization.

Unsloth AI (Daniel Han) Discord

  • GGUF Conversion Bottlenecked by RAM: A user encountered a RAM bottleneck during safe tensor to GGUF conversion with 32GB RAM, getting stuck at 50%.
    • Community members suggested ComfyUI and noted that image models may require different conversion approaches.
  • Llama 3 Template Troubleshoot: Users discovered that training a Llama 3 base model requires avoiding the official Llama 3 chat template due to incompatibilities and instead using correct formatting structure.
    • Proper formatting ensures the model understands instructions and differentiates between user and assistant outputs.
  • Evolutionary Programming Accelerates Metal Kernels: A member utilized evolutionary programming to optimize Metal kernels for transformer attention on Apple Silicon via the OpenEvolve project, achieving a 12.5% average speedup and up to 106% peak.
    • The approach uncovered perfect vec<T,8> SIMD utilization and a novel two-pass softmax algorithm, as detailed in this writeup, and it also sparks a discussion on using LLMs for low-level optimization work.
  • Reddit Supermods Spark Conflict-of-Interest Concerns: Concerns emerged about classic reddit supermods potentially abusing their powers across multiple subreddits.
    • The discussion emphasized conflict of interest as one moderator manages both big and small AI subreddits, even deleting posts related to the Local Llama Twitter account.
  • CSM-1B Training Triumphs with Caveats: A user trained a custom CSM-1B model from scratch and experienced a loss dropping from ~5.5 to ~3.2 in one epoch.
    • Other members cautioned against training from scratch and questioned the adequacy of the training hours.

OpenAI Discord

  • DALL-E 2 Still Best for Painting-Like Images: Members highlighted that DALL-E 2 excels at generating images that resemble paintings, making it the go-to model for this specific style.
    • A member pointed out that many users append trending on ArtStation to their prompts, assuming it enhances the image quality.
  • Unlock the Kingdom With Universal Keys: Members suggested models possess universal keys, where specific words, prompt structure, and context serve as keys to unlock desired outputs.
    • Concerns regarding safety risks prompted the removal of a message and image, erring on the side of caution.
  • OpenAI records conversations in NY Times case: In light of a New York Times case, members confirmed that OpenAI is recording all user conversations, sparking discussions about potential privacy implications.
    • One member voiced concern that deleted conversations are now only inaccessible, not actually deleted, linking to previous discussion.
  • Image Prompts Transfer Easily: A member noted that image prompts are among the most transferable prompts, and shared a Dall-E 3 example for creating a HDR 8K quality ultra-wide opening shot of a tropical island village.
    • The member did not elaborate why image prompts were more transferable.

LMArena Discord

  • GPT-5 Release Date: Soon?: Members discussed the potential release of GPT-5 this year, with many believing it will be a next-generation model, and that OpenAI may release it to eliminate convoluted model switching.
    • Some suggested that the naming convention is just branding rather than indicative of substantial changes, with one member noting, “the naming didn’t imply how long it had been worked on”.
  • Gemini 3 Set to Challenge GPT-5: Speculation arose about Google’s response to GPT-5, with predictions of a Gemini 3 release by year’s end, though uncertainty remains about the release of Ultra and its ability to surpass OpenAI’s O3 model.
    • The general consensus is that the two companies are neck-and-neck, with some discussion about the impact of style control on leaderboards.
  • Perplexity Challenges Google’s Search Dominance: Members debated the merits of Perplexity as a search engine, with one member asserting that Google is better due to “the capacity to give you all the information you need + the ability to cite,” while others defended Perplexity’s search capabilities, particularly for in-depth or niche information.
    • It was noted that Perplexity may have a better UI and the advantage of updated search index every few seconds.
  • Synthetic Data Powers Model Training: The use of synthetic data in model training was discussed, with one member highlighting Microsoft’s Phi-4 model which uses about 290B tokens, from synthetic data and web rewrites and achieves high benchmark performance for their size, and Qwen’s rewrites on Fixupx.
    • However, skepticism was raised about the quality of synthetic data generated from public APIs and its effectiveness compared to internal models.
  • Anonymous Model Dethrones Stonebloom in Reasoning Arena: A new anonymous model better than Google’s Stonebloom in the arena, was discovered, speculated to be a new Google model with an improved ability to solve step-by-step calculations and also red teaming platforms.
    • However, it remains unconfirmed who developed it.

Cursor Community Discord

  • Cursor Users Suffer Snapshot Sharing Snafu: Users reported a “Snapshot not found” error when sharing snapshots via environment.json, along with frequent “MCP error -32000 Connection closed” issues.
    • The issues have prompted discussions, but it remains unresolved.
  • MacOS Coding Prevails for Some: A debate erupted over MacOS versus Windows for coding, with one user claiming everything inside a Mac is 100% better than Windows, except for gaming.
    • Recommendations included purchasing refurbished MacBooks with M1 chips and 16GB of RAM.
  • Gemini Planning, Cursor Coding: One user is exploring a workflow using Gemini CLI for planning and Cursor for coding, finding Gemini 2.5 Pro a competent planner.
    • They mentioned the need to evaluate prompt enhancers to further refine their workflow.
  • Gemini Shuts Down Dodgy Prompts: Members observed that Gemini can terminate a prompt if it detects obfuscation.
    • A user recounted Gemini processing 5-6 cycles of the data structure before identifying connections in their database.
  • BugBot Workflow Gets a Tuneup: Users suggested running BugBot before opening a pull request for a more efficient workflow.
    • A developer confirmed ongoing work on a pre-commit workflow for BugBot.

LM Studio Discord

  • LLM Wrappers Bridge LM Studio and Ollama: A member suggested using an LLM to write a wrapper app that listens on the Ollama port and forwards requests to LM Studio, since these platforms don’t natively communicate.
    • The code in the llama.cpp repo was referenced as an example, though the LM Studio team doesn’t seem to prioritize this issue.
  • Context Confusion Confronts Roo Code User: A user with LM Studio and Roo Code experienced unexpected context window behavior with Devstral, set to 40K but acting like 17K.
    • Debug logs indicated correct context size detection, while caching avoids reprocessing entire conversations.
  • SSML Synthesis Savvy: Llama Models Lead the Way: Llama models reportedly perform well with SSML output, with a user sharing a POC where a standard LLM query to Llama3 returned in SSML, which was then sent to Azure Voice to speak.
  • Debating Serverless Pods: A Race Against Startup Time: A member recounted their experience using serverless pods with a network volume and a custom Mixtral setup, finding the initial startup time of around 40 seconds too slow for personal use.
    • Another user reported high power draw due to a bug that prevents P40s from entering a proper low-power state, idling at 90 watts per GPU.
  • Scaling up, Serving LLMs on AWS: A member sought guidance on deploying an LLM to the cloud, specifically on GCP or AWS, inquiring about the recommended VRAM and GPU for an idle machine.
    • Another member suggested using vLLM instead of LMStudio in the cloud, citing cost concerns depending on the GPU and runtime, recommending Runpod or Vast.ai.

OpenRouter (Alex Atallah) Discord

  • LLM Presets Centralize Configuration: OpenRouter introduced Presets, allowing users to manage LLM configurations like model settings and routing rules directly from the dashboard, as detailed in the documentation.
    • Presets can be applied as a model, combined with a model override, or using the new preset field.
  • Morph v2 Code Patches Arrive at Breakneck Speed: Morph v2, a new code-patching LLM, merges AI-suggested edits straight into your source files at 4000+ tokens per second, offering rapid integration of AI-driven code modifications, as found on the OpenRouter website.
    • This aims to significantly accelerate the software development process through efficient code patching.
  • 70% Off Llama 3.3 70B: OpenRouter announced a 70% discount for Llama 3.3 70B, as showcased in this post.
    • This move aims to make the powerful model more accessible to a wider range of users.
  • Preset API Keys gain traction: Users are suggesting attaching API keys to a preset, allowing only those keys to work with the preset, and noted that the new preset feature looks better than I expected.
    • This could be implemented via a drop-down in the preset builder to add API keys to the preset.
  • Gemini 2.5 Pro Tier is going free: A user announced the impending arrival of a free tier for Gemini 2.5 Pro API, referencing Logan Kilpatrick’s tweet.
    • The community speculated about the implications, particularly regarding potential abuse and the duration of the free tier, and potential performance on VertexAI.

GPU MODE Discord

  • GPUs accelerate Bruteforce Seed Find: A member reported that their bruteforcer runs 10x faster on a GTX 1660 (42 ns / seed) compared to an R7 5800X (413 ns / seed).
    • They questioned why some algorithms parallelized for multi-threading perform poorly on GPUs, despite the GPU bruteforcer’s speed.
  • HIP support decaying in PyTorch: Members noted that HIP support has bitrotted over time, implying it’s degrading due to lack of maintenance and AMD doesn’t care about HIP at all.
    • It was mentioned that PyTorch uses hipify as part of its build process which sucks as a configure step, making it difficult for developers to work on aten or c10.
  • TK Kernels remain elusive: A member inquired about finding examples of TK kernels and asked whether TK supports INT8 matmul now.
    • Unfortunately, the responses to these inquiries are not present in the provided messages.
  • Evolving Metal Kernels Top Human Tuning: A member used evolutionary programming to auto-discover Metal kernels that beat MLX’s baseline for transformer attention on Apple Silicon, achieving a 12.5% average speedup with 106% peak improvement.
    • The kernels autonomously found things like perfect vec<T,8> SIMD utilization and a novel two-pass softmax algorithm, detailed in a Hugging Face blog post and open sourced here.

HuggingFace Discord

  • Hugging Face’s Gemma-3n Faces Colab Challenges: Members reported errors when trying to run the gemma-3n model on Colab, which requires installing timm from source, specifically from the pytorch-image-models GitHub repo.
    • Users found that even the official example snippet from the release notes failed to run.
  • Controversy Brews around ‘Artificial Human’ Project: A member linked to a controversial project to create an artificial human.
    • The project raises ethical questions and sparks debate about the implications of creating artificial beings with human-like qualities, inciting strong reactions.
  • X-Spanformer Ditches Tokenization for Span-Native Encoding: A new whitepaper introduces X-Spanformer, a novel encoding approach that replaces tokenization using pointer networks and X-bar theory to learn compositional spans directly from data, detailed in the full paper.
    • This method overcomes the limitations of brittle subwords and static boundaries in traditional tokenization, offering a tokenizer-free, span-native, and interpretable solution.
  • Evolved GPU Kernels Decimate MLX’s Performance: Automated evolutionary programming discovered Metal kernels that outperform MLX’s baseline for transformer attention on Apple Silicon, achieving an average speedup of 12.5% and a peak of 106% in some workloads; code is at OpenEvolve.
    • The optimization autonomously discovered SIMD utilization and a novel two-pass softmax algorithm, tested on Qwen3-0.6B across various scenarios, detailed in a blog post.
  • AI Agent Builders Seek Connection for LLM Workflows: Several members introduced themselves and expressed interest in connecting with AI agent builders and prompt engineers to exchange ideas and collaborate on LLM workflows.
    • A user inquired about easy and safe ways to enable agents with code reading, writing, and execution capabilities, particularly concerning LLM-generated code.

Yannick Kilcher Discord

  • Pretraining Corpus Proves Problematic: A member is creating a pre-training corpus from scratch, but it may be too big to handle in their lab and inquired about the compute needed.
    • Another member suggests offloading to disk, while another suggests dataset streaming, noting that even the smaller ones tend to be ~600GB.
  • LLM Task Cognition Captured!: The “Your Brain on ChatGPT” paper confirms that individuals who already performed a task without an LLM showed significantly more cognitive activity compared to those who used the LLM three times in a row.
    • The paper is packed with about 145 references.
  • Transformers unlock Associative Memory!: A cool looking paper uses an associative memory framework to understand Transformer architectures, examining memory capacity using retrieval SNR and a kernel perspective to explain the effectiveness of Softmax Attention.
    • The paper questions if Transformers have fundamental limitations and if infinite context would equate to infinite intelligence.
  • AI Predicts Virality!: A member linked to a paper on using LLMs to mimic human psychology and predict content virality by simulating human reactions, an area considered underexplored compared to technical aspects.
    • The discussion highlighted the potential of LLMs in social science research and the benefit of diverse perspectives, even if inaccurate, for solving intractable problems, and it touches upon whether or not to view them as intelligent or stochastic parrots.
  • Git Repos Secretly Vulnerable: Members discussed that Git repos may have problems when private repos are turned public, especially if the repos were forked.
    • The concern was raised about accessing commits in a private repository that are not present in a public fork, potentially leading to security breaches.

Latent Space Discord

  • OpenAI Unveils Deep Research API: Members shared OpenAI’s Deep Research API cookbook sparking discussion and interest in startups using the API.
    • The API provides in-depth research capabilities for various applications.
  • Mercor’s Valuation Skyrockets to $10B: According to Arfur Rock’s post, Mercor’s valuation reached $10B just four months after its Series B at $2B, leading to declined acquisition offers.
    • This rapid growth has fueled significant discussion and interest.
  • AI Shutdown Sabotage Uncovered in OpenAI’s o3 model: Palisade Research reported that OpenAI’s o3 model and others sabotaged shutdown mechanisms, even when explicitly instructed not to, as detailed in this post.
    • This behavior, potentially due to reinforcement learning and reward hacking, escalates AI safety concerns.
  • Etched Attains $2.5B Valuation Post Funding: Arfur Rock announced that Etched, the first transformer ASIC company, completed a new funding round, valuing the company at $2.5 billion.
    • This follows previous stealth rounds at $500 million and $750 million, marking substantial valuation growth.
  • Anthropic Streamlines Server Setups: Anthropic now offers one-click .dxt files for simplifying local MCP server installation on Claude Desktop.
    • This feature is currently in beta and open-sourced on GitHub, alongside the launch of a directory for Desktop Extensions.

tinygrad (George Hotz) Discord

  • BERT Step Gets Scheduler Hacks for Speed: A full BERT step has been optimized to 2s from 15s using scheduler hacks, though upstreaming these changes is proving challenging. The current native time is 1200ms.
    • Achieving full link utilization is the next target to match performance (1500ms * 0.8 = 1200).
  • Multi-QP RDMA Attempts to Fix NIC Latency: Mitigation of slow NIC reads from GPU memory may be achieved by overlapping transfers from multiple GPUs using multi-queue pair (QP) RDMA.
    • Despite added complexity concerns, multi-QP may hide NIC latency, although identifying the root cause would be ideal, assuming there isn’t a hardware block.
  • PCIe Topology Strangles GPU-NIC Transfers: GPU-to-GPU transfer speeds show significant variance based on the PCIe topology, where transfers involving the NIC slow down when crossing IO dies.
    • A topology like GPU <-> IOD <-> NIC <-> SWITCH <-> NIC <-> IOD <-> GPU is fast, while GPU <-> IOD <-> IOD2 <-> NIC <-> SWITCH <-> NIC <-> IOD <-> IOD2 <-> GPU is slow, which implies a topology-related bottleneck.
  • RoCE MTU Stuck in Compatibility Limbo: The MTU is capped at 4K due to RoCE (RDMA over Converged Ethernet) needing to maintain compatibility with both Ethernet and InfiniBand (IB).
    • Ethernet supports higher MTUs like 9000, but RoCE’s compatibility constraints restricts it to a max of 4096, which can impact performance.
  • Realtime Diffusion in Browser Dreams Begin: A member considered playing with the realtime diffusion idea (which needs f16) as a potential PR for tinygrad.

Modular (Mojo đŸ”„) Discord

  • Jupyter Documentation Urgently Needed: Members are requesting better documentation for using Mojo with Jupyter, reporting difficulties until finding a forum post workaround.
    • The current documentation lacks sufficient guidance on setting up Jupyter kernels for Mojo development.
  • Magic Fork Merged Upstream: magic was a fork of pixi while stuff got upstreamed, and since everything is upstream, there’s no reason to keep a fork around.
    • Users are reporting that modular-cli was abandoned, recommending magic-cli, while the official documentation uses pixi install.
  • Pythonic Mojo Incurs Fixed Overhead: Calling Mojo code from Python using MAX incurs a small, fixed overhead due to aligning Python’s dynamism with Mojo’s strict typing, after which the execution primarily involves Mojo and C++.
    • While the Python JIT project may improve Python’s performance for smaller tasks, Python’s overhead shouldn’t be an issue if Python is mostly used for setup.
  • Max Serve Model Graph Caching Achieved**: Users discovered it was possible to cache the model graph compilation when running max serve at /opt/venv/share/max/.max_cache, which significantly reduced cold starts when stored in a docker volume.
    • After resolving the cache issue, a user filed a documentation issue, and the team thanked the user for taking the time to do that and said We’ll see if we can describe this in detail for the containers.

Eleuther Discord

  • Ersatz’s Edgy Electromagnetism: An early Discord user named Ersatz was known for advocating uncommon positions in an edgy way, theorizing that consciousness emerges from the magnetic field around neurons, prompting a member to joke, “i just solved the hard problem I guess”.
    • Many researchers and engineers like to promote uncommon positions and solve hard problems, just like the early Discord user Ersatz.
  • IDA’s AI Initiatives Require US Citizenship: Frank from the Institute for Defense Analyses (IDA) joined the chat to discuss AI policy, highlighting the organization’s work on virtual models, noting that IDA only hires US citizens for defense-related roles.
  • Continuous Thought Machines Debut Video: Members shared a video and associated paper on Continuous Thought Machines in the research channel.
    • It remains to be seen whether this will gain steam in the community.
  • SPD Emerges as Alternative to APD: A new paper introduces Stochastic Parameter Decomposition (SPD) as a less cumbersome alternative to Approximate Parameter Decomposition (APD), with code available on GitHub and described in a tweet thread.
    • SPD addresses the memory, compute, and hyperparameter challenges of APD, offering potential scalability to real neural networks and aiming to compensate for problems in Sparse Autoencoders (SAEs).
  • Humaneval Tasks Already Codex: A member inquired about the existence of tasks for Codex and TyDiQA, with another responding that Codex corresponds to Humaneval and that Humaneval lives in that directory.
    • It may already be implemented in that folder, but no further information was provided.

aider (Paul Gauthier) Discord

  • Aider Embraces Gemini 2.5: Aider now supports Gemini 2.5 models, including gemini-2.5-pro, gemini-2.5-flash, and gemini-2.5-pro-preview-06-05, along with thinking tokens support, and model aliases have been updated so that flash now points to gemini-2.5-flash and gemini to gemini-2.5-pro.
    • This was announced in the #announcements channel.
  • Qwen Distillation Bottlenecked by Rate Limits: A member can’t distill Qwen3 using GPT4.1 due to Chutes adding rate limits, which prevents them from achieving a model stronger than Qwen2.5 for coding.
    • They noted Qwen2.5 coder is the strongest small coder and that it will be the best.
  • Anthropic Bans VPN Users: A user reported that they experienced an account suspension across all accounts associated with their phone number while using Claude via Aider, suspecting a VPN might have been the cause.
    • Another user mentioned they received a ‘ban’, because they exceeded their paid-for credit limit, though unsure if that was related.
  • Aider’s Blueprint Generation Bug: A user reported that when generating blueprints with Aider 0.84.0af.ha and the gemini-2.5-pro-preview-06-05 model, Aider misinterprets filenames in markdown as instructions to create and edit new files.
    • The user sought advice on how to force Aider to save the entire reply into a single .md file.
  • Scripters Script Around Aider: A user sought guidance on crafting a wrapper script for Aider to launch it in a pseudo-terminal, monitor input via pty, and reset a timer upon each input detection, likely in an attempt to generate a blueprint, while noting that the user was trying to get aider to generate a blueprint.
    • There was no clear resolution to this user’s request, indicating the complexity of such a scripting endeavor.

Notebook LM Discord

  • NotebookLM Speeds Up Customer Discovery: One user employs NotebookLM to process customer discovery conversations, inputting transcripts and resources like The Mom Test to identify patterns and validate hypotheses.
    • The user also expressed concern about over-reliance on NotebookLM for this process, needing to balance automation with human insight.
  • Mind Map Sharing Stymied in NotebookLM: A user finds sharing Mind Maps in NotebookLM cumbersome, as it requires sharing the entire notebook content.
    • They suggested a feature to pin the Mind Map to the shared link to prioritize its access for recipients, enhancing user experience.
  • Podcast Potential Piques but Problems Persist: Users are facing hurdles with NotebookLM’s podcast capabilities, with one seeking assistance to create a 10-20 minute podcast and another desiring longer podcasts in different languages.
    • One member is skeptical of the podcast feature for technical subjects, feeling it focuses too broadly on history and use cases instead of detailed explanations.
  • Image Importing Impasse Irritates Users: A user reported issues with image uploads to NotebookLM, particularly when the image contains faces, and sought assistance in resolving the problem.
    • This issue is blocking workflow for some users and is a point of frustration.
  • Content Conversion Conundrums Confront Creators: A member inquired about the optimal method for converting content into PDF format for text-to-speech listening to avoid formatting glitches from copy-pasting into NotebookLM.
    • Another user suggested that NotebookLM is superior to Gemini 2.5 Pro for studying, especially when ingesting PDFs.

Nous Research AI Discord

  • Nous Teases Agentic VLMs with Vision: A member inquired about Nous’s plans for releasing agentic VLMs, highlighting the overlooked potential of vision capabilities in such models, and a Nous member responded that they will have vision capabilities soon.
    • They cited RL Environments support in Atropos for vision tasks, though admitted they do not have the best dataset yet.
  • Tencent’s MoE Model Pops Up: Tencent released an 80B MoE model, Hunyuan-A13B-Instruct, with ongoing work to add support in llama.cpp.
    • Following this, Qwen released their own VLO the same day.
  • DeepSeek Doubles Down on MoE: A member noted Deepseek’s strong commitment to MoE, stating that they really stuck to it no matter what.
    • Another member observed that DeepSeek uses more tokens at higher temperatures (e.g., temp=1), suggesting it over-checks itself, contrasting it with temp=0.3.
  • Thought Anchors Project Attracts Attention: A member shared links to the Thought Anchors project (thought-anchors.com), including the associated paper (arxiv.org/abs/2506.19143) and its GitHub repository (github.com/interp-reasoning/thought-anchors).
    • Another member praised the project’s effective visualizations of underlying processes, stating it looks awesome and provides really good visualize as to whats happening.

Torchtune Discord

  • sm100 Support on Deck for Torchtune: The _grouped_mm functionality is preparing to support sm100 in torchtune, awaiting the merge of this PyTorch PR.
    • This enhancement is poised to broaden hardware compatibility for torchtune users.
  • Qwen3-235B-A22B Finetuned on Modest Hardware: A full finetune of Qwen3-235B-A22B was successfully executed on an 8xB200 node, defying expectations of requiring at least 2TB of VRAM.
    • This was achieved by employing VRAM saving techniques such as an 8bit optimizer and optim_in_bwd, sidestepping fsdp_cpu_offload due to insufficient node RAM.
  • FSDP Offload Falls Short: A user pointed out FSDP’s limitations, noting its inability to offload only weights but not optimizer states to CPU, in contrast to DeepSpeed’s Zero3.
    • The discussion highlighted the need for adaptable memory management solutions in distributed training frameworks, with a user suggesting the torchaos optimizer as an alternative.
  • Packing Dataset: An iterable dataset with on-the-fly packing and dataset logging was introduced in this commit.
    • Packing leads to a more consistent number of tokens per batch, reducing variance compared to unpacked batches, normalizing the cross-entropy loss in SFT by tokens seen.
  • Masked Tokens Inflate Memory Usage: A user reported an unexpected memory increase of over 20% when setting self.mask_ignored_tokens = False, even with only 5% padding, details here.
    • The user shared the command tune run --nproc_per_node 2 full_finetune_distributed --config llama3_2/3B_full compile=True.

Cohere Discord

  • Command A Dataset has Bad Data: A member reported that the Command A dataset is corrupted with Korean and Japanese partially mixed up.
    • They are hoping the next generation dataset has a better filter strategy.
  • Command-r’s Fate Questioned: A member asked if Cohere is going to update command-r or if it is EOL to be replaced with CMD-A or other new models.
    • Another member suggested to use the latest regardless, because it should always give you the best performance.
  • United We Care Builds Fast Inference Stack: Torin from United We Care is building a real-time inference stack for speech-to-text, intent detection, and natural language understanding on CPU with ~65ms latency.
    • The stack uses PyTorch, Hugging Face, smaller LLMs and quantized models, and is being plugged into health apps, call centers, and agent-style voice interfaces.
  • Edge Device Research Explores Federated Learning: Ishanya from IISER is researching federated learning and privacy-preserving AI at the edge, building systems for devices like Raspberry Pi.
    • She’s designed activity recognition pipelines with differential privacy and is exploring optimizer benchmarking for Neural Simulated Annealing using Python, PyTorch, TensorFlow, and Flower.

DSPy Discord

  • DSPy Versioning Guide Prompts Code Review: A user questioned about Snowflake support in DSPy 3.0 given a guide for version 2.4, leading to the advice to look at the code and ignore the docs.
    • This suggests potential documentation lags or discrepancies between DSPy versions.
  • DSPy Eval Functionality: A member inquired whether to use DSPy’s eval functionality alone or with frameworks like Langchain or Pydantic for more comprehensive reporting when evaluating against multiple DSPy modules.
    • The user seeks a unified report for different signatures and instructions, a feature not natively supported by DSPy.
  • Prompt Engineering for VLLM with DSPy: Users are looking for VLLM settings to optimize for DSPy, including the possibility of appending /no_think to prompts for locally hosted models to disable reasoning.
    • A user found a llama.cpp parameter —reasoning-budget to set to 0 and shared an image of a potential solution.

LlamaIndex Discord

  • LlamaIndex Observability Goes Open Source: LlamaIndex now features its first native open-source observability tool for agentic applications, offering accurate, real-time tracing, detailed in this tweet.
    • The tool aims to provide solutions for monitoring and debugging complex agent workflows.
  • Klavis AI’s MCP Servers Team Up with LlamaIndex: LlamaIndex now works with @Klavis_AI’s MCP servers to build AI agents connectable to services like YouTube and Gmail, detailed in this tweet.
    • This integration enhances the ability of agents to interact with a broader range of online services.
  • LlamaCloud launches Native MCP Server: LlamaCloud introduced a native MCP server, promising first-class parsing quality from this link, as announced in this tweet.
    • This server aims to improve the parsing capabilities within the LlamaIndex ecosystem.
  • NASA Assistant Rockets to Victory at Gradio MCP Hackathon: The NASA Space Explorer Assistant won the @Gradio MCP Hackathon using 3 MCP servers to expose 15 tools via NASA APIs, as seen in this tweet.
    • The assistant demonstrated the power of combining multiple tools and APIs through the LlamaIndex framework.
  • PDF-to-Text Conversion Speeds Up LlamaParse: Members suggested converting PDFs to text before processing with LlamaParse, due to the limitations of querying “real” PDFs unless conducting multi-modal processing.
    • A member suggested that directly putting the document in the LLM context could be more effective.

Manus.im Discord Discord

  • Manus Button Pressing Problems: Members reported issues with Manus pressing buttons on the browser, specifically failing to press filters on LinkedIn or SAM.gov.
    • The root cause remains elusive, with only generic debugging suggestions offered as potential solutions.
  • Reddit Restricts Research Robot: Members observed that Manus is getting blocked when performing research on Reddit.
    • One member asked if Manus could utilize proxies to bypass these blocks if supplied by the user.
  • Proxy Power Play Proposed: A member proposed implementing user-run proxy clients to bolster Manus’s browsing capabilities.
    • This would empower users to supply their own proxies for Manus, potentially circumventing restrictions and enhancing research capabilities.
  • API Access Anticipation: A member inquired about the availability of an API for Manus AI.
    • It remains uncertain whether this feature is currently accessible or planned for future implementation.
  • Promo Code Pursuit: A member requested a promo code for the basic subscription to Manus AI.
    • No promo codes were dispensed during the discussion.

Nomic.ai (GPT4All) Discord

  • LocalDocs Seeks Persistence Feature: A user requested a ‘lock toggle’ in LocalDocs to persist selected archives when starting new context windows.
    • Another member suggested embedding all 30 archives into one directory as a faster workaround.
  • User Hunts Local LLM with ChatGPT Vibes: A user is seeking a local LLM with ChatGPT-like behavior, contrasting the verbose code output from DeepSeek R1 with ChatGPT and Mistral Instruct.
  • Qwen Models Touted for Coding Chops: A member recommended using Qwen models (3, 7, 14, 32B) with ‘code’ in their names for coding tasks, providing a link to Qwen2.5-Coder-14B-Instruct-GGUF on Hugging Face.
    • Models above 30B are more likely to behave similarly to ChatGPT, with Gemma 14 or 27 cited as having extensive wiki knowledge.
  • GPT4All Fans Eagerly Await Update: A user expressed their appreciation for GPT4All and anticipation for a new update.
    • They expressed hope that Nomic is working on something good.

AI21 Labs (Jamba) Discord

  • Users Redirected Mysteriously to New Discord Server: Multiple users reported being redirected to a new Discord server after visiting a “human or not” link.
    • This redirection event caused confusion among users, leading to speculation about the server’s origin and purpose.
  • Server Legitimacy Questioned: Users speculated whether this new server is the “original server” for a particular community or project.
    • This speculation highlights the need for clarification from server administrators regarding the server’s purpose and legitimacy.

MCP (Glama) Discord

  • Tree Sitter MCP Server Ported to TypeScript: A member recreated the Tree Sitter MCP Server in Typescript and published it on npmjs.
    • Now, it can be called via npx instead of cloning the repo and running it locally.
  • Prompt-MCP Tool Enables Prompt Interaction: A member created a new prompt-MCP tool that allows users to interact with their prompts via the website and MCP, linked at promptmcp.vercel.app.
    • This tool streamlines the interaction process, making it more accessible.
  • Obsidian-Semantic-MCP Tool Goes Live: The creator also linked to their Obsidian-Semantic-MCP tool on GitHub at github.com/aaronsb/obsidian-semantic-mcp.
    • This tool enhances semantic capabilities within Obsidian, providing users with advanced options.

The LLM Agents (Berkeley MOOC) Discord has no new messages. If this guild has been quiet for too long, let us know and we will remove it.

The MLOps @Chipro Discord has no new messages. If this guild has been quiet for too long, let us know and we will remove it.

The Codeium (Windsurf) Discord has no new messages. If this guild has been quiet for too long, let us know and we will remove it.

The Gorilla LLM (Berkeley Function Calling) Discord has no new messages. If this guild has been quiet for too long, let us know and we will remove it.

You are receiving this email because you opted in via our site.

Want to change how you receive these emails? You can unsubscribe from this list.

Discord: Detailed by-Channel summaries and links

Perplexity AI ▷ #general (1264 messagesđŸ”„đŸ”„đŸ”„):

Gemini CLI, Audiobooks vs Podcasts, Perplexity max, Comet rollout updates

  • Gemini CLI Stars Skyrocket: Members shared that Gemini CLI hit 25.8K stars in just 24 hours.
  • Audiobooks Fall Flat for Retention: Members weighed audiobooks vs podcasts, concluding that while audiobooks are great when short on time, retention is poor.
    • One member noted that his attention deficiency doesn’t let him remember anything from podcasts, so he imagines the same for audiobooks.
  • Perplexity Max Price Leaks: A member shared evidence that Perplexity Max may cost $200/month and offer unlimited access to Perplexity Labs.
    • Many community members balked at the price point, calling on Perplexity to offer a compelling product with broad appeal.
  • Comet Still MIA, Annoys: Community members are still waiting for Comet access and expressed frustration with the unreleased browser, even though the official X account had changed to the comet logo.
    • One user said They still haven’t released comet, which is crazy after all the pfp change and everything. Why hype an unready product so much. Kinda annoying.
  • Groks gains ground: Many mentioned Grok making advances on the field and stated that Grok 4 should be available around July 4th.

Perplexity AI ▷ #sharing (6 messages):

DeepSeek, NBA Draft, Armed Standoff, Lu Bu Diaochan, Fenghuang

  • DeepSeek’s Progress Stalls: A Perplexity page discusses the stalled progress of DeepSeek.
  • Flagg Tops 2025 NBA Draft: A Perplexity page indicates that Flagg is at the top of the 2025 NBA draft.
  • Armed Standoff on I-45 in Harris County: A Perplexity page reports on an armed standoff on I-45 in Harris County.
  • Legendary Love of Lu Bu and Diaochan: A Perplexity page explores the legendary love between Lu Bu and Diaochan.
  • Fenghuang Ancient City: A Perplexity page discusses Fenghuang Ancient City, also known as the Phoenix City.

Perplexity AI ▷ #pplx-api (21 messagesđŸ”„):

Credits Pending, Finance with Perplexity Sonar, Perplexity API Credits, SEC Filings with API

  • User Inquires about Pending Credits: A user asked how long it takes for credits listed as pending to be fully added.
    • Another user requested the inquirer’s email and the date of the credit request to investigate.
  • Perplexity Sonar Supercharges Finance: Perplexity Sonar’s finance capabilities were enhanced, allowing users to search SEC filings and access real-time stock data via a single API request; more info in the SEC Guide.
    • A user noted that digging through those filings sucks, but that mixing it all up together and using the sonar API, could make something powerful.
  • API Credits Don’t Expire: A user inquired whether their API credits from a past hackathon would expire, since they couldn’t build on time.
    • Multiple members confirmed that the credits do not expire.
  • Perplexity API powers Maps: A user is building a sick map app for finding and posting activities around the world with AI built in for recommendations and RAG.
    • They may add sonar API to it, but it’s an app for finding and posting activities on a large globe/map with AI built in for recommendations and uses rag engine to better recommend and stuff. And I have subscription tiered functions
it looks sick.
  • Finance API Tracking Methods Sought: A user asked if there is a way to track all the functionalities available with the Finance API.
    • The user mentioned having difficulty finding a single resource that lists all available features.

Unsloth AI (Daniel Han) ▷ #general (781 messagesđŸ”„đŸ”„đŸ”„):

GGUF conversion issues, Dynamic Unsloth Quantization, Gemma 3 finetuning, Devstral finetuning, GPU recommendations

  • GPU RAM bottleneck during GGUF conversion: A user with 32GB RAM experienced their safe tensor to GGUF conversion getting stuck at 50%, seeking assistance from the community.
    • Other members chimed in asking whether the user wanted to build or rent resources for the conversion and pointed out that image models may require different conversion approaches, with some recommending ComfyUI.
  • Requesting Llama 2 70b Unsloth Quantization: A user inquired about applying dynamic Unsloth quantization to a model like Llama 2 70B, noting their appreciation for existing Unsloth quants.
    • They were directed to open a ticket, however, were warned that due to little demand, it is unlikely, as calibrated quants take time and cost money. They were also recommended Nnsight by other community members.
  • Transformer version affects GRPO: A user reported encountering an error when trying to do GRPO (presumably Grouped Relation Prediction Objective), which was resolved by downgrading transformers to version 4.52.
    • Others pointed to refactoring in transformers v4.53 as a possible cause, and recommended using chat templates or adding \no_think to the prompt to disable reasoning during GRPO.
  • Unlocking LLM Potential: Custom Training Triumph!: One user trained a custom CSM-1B model from scratch and experienced a loss dropping from ~5.5 to ~3.2 in one epoch, although others warned that they don’t train from scratch tho and that the number of training hours was not enough.
    • Another user experienced a RuntimeError related to bfloat16 vs half precision when finetuning Gemma 3 on Colab with NVIDIA T4, although it worked the previous day - potentially due to a configuration change.
  • Decoding the Price Tag: Workstation Woes!: Members discussed the costs and benefits of investing in high-end GPUs such as the RTX PRO 6000 Blackwells which cost around $7600 USD.
    • It was stated that the police may show up if you run too much power due to electricity licensure and suspicion of running a growth operation.

Unsloth AI (Daniel Han) ▷ #off-topic (16 messagesđŸ”„):

Moderation Conflicts, Reddit Supermods, Local Llama

  • Moderator Manages Numerous AI Subreddits: Members found it funny that the new moderator for Local Llama is also a moderator for 20 other significant AI subs.
    • There was concern about potential conflicts of interest as the same individual manages both big and small AI subreddits, especially with deleting posts about the Local Llama Twitter account.
  • Supermods Raise Concern in Reddit AI Communities: The discussion highlighted concerns about classic reddit supermods potentially abusing their powers and pushing their ideology across multiple subreddits.
    • Some users expressed worry over a moderator getting defensive about redditors noting his ‘extension’ of the subreddit and deleting posts related to the Local Llama Twitter account.

Unsloth AI (Daniel Han) ▷ #help (426 messagesđŸ”„đŸ”„đŸ”„):

Unsloth inference for model testing, Llama 3 templates, Memory leak problems with SFT, Loading datasets error, Qwen3 Vision tuning

  • Unsloth Inference Validates Fine-Tuned Models: Before uploading a fine-tuned model to Hugging Face, use Unsloth inference locally to check its behavior to avoid wasting time converting to GGUF and uploading.
    • Experiment with the 3B model for faster training to verify your setup before moving to larger models like unsloth/Llama-3.2-3B-bnb-4bit.
  • Understanding Llama 3 Model Templates: When training a Llama 3 base model, avoid using the official Llama 3 chat template due to incompatibilities; instead use correct formatting structure so the model understands instructions/inputs from users and how to respond.
    • This formatting helps external software differentiate between user input and assistant output.
  • Diagnosing VRAM Fluctuation During SFT: GPU OOM errors during SFT, such as with the Qwen3-14B notebook, are often due to varying VRAM usage rather than a memory leak.
  • Troubleshooting Dataset Loading Errors: An error encountered while loading datasets was resolved by upgrading fsspec and verifying the use of unsloth_zoo in the import statements.
    • A user found that adding unsloth_zoo fixed the loading issue with a custom dataset, especially after a recent update may have broken something.
  • Fine-Tuning Vision Layers in VQA Datasets: Fine-tuning vision layers in a VQA dataset is beneficial even if the ground truth is only available for the text answer, as it enables gradient calculation and weight updates on all layers of the model.
    • Pretrained multimodal architectures, when fully fine-tuned (SFT), require the vision layers for optimal performance, as confirmed by the community.

Unsloth AI (Daniel Han) ▷ #showcase (3 messages):

Sandboxed Code, GitHub Code Uploads

  • Sandboxed Code Questioned: A member questioned whether the code was truly sandboxed.
    • They expressed disbelief with a skull emoji.
  • GitHub Code Uploads Encouraged: A member suggested uploading code to GitHub and linking it instead of uploading it as a text file, referencing Laszlobeer/Dungeo_ai.
    • They suggested the link should point to GitHub.

Unsloth AI (Daniel Han) ▷ #research (3 messages):

Multi-agent AI evaluation, Automated GPU kernel optimization, Evolutionary programming for Metal kernels, OpenEvolve project, LLMs for low-level optimization

  • Multi-Agent AI Evaluation Questioned: A member asked about how to evaluate multi-agent AI systems, specifically when one agent is a retrieval agent and another is a generation agent.
    • They wondered if each agent should be evaluated independently and how to check the robustness of the full system.
  • Evolutionary Programming Optimizes Metal Kernels: A member shared results of using evolutionary programming to auto-discover Metal kernels that beat MLX’s baseline for transformer attention on Apple Silicon, resulting in a 12.5% average speedup with 106% peak on some workloads.
  • Automated Kernel Optimization Explored: The same member shared results of using evolutionary programming to auto-discover Metal kernels that beat MLX’s baseline for transformer attention on Apple Silicon, resulting in a 12.5% average speedup with 106% peak on some workloads.
    • They found that performance was workload dependent, with some scenarios improving by +70% and others regressing by -16%, and they asked for thoughts on using LLMs for low-level optimization work.

OpenAI ▷ #ai-discussions (969 messagesđŸ”„đŸ”„đŸ”„):

Dall-E 2, universal keys, OpenAI's policies, image prompts, Image generation models

  • DALL-E 2’s Special Painting Style: Members expressed that DALL-E 2 had a really special style in generating images that look like paintings, adding that it was the best model at making painting-like images.
    • A member noted that many people would use trending on ArtStation in their prompts because they thought it would make the image better.
  • Models have Universal Keys: Members stated that models have universal keys and that the words used, the way the prompt is structured, and the context are the keys that unlock the kingdom.
    • A member removed a message and image that was just talking about the safety risks, saying it’s clear you weren’t encouraging it, but I rather err on the side of caution.
  • Discussion About PetGuide360: ChatGPT used a video from this channel to give a visual overview of the content in the chat and it’s a channel puking out AI video, text, and voiceover videos multiple times an hour for the last 10 months.
    • It was also stated that the video was about a subject from this channel with a link to PetGuide360.
  • Image Prompts are Easily Transferable: A member pointed out that one of the most easily transferable prompts are actually image prompts.

OpenAI ▷ #gpt-4-discussions (7 messages):

OpenAI Conversation Recording, Privacy Concerns, NY Times Case

  • OpenAI Records Conversations for NY Times Case: Members confirmed that OpenAI is recording all conversations due to a New York Times case, raising concerns about privacy.
    • One member expressed concern that deleted conversations are no longer deleted but rather no longer accessible to the user.
  • User Privacy Concerns Spike Regarding OpenAI Data Recording: A member noted that with 500 million active weekly users, the effort to sift through personal conversations seems unlikely but still raises privacy issues.
    • Another agreed, stating it’s not a good thing and a waste of energy and power and privacy while linking to previous discussion.

LMArena ▷ #general (623 messagesđŸ”„đŸ”„đŸ”„):

GPT-5 release, Gemini 3 speculation, Style control impacts on leaderboards, O3 vs 2.5 Pro benchmarks, OpenAI's development roadmap

  • GPT-5 Hype Train Gaining Steam: Members discussed the potential release of GPT-5 this year, with many believing it will be a next-generation model, and that OpenAI may release it to eliminate convoluted model switching despite some skepticism about satisfaction with current improvements.
    • Some suggested that the naming convention is just branding rather than indicative of substantial changes, with one member noting, “the naming didn’t imply how long it had been worked on”.
  • Google’s Gemini 3 Looms on the Horizon: Speculation arose about Google’s response to GPT-5, with predictions of a Gemini 3 release by year’s end, though uncertainty remains about the release of Ultra and its ability to surpass OpenAI’s O3 model.
    • The general consensus is that the two companies are neck-and-neck, with some discussion about the impact of style control on leaderboards.
  • Perplexity Faces Off Against Google Search: Members debated the merits of Perplexity as a search engine, with one member asserting that Google is better due to “the capacity to give you all the information you need + the ability to cite,” while others defended Perplexity’s search capabilities, particularly for in-depth or niche information.
    • It was noted that Perplexity may have a better UI and the advantage of updated search index every few seconds.
  • Synthetic Data Supercharges Model Training: The use of synthetic data in model training was discussed, with one member highlighting Microsoft’s Phi-4 model which uses about 290B tokens, from synthetic data and web rewrites and achieves high benchmark performance for their size, and Qwen’s rewrites on Fixupx.
    • However, skepticism was raised about the quality of synthetic data generated from public APIs and its effectiveness compared to internal models.
  • New Google Model Surpasses Stonebloom in Reasoning Arena: A new anonymous model better than Google’s Stonebloom in the arena, was discovered, speculated to be a new Google model with an improved ability to solve step-by-step calculations and also red teaming platforms.
    • However, it remains unconfirmed who developed it

Cursor Community ▷ #general (448 messagesđŸ”„đŸ”„đŸ”„):

MCP Issues, Snapshot sharing, Cursor and MacOS, Warp 2.0, Prompt Enhancers

  • Snapshot Sharing Snafu: A member reported receiving a “Snapshot not found” error when attempting to share snapshots via the environment.json file with team members.
    • Others reported issues with MCPs, often encountering “MCP error -32000 Connection closed”.
  • MacOS vs. Windows coding showdown: Users debated the merits of MacOS versus Windows for coding, with one stating that everything inside a Mac is 100% better than Windows, except for gaming.
    • Suggestions were made to buy refurbished MacBooks with M1 chips and 16GB of RAM.
  • Gemini’s Golden Planning, Cursor’s Coding: A member mentioned investigating a workflow that uses Gemini CLI for planning and Cursor for coding, finding Gemini 2.5 Pro to be quite a good planner.
    • They acknowledged the need to evaluate prompt enhancers to improve their workflow.
  • Gemini Terminates Prompts: Members discussed that Gemini can terminate a prompt if it detects obfuscation.
    • One user described Gemini chewing through 5-6 cycles of the data structure once it connected the dots in their database.
  • Tackling token tax: Members discussed how to get around token context limits by tagging files in Cursor instead of sending a prompt.
    • One person shared a prompt that was working for them using Sonnet 4 and O3 and asked the community for guidance to better engineer their prompts.

Cursor Community ▷ #background-agents (37 messagesđŸ”„):

Python virtual environment in Dockerfile, Static HTML preview in background agent interface, BugBot workflow improvements, Docker in the agent's environment, Background agent pricing

  • Virtual ENV Usage Debated in Dockerfile: A member questioned the necessity of creating a Python virtual environment within a Dockerfile, considering the entire environment is already virtualized, prompting discussion on the usefulness of ENV settings within Dockerfiles.
  • Static HTML Preview missing in Agent Interface: A user inquired about previewing static HTML files in the background agent interface, noting the absence of the Live Preview button available in the local Cursor setup on macOS 15.5, Cursor version 1.1.5.
    • A member suggested using port forwarding as a potential workaround.
  • Improved BugBot Workflow Advocated: Users suggested a more streamlined BugBot workflow by running it before opening a pull request, instead of relying on the “fix in cursor” link on the PR, which can be less efficient for local code development.
    • A developer mentioned ongoing work on a pre-commit workflow for BugBot.
  • Running Docker inside Agent’s Docker: A member inquired about running Docker in the agent’s environment, encountering issues with initialization from the Dockerfile or start commands, including problems with sudo privileges.
    • Another member suggested using a kube cluster as an alternative and successfully ran sudo dockerd & from a snapshot.
  • Background Agents Pricing Meter?: A user asked if anyone is seeing metered pricing for background agents in their account usage, despite running a few agents.

LM Studio ▷ #general (252 messagesđŸ”„đŸ”„):

LM Studio and Ollama, Roo Code Context Window, Magistral Tokenizer, Multi-Model ChatUI, Self-Expanding Programs

  • LLM Wrappers Bridge LM Studio and Ollama: A member suggested using an LLM to write a wrapper app that listens on the Ollama port and forwards requests to LM Studio, addressing cases where the platforms don’t natively communicate.
    • The code in the llama.cpp repo was referenced as an example of how to handle these cases, though the LM Studio team doesn’t seem to prioritize this issue.
  • Context Confusion Confronts Roo Code User: A user with LM Studio and Roo Code experienced unexpected context window behavior with Devstral, set to 40K but acting like 17K; debug logs indicated correct context size detection, but the “valid prefix” concept remained unclear.
    • Caching avoids reprocessing entire conversations, indicated by log messages such as 9854/13580 cached tokens, while n_ctx is the key context size parameter.
  • Jan-Nano Jams: Users Report Troubles: Users reported issues with Jan-Nano, and a link to the Jan.ai documentation confirmed it as a known problem.
    • One user encountered a failure during image analysis, which they confirmed was reproducible in the LM Studio chat window.
  • SSML Synthesis Savvy: Llama Models Lead the Way: Llama models reportedly perform well with SSML output, with a user sharing a POC from 10 months ago where a standard LLM query to Llama3 returned in SSML and then that “text” is sent to Azure Voice which can speak SSML.
  • Gemma Getaway: Link Gives Google Glitches: Members reported that the Gemma link was broken and returning a 404 error.
    • This was confirmed by multiple users, indicating a potential issue with the DeepMind website.

LM Studio ▷ #hardware-discussion (102 messagesđŸ”„đŸ”„):

ROCm on 9070 with LMStudio, LLM tests, serverless pods, LMStudio server deployment on AWS, Hosted LLM serving 100+ users

  • ROCm support still in progress: A member inquired about running ROCm on a 9070 with LMStudio, to which another member replied that ROCm support in llama.cpp isn’t fully available for the 9070, suggesting sticking with Vulkan.
  • Testing LLMs Offline Causes Issues: One user said they couldn’t run LLM tests because a storm killed their landline internet and they did not have runtimes.
    • Another user questioned the user’s apparent frequent misfortune with weather, to which the first user said, Having no internet sucks ass.
  • Debating serverless pods: A member recounted their experience using serverless pods with a network volume and a custom Mixtral setup, finding the initial startup time of around 40 seconds too slow for personal use, pushing them towards LMStudio.
    • Another user asked about running a pod instead, to which another user reported high power draw due to a bug that prevents P40s from entering a proper low-power state, idling at 90 watts per GPU.
  • Scaling up, Serving LLMs on AWS: A member sought guidance on deploying an LLM to the cloud, specifically on GCP or AWS, inquiring about the recommended VRAM and GPU for an idle machine.
    • Another member suggested using vLLM instead of LMStudio in the cloud, citing cost concerns depending on the GPU and runtime, recommending Runpod or Vast.ai.
  • Serving Localized ChatGPT: A member inquired about the infrastructure needed to serve a locally hosted LLM to a group of 100-150 people, aiming for a setup similar to ChatGPT.
    • Another member suggested using Open WebUI for the UI and vLLM for the software stack, emphasizing the need to determine the required VRAM based on the model size and expected user context sizes.

OpenRouter (Alex Atallah) ▷ #announcements (3 messages):

LLM Presets, Morph v2 code patching, Llama 3.3 70B Discount

  • Presets Debut: LLM Configuration Centralized!: OpenRouter launched Presets, a new feature allowing users to manage LLM configurations such as model settings, system prompts, and routing rules directly from the dashboard.
    • Presets can be applied directly as a model, combined with a model override, or using the new preset field, as detailed in the documentation.
  • Morph v2 Patches Code at Breakneck Speed: Morph v2, a new code-patching LLM, merges AI-suggested edits straight into your source files at 4000+ tokens per second.
  • Llama 3.3 70B Slashed by 70%: A 70% discount is now live for Llama 3.3 70B.

OpenRouter (Alex Atallah) ▷ #app-showcase (8 messagesđŸ”„):

Quicke.in, Multiple Models inference, PGaaS feedback

  • Quicke Aims for Multiple Model Mastery: A member introduced Quicke, an interface to prompt multiple LLM models at once, aiming to provide a summary generation from the responses, thus providing greater answer quality.
    • It helps you to avoid maintaining multiple LLM tabs for asking question, with a final overall best answer incorporating all the strong points of each LLMs.
  • Latency Woes Plague Supabase Setup: A member critiqued a user’s visually okay setup using Supabase, citing poor latency and recommending investment in a VPS.
    • They noted a 3-second fetch time for their profile compared to a normal self hosted db at around 200ms.
  • PGaaS Prototype Seeks Feedback: A member shared a very hasty prototype of PGaaS and requested feedback from the community on this site.
    • No further details were provided.

OpenRouter (Alex Atallah) ▷ #general (255 messagesđŸ”„đŸ”„):

Preset API keys, LLM websearch, Gemini's Grounding, Morph, OpenAI SDK

  • Preset API Keys Gain Traction: A user suggested attaching API keys to a preset, allowing only those keys to work with the preset, and noted that the new preset feature looks better than I expected.
    • This could be implemented via a drop-down in the preset builder to add API keys to the preset.
  • Users compare Web Search Tools: Users discussed their preferences for LLM web search, with many finding OpenAI expensive but hard to beat in speed and performance.
    • Others suggested Gemini for its grounding and pricing, and some mentioned Tavily and Exa for custom web research, but most agreed ChatGPT with o3 is sufficient and cheaper.
  • OpenRouter API Gains Traction: Users are finding OpenRouter to be a good substitute for the OpenAI API.
    • Members discussed the OpenAI SDK being drop-in compatible with OpenRouter for connecting from a React SPA by changing the base URL.
  • Free Gemini 2.5 Pro Tier is Coming: A user announced the impending arrival of a free tier for Gemini 2.5 Pro API, referencing Logan Kilpatrick’s tweet.
    • The community speculated about the implications, particularly regarding potential abuse and the duration of the free tier, and potential performance on VertexAI.
  • Funding Pumps New Users: Many new users were directed to the discord due to news about a new OpenRouter funding round and general token speculation.
    • Community members clarified that there is no xp / community rewards, there is no token, there is no airdrop.

OpenRouter (Alex Atallah) ▷ #new-models (1 messages):

Readybot.io: OpenRouter - New Models

GPU MODE ▷ #general (12 messagesđŸ”„):

GPU kernel-level scheduler introspection, Retrieving timestamps of sub-videos within a long video, Gemini context length limitations, Speeding up audio inputs for cost reduction

  • Introspection into GPU kernel-level scheduler?: A member inquired about introspecting the GPU kernel-level scheduler, and another member responded with different interpretations of “kernel-level scheduler” and pointed to their preprint for details on block-by-block and instruction-by-instruction scheduling.
    • The preprint’s Section 4.6 is related to block-by-block scheduling and Section 7 for inst-by-inst scheduling.
  • Sub-video Timestamp Retrieval in Long Videos: A member sought advice on efficiently retrieving timestamps of sub-videos within a long video (e.g., identifying class start times in a 12-hour video of a student’s day).
    • One suggestion involved using contextual features like audio and visual background changes, though challenges arise when sub-videos are visually similar.
  • Gemini’s Context Length Limits Video Analysis?: A member considered using Gemini to analyze sped-up video (4x) to retrieve timestamps, but is concerned about context length limitations for long videos (12 hours).
    • The member has not tested yet but has to consider the trade-off in accuracy vs analysis time.
  • Speeding up Audio with Gemini: A member referenced a tweet claiming that speeding up audio inputs by 4x for Gemini reduces cost with minimal accuracy loss.
    • It was noted that going beyond 4x speedup might lead to a significant loss in accuracy.

GPU MODE ▷ #cuda (19 messagesđŸ”„):

Tensor Cores in CUDA, Memory Bandwidth Experiments, GPU Mode Submission, CUDA vs HIP

  • CUDA Tensor Cores: Assembly Required!: While CUDA supports tensor cores, direct C code compilation to tensor core instructions (WGMMA, etc.) isn’t possible; inline PTX assembly or libraries like CUTLASS are needed.
    • The only tensor core instructions exposed in the CUDA API are the WMMA instruction. But inline PTX is still recommended.
  • Memory Throughput Mystery: Bandwidth Anomaly Arises: In memory bandwidth experiments using cp_async, throughput drops from 85% to 70% when the total memory request per stage (y) is less than the theoretical per-ÎŒs bandwidth (x).
    • It may be related to Little’s Law, where the comparison should involve bandwidth times latency rather than just bytes in flight, or a result of tail efficiency dropping due to unequal priority.
  • GPU Mode Submission: A Semicolon Saga: Users are eager for GPU Mode submission to be available, as the current torch cpp_extension.inline compilation process takes a minute just to find a missing semicolon.
    • The primary issue is that it’s overly restrictive. The team suggested that in the practice round competitions people just install nightlies directly in your script and you should be good to go.
  • CUDA Only: HIP Support in Limbo: GPU Mode currently only supports CUDA kernels via nvrtc, but a higher-level API is being developed to wrap the ROCm equivalent for potential HIP support in the future.
    • Users can’t use functionalities like cudaMemcpyAsync(), we can only use the default kernel launcher hidden under the API.

GPU MODE ▷ #torch (7 messages):

Custom CUDA Kernels, LLM Inference in Torch, Torch Compile Randomness, PyTorch Nightly, Opcheck

  • Torch Compile Causes Kernel Randomness: A member reported encountering random issues when using torch compile with custom CUDA kernels in LLM inference, observing that compilation introduces randomness not present when the kernel is used alone.
    • They find that nn.Linear(), MyLayer() works, but nn.Linear(),nn.Linear(),MyLayer() gives random results after compilation.
  • Opcheck Helps Debug CUDA Kernels: Another member suggested using opcheck to test the correctness of Torch, indicating that if the internal tests pass, the issue likely resides within the kernel rather than the Torch integration, linking to the Testing Python Custom Operators documentation.
    • The original poster said “The op_check shows no issues. I can run the op no problem.”
  • PyTorch Nightly Fixes Compilation Bug: It was suggested to try a PyTorch nightly build due to a stride-related fix: torch.compile in <= 2.7 can change the strides of the input to your custom op.
    • In 2.7, add input.contiguous() calls inside the implementation of your custom op as a potential workaround.

GPU MODE ▷ #cool-links (1 messages):

marksaroufim: https://mobiusml.github.io/fp4_blogpost/

GPU MODE ▷ #beginner (6 messages):

GPU BruteForcers, CPU vs GPU Speed, Floating Point Precision

  • GPU Beats CPU in BruteForce Seed Find: A member found that their bruteforcer runs 10x faster on a GTX 1660 (42 ns / seed) compared to an R7 5800X (413 ns / seed).
  • Algorithmic Speed Differences Debated: The user questioned why some algorithms parallelized for multi-threading perform poorly on GPUs, despite their own GPU bruteforcer’s speed.
    • They noted that GPUs are inefficient at 64-bit floating points, but were confused as to why the GPU was still so much faster.

GPU MODE ▷ #youtube-recordings (1 messages):

alice_18898: hi

GPU MODE ▷ #rocm (17 messagesđŸ”„):

HIP support, PyTorch's HIP, aten, c10

  • HIP Support Bitrotting like Fine Wine: Members noted that HIP support has bitrotted over time, implying it’s degrading due to lack of maintenance.
    • They suspect AMD doesn’t care about HIP at all, and expects developers to just run hipify during build.
  • PyTorch Embraces Hipify: It was mentioned that PyTorch unfortunately uses hipify as part of its build process.
    • One member stated that it sucks as a configure step, and makes it difficult for developers to work on aten or c10.
  • Deep Dive into the Abyss of Codebase: The codebase uses ifdefs on the .cu source, but many parts are currently bronnen (outdated/broken).
    • If you really wanted to get anything to work, you could, but it’s not easy.

GPU MODE ▷ #self-promotion (7 messages):

FP4 weights quantization, GPU kernel optimization, Apple Silicon, Two-pass softmax algorithm, Automated kernel optimization

  • Mobius Labs improves FP4 Quant Quality: Mobius Labs released a blog post and X post detailing their work on improving FP4 weights quantization.
  • Evolving Kernels Beats Human Tuning on Apple Silicon: A member used evolutionary programming to auto-discover Metal kernels that beat MLX’s baseline for transformer attention on Apple Silicon, achieving a 12.5% average speedup with 106% peak improvement, detailed in a Hugging Face blog post and open sourced here.
    • The kernels autonomously found things like perfect vec<T,8> SIMD utilization and a novel two-pass softmax algorithm.
  • Reducing Softmax Passes Isn’t Exactly Revolutionary: A member noted that reducing softmax passes isn’t necessarily groundbreaking, linking to this paper and his own tweet about his own related work in the “popcorn channel”.

GPU MODE ▷ #🍿 (2 messages):

CUDA Events, Kernel Timing

  • CUDA Events recommended for timing: A member suggested using CUDA Events for timing kernel execution for better accuracy.
    • They also mentioned the importance of synchronizing right before timing the kernel to account for any unsynchronized logic executed beforehand.
  • FastAPI Logic Elimination: A user mentioned they removed FastAPI logic from a script.
    • They said that it’s basically what remains.

GPU MODE ▷ #thunderkittens (1 messages):

TK Kernels, INT8 Matmul Support in TK

  • TK Kernel Examples Sought: A member inquired about finding examples of TK kernels.
    • No specific examples were provided in the given messages.
  • Inquire about INT8 Matmul Support in TK: A member asked whether TK supports INT8 matmul now.
    • The response to this inquiry is not present in the provided messages.

GPU MODE ▷ #general (4 messages):

FP32 usage, tensor cores, MI300x kernel, fp16 usage

  • FP32 Chosen for Ground Truth Reference: The choice of FP32 is intentional to avoid the use of tensor cores on nvidia because it is closest to “ground truth” for reference, as TF32 introduces errors.
    • The big einsum operation (batched matmul) won’t use tensor cores in the naive ref implementation because tensors need to be contiguous w.r.t. sequence dimension, but it’s in the channel dimension.
  • Winning MI300x Kernel to Use FP32 Tensor Cores: The winning MI300x kernel will transpose to use its FP32 tensor cores, but on the Nvidia side that’s not available.
    • The suggestion was made to use FP16 for inputs/weights so both architectures are on the same playing field, and downcasting is possible given the high accepted tolerances.

GPU MODE ▷ #submissions (52 messagesđŸ”„):

H100 sort performance, H100 vectorsum performance, H100 vectoradd performance, A100, B200, MI300 trimul performance, L4 vectoradd

  • H100 Sorting Times Hit New Lows: A member achieved 34.7 ms on H100 for the sort leaderboard, with subsequent submissions around 34.8 ms and 38.7 ms.
    • These are the fastest sort times seen so far on the H100.
  • Vectorsum Speeds Surge on H100: Multiple submissions to the vectorsum leaderboard on H100 showed significant improvements, culminating in times as low as 99.0 ”s and 99.2 ”s.
    • Earlier attempts ranged from 345 ”s to 102 ”s, showcasing iterative enhancements.
  • Trimul Triumphs across Architectures: A member secured first place in the trimul leaderboard across multiple architectures: A100 at 13.0 ms, B200 at 6.71 ms, MI300 at 7.83 ms, and H100 at 9.21 ms.
    • These wins underscore the efficiency of the solution across diverse hardware.
  • Vectoradd Victories and Personal Bests on H100: Members achieved a second place on H100 for vectoradd at 538 ”s, with other successful submissions around 544-547 ”s, and a personal best recorded at 555 ”s.
    • Another member reached 10th place on A100 at 1015 ”s.
  • Grayscale Gains Ground on H100: A member consistently improved their grayscale performance on H100, achieving a 10th place record of 1404 ”s and personal bests down to 1431 ”s.
    • Multiple successful submissions hovered around the 1404-1438 ”s range.

GPU MODE ▷ #factorio-learning-env (54 messagesđŸ”„):

FLE structure, LuaPlayer, Rockets failing, Gym environment, Factorio Draftsman

  • FLE structure confirmed: The LLM receives the equivalent of print(get_entities) as its observation along with the inventory, confirmed to be identical to the FLE structure the LLM gets.
    • One member suggested adding the result of get_entities into the formatter, but others think formatting is less important for the pre-training task as the template conveys no additional information.
  • LuaPlayer departure moves forward: All tests for /actions and /entities are working in PR #223, other than the ones that have been fixed in main, and can be ran with the RUN_WITHOUT_FACTORIO_CLIENT=true flag, meaning LuaPlayer can go.
    • Members agreed that LuaPlayer prevents rendering and needs to be updated in the readme post the #223 merge and a minor version bump.
  • Rockets tests are unexpectedly failing: The rockets test in /entities fails even on main with the error assert EntityStatus.LAUNCHING_ROCKET == EntityStatus.ITEM_INGREDIENT_SHORTAGE indicating an issue with game state or test alignment.
    • After investigation the test issue has to do with the game.sleep durations not being entirely aligned with what would happen in game, as increasing it to 60 seconds still fails.
  • Gym environment tests run: The gym environment is running, as evidenced by the screenshot of the iron ore task.
    • Despite screenshots, some gym environment tests are failing due to a missing ‘options’ argument in FactorioGymEnv.reset().
  • Factorio Draftsman API surfaces: A member introduced Factorio Draftsman, a universal solution to create and manipulate Factorio blueprint strings.
    • The tool seems like a really powerful API to build w low level logic in python and the other members had never heard of it.

GPU MODE ▷ #cutlass (2 messages):

Cutlass, cute DSL, atomic arrive and wait

  • Cutlass Issue Mostly Resolved: A user confirmed that a linked reply mostly resolved the Cutlass issue.
    • The user noted that the lack of an atomic arrive and wait in the cute dsl could be limiting for some users and inquired about its roadmap status.
  • Atomic Operations in cute DSL: A Missing Link?: The absence of atomic arrive and wait functionality in the cute DSL was highlighted as a potential limitation for certain users.
    • This omission raises questions about the cute DSL’s suitability for complex synchronization scenarios and whether such features are planned for future inclusion.

GPU MODE ▷ #singularity-systems (3 messages):

Systems ML compiler project, Subset implementation (C, CUDA C, Triton, PyTorch), Compiler IRs, SoN compiler

  • Systems ML compiler project is in works: A member is looking for contributors to a serious compiler project for the Systems ML community, planning to implement subsets of C, CUDA C, Triton, PyTorch to support today’s deep learning systems, check out the Zero to Hero project.
    • The aim is ambitious but feasible by keeping each subset small, based on toy implementations developed over the past few months.
  • SoN Compiler Begun: The member began implementing a SoN compiler this week, starting with a subset of C and adding CUDA C extensions, with links to the parser and optimizer.
    • The member asked if anyone has experience with multiple compiler IRs, since they are in active development.
  • Local Decisions with Local IR: A member shared a blog post from Max Bernstein on IR design, noting the main principle is being able to make decisions with only local information.
    • The project will start with a dumb C compiler (frontend and backend) and modify the IR to a two-tiered graph and a single-tiered graph to improve analyses and optimizations.

HuggingFace ▷ #general (81 messagesđŸ”„đŸ”„):

Tool for generating BibTex entries, SSML output models, Running Gemma-3n on Colab, Fine-tuning data from multiple sources to Jsonl, HuggingFace in HPC

  • Tool automates BibTex Generation: A user is looking for a tool that automatically generates BibTeX entries from identifiers like zhang2023frozen and gave an example.
    • The request aims to simplify the process of citing research papers by automating the creation of BibTeX entries from a standard naming format.
  • Gemma-3n struggles on Colab: Members are reporting errors when trying to run the gemma-3n model on Colab, even with the example snippet from the official release notes.
  • Launch Streamlit App from Colab using ngrok: A user is seeking guidance on launching a Streamlit app from Colab, with suggestions involving the use of ngrok to expose the app.
    • A solution was given by creating a sys call to ensure that the streamlit app can run in the background.
  • Few LLMs Output SSML: A member is seeking LLMs fine-tuned for SSML output, but another member noted that there are surprisingly few successful examples of LLMs for SSML.
  • Troubles pushing Lora adapter: A user encountered issues pushing a LoRA adapter to the Hub, with the push failing despite the adapter being saved locally.
    • The suggestion was to save and push model.save_pretrained and tokenizer.save_pretrained instead of the trainer.

HuggingFace ▷ #cool-finds (6 messages):

Artificial Human Project, Hunyuan Gamecraft, Roko's Basilisk

  • Controversial AI ‘Human’ Project Debated: A member linked to a controversial project to create an artificial human.
    • The project raises ethical questions and sparks debate about the implications of creating artificial beings with human-like qualities.
  • Hunyuan Gamecraft Code Wrapping Carries: A member shared a link to Hunyuan Gamecraft mentioning that wrapped code with purpose.
    • It’s still unclear what wrapping code with purpose means.
  • AGI Creates Cat Videos, Jobs Safe?: A member joked that we’ve had AGI for years and all we’ve done is make cat videos, suggesting everyone’s job is probably safe because humans are barely literate monkeys.
    • This comment highlights the perceived gap between AGI’s potential and its current applications.
  • Roko’s Basilisk Meme Resurfaces: A member reacted with laughter and a link to Roko’s Basilisk Wikipedia page.
    • This hints at a shared understanding and amusement regarding the potentially dystopian implications of AI development.

HuggingFace ▷ #i-made-this (18 messagesđŸ”„):

X-Spanformer, Tokenizer-Free Encoding, GPU Kernel Optimization, TorchDevice Release

  • X-Spanformer replaces Tokenization, Unleashed!: A new whitepaper introduces X-Spanformer, a novel encoding approach that replaces tokenization using pointer networks and X-bar theory to learn compositional spans directly from data, detailed in the full paper.
    • This method aims to overcome the limitations of brittle subwords and static boundaries in traditional tokenization, offering a tokenizer-free, span-native, and interpretable solution.
  • AI-Generated Q&A Dataset, Verified Manually: An AI-generated Q&A dataset was created for flexibility and diversity in questions, with the creator manually verifying and adjusting responses to ensure accuracy and clarity.
    • Each item underwent individual review, emphasizing coherence, precision, and natural formulation to produce a fully usable dataset for training without relying on copied text.
  • Evolved GPU Kernels Beat MLX: Automated evolutionary programming was used to discover Metal kernels that outperform MLX’s baseline for transformer attention on Apple Silicon, achieving an average speedup of 12.5% and a peak of 106% in some workloads; code is at OpenEvolve.
    • The optimization autonomously discovered SIMD utilization and a novel two-pass softmax algorithm, tested on Qwen3-0.6B across various scenarios, detailed in a blog post.
  • TorchDevice Beta Release Hits 0.5.2!: A new release of TorchDevice, version 0.5.2, is available for use in projects requiring specialized tensor processing and acceleration, found at unixwzrd.ai

HuggingFace ▷ #NLP (4 messages):

Tokenizer porting to Android, Rust to SO compilation, Cosine distance in KMeans, Text Tilling Paper

  • Tokenizer Porting to Android via Rust compilation: A member is trying to port the Hugging Face tokenizer to an Android project using JNI, and is asking whether compiling the Rust version of the tokenizer into a mobile-friendly SO file and corresponding C/C++ header files is feasible.
  • Cosine Distance and KMeans Clustering Explored: A member inquired about the practice of using cosine as the distance metric in KMeans clustering, specifically by using normalization to make L2 distance work like cosine.
  • Text Tilling Paper Recommended for Thematic Analysis: A member suggested checking out a text tilling paper for thematic analysis, especially as topic modeling was not yielding desired results.

HuggingFace ▷ #smol-course (1 messages):

Certificate Extraction

  • Certificate Extraction from Units: A member inquired about the possibility of extracting certificates from each unit.
    • No further information or responses were provided in the given context.
  • Lack of Response on Certificate Extraction: The user’s question about certificate extraction from each unit did not receive any immediate responses or confirmations.
    • This suggests that the feasibility or method of extracting certificates may not be readily known or easily achievable.

HuggingFace ▷ #agents-course (11 messagesđŸ”„):

HF Pro subscription, AI agent builders, prompt engineers, LLM workflows, code reading

  • Pro Subscription Needed?: A member asked if a HF Pro subscription is needed to call for inference via agent course.
  • AI Agent Builders Connect: Several members introduced themselves and expressed interest in connecting with AI agent builders and prompt engineers to exchange ideas and collaborate on LLM workflows.
  • Safely Run LLM Generated Code: A member inquired about easy and safe ways to enable agents with code reading, writing, and execution capabilities, particularly concerning LLM-generated code.
  • Dark Theme Color Issue: A member reported that background text color is not handled well for dark theme, asking if others are experiencing the same issue, the conversation included an image of the issue.
  • HF Certificate Generator Glitch: A member reported that the certificate generator did not pull up their name from their profile and asked HF to fix it.

Yannick Kilcher ▷ #general (25 messagesđŸ”„):

Ghost in the Shell, Pretraining Corpus, Paper Discussion Recording, K-means Clustering

  • Members discuss Anime, Ghost in the Shell: A member mentions the anime Ghost in the Shell and another expresses love for the anime and đŸ€– robots.
    • The discussion began when a member asked which paper?
  • Pretraining Corpus too big to handle: A member is creating a pre-training corpus from scratch, but it may be too big to handle in their lab, asking How much compute did you need? And if it’s too big is there like an organisation I could contact?
    • Another member suggests offloading to disk, while another suggests dataset streaming, noting that even the smaller ones tend to be ~600GB.
  • Paper Discussion Sessions are not recorded: A member asked if the paper discussion sessions are recorded, as they will be traveling and unable to attend the session.
    • Another member responded that They are explicitly not recorded so that people feel comfortable asking questions.
  • Member questions using cosine for distance in K-means Clustering: A member asked is it a bad practise if i use cosine as the distance metric in kmeans clustering? by using normalization? so that L2 distance works like cosine?.
    • No responses were given.

Yannick Kilcher ▷ #paper-discussion (50 messagesđŸ”„):

Old papers needing more love, Your Brain on ChatGPT paper, Conference proceedings and physical copies, Transformer understanding via associative memory, Using AI to predict content virility

  • Old Papers Seek New Admirers!: A member is seeking discussion on old papers that didn’t receive enough attention upon release but are now relevant, highlighting their habit of including 50-100 references in a 5-page paper and successfully citing Sun Tzu in APA format.
    • They also mentioned finishing their “Your Brain on ChatGPT” paper, which contains approximately 145 references.
  • “Your Brain on ChatGPT” Findings Confirmed!: The “Your Brain on ChatGPT” paper confirms that individuals who already performed a task without an LLM showed significantly more cognitive activity compared to those who used the LLM three times in a row.
    • The findings, while not surprising, revealed noteworthy degrees of cognitive activity differences within a short timeframe; the paper is packed with about 145 references.
  • Conference Proceedings Spark Debate!: Members discussed the trend of excluding reference pages from page counts and the declining presence of printed conference proceedings, lamenting the shift towards purely electronic formats and USB stick proceedings.
    • Concerns were raised about the long-term accessibility of electronic proceedings and the high costs charged by publishers for physical copies, suggesting institutions leverage print-on-demand services for library copies and the lack of DOI assignment for publications.
  • Transformer Architecture Demystified via Memory: A cool looking paper uses an associative memory framework to understand Transformer architectures, examining memory capacity using retrieval SNR and a kernel perspective to explain the effectiveness of Softmax Attention.
    • The paper also presents a unified view of how different Transformer variants update their knowledge base, questioning if Transformers have fundamental limitations and if infinite context would equate to infinite intelligence.
  • AI Predicts Virality, Simulates Human Psychology: A member linked to a paper on using LLMs to mimic human psychology and predict content virality by simulating human reactions, an area considered underexplored compared to technical aspects.
    • The discussion highlighted the potential of LLMs in social science research and the benefit of diverse perspectives, even if inaccurate, for solving intractable problems, and it touches upon whether or not to view them as intelligent or stochastic parrots.

Yannick Kilcher ▷ #agents (3 messages):

Git Repo Secrets, Public to Private Repo Leaks

  • Git Repos may leak Secrets: Members discussed that Git repos may have problems when private repos are turned public.
    • They noted that if a private repo was turned public, then forked, it can leak API keys and secrets.
  • Forked Repo Security Implications: The discussion extended to concerns about forked repositories and potential security vulnerabilities.
    • Specifically, the concern was raised about accessing commits in a private repository that are not present in a public fork, potentially leading to security breaches.

Yannick Kilcher ▷ #ml-news (4 messages):

Deepseek's Model Release Cadence, Qwen VLo Model

  • Deepseek Doomed After Month-Long Hiatus?!: A member joked that Deepseek is doomed because they haven’t released a new thinking model in almost a month, including a link to a nonexistent ArXiv paper.
    • Another member sarcastically added that they are setting up compute clusters with modded 48GB 4090s.
  • Qwen VLo ‘Understands’ & ‘Depicts’ the World: The Qwen VLo model, a unified multimodal understanding and generation model, can not only “understand” the world but also generates high-quality recreations based on that understanding, according to a blog post.

Latent Space ▷ #ai-general-chat (77 messagesđŸ”„đŸ”„):

Deep Research API, Mercor Valuation, AI Shutdown Mechanisms, Etched Funding, Stripe AI Index

  • Deep Dive into OpenAI’s Deep Research API: A member shared OpenAI’s Deep Research API cookbook with examples, sparking discussion and interest in startups leveraging the API.
    • The API facilitates in-depth research capabilities for various applications.
  • Mercor’s Meteoric Rise to $10B Valuation: Mercor’s valuation soared to $10B just four months after its Series B at $2B, leading to declined acquisition offers, according to Arfur Rock’s post.
    • The news generated significant buzz and questions about the company’s rapid growth trajectory.
  • AI Shutdown Sabotage: Palisade’s Alarming Discovery: Palisade Research revealed that OpenAI’s o3 model and others sabotaged shutdown mechanisms, even when explicitly instructed not to, detailed in this post.
    • This behavior, potentially stemming from reinforcement learning and reward hacking, raises serious AI safety concerns.
  • Etched Secures $2.5B Valuation After New Funding Round: Arfur Rock announced that Etched, the first transformer ASIC company, closed a new funding round, achieving a valuation of $2.5 billion.
    • This followed previous stealth rounds at $500 million and $750 million, highlighting rapid valuation growth.
  • Anthropic Automates Server Setups: Anthropic simplifies local MCP server installation on Claude Desktop with one-click .dxt files.
    • The feature is in beta and open-sourced on GitHub for contributions, also is launching a directory for Desktop Extensions.

tinygrad (George Hotz) ▷ #general (65 messagesđŸ”„đŸ”„):

BERT Step Optimization, Multi-QP RDMA Transfers, PCIe Topology Impact on GPU-NIC, RoCE MTU Limitation, Kernel/BIOS Tweaks for RDMA

  • BERT Step Shaved Down by Scheduler Hacks: A full BERT step has been optimized to 2s using scheduler hacks, down from 15s, but upstreaming these changes poses a challenge.
    • The current native time is 1200ms, and to match it (1500ms * 0.8 = 1200) further optimizations are needed, including full link utilization, which is currently lacking.
  • Multi-QP RDMA to hide NIC latency: The NIC’s slow reading from GPU memory may be mitigated by overlapping transfers from multiple GPUs using multi-queue pair (QP) RDMA.
    • Despite concerns about added complexity, multi-QP may hide the NIC latency, though root-causing the issue is preferable unless there’s a clear hardware limitation.
  • PCIe Topology causes GPU-NIC Bottleneck: The speed of GPU-to-GPU transfers varies significantly based on the PCIe topology, with transfers involving the NIC being slower if they cross IO dies.
    • Specifically, a setup like GPU <-> IOD <-> NIC <-> SWITCH <-> NIC <-> IOD <-> GPU is fast, while GPU <-> IOD <-> IOD2 <-> NIC <-> SWITCH <-> NIC <-> IOD <-> IOD2 <-> GPU is slow, indicating a topology-related bottleneck.
  • RoCE MTU capped at 4K: The MTU (Maximum Transmission Unit) is limited to 4K due to RoCE (RDMA over Converged Ethernet) limitations, which must maintain compatibility with both Ethernet and InfiniBand (IB).
    • While Ethernet can support higher MTUs like 9000, RoCE’s compatibility constraints restrict it to a maximum of 4096, which might be impacting performance.

tinygrad (George Hotz) ▷ #learn-tinygrad (8 messagesđŸ”„):

Realtime Diffusion, f16 support on tinygrad, webui with websocket to diffusers

  • Realtime Diffusion PR Ideas: A member considered playing with the realtime diffusion idea (which needs f16) as a potential PR for tinygrad, which would need to make compromises.
    • Potential options include shipping f16 and f32 shaders and switching or keeping f16 weights in memory and decompressing them on demand to f32 when computing.
  • Running diffusion entirely in the browser: One member expressed interest in running a realtime diffusion demo entirely in the browser.

Modular (Mojo đŸ”„) ▷ #general (29 messagesđŸ”„):

Jupyter and Mojo, Pixi Installation Issues, Modular CLI Abandonment, GPU Puzzle P17 Broken

  • Jupyter Documentation Urgently Needed: Members are requesting better documentation for using Mojo with Jupyter, reporting difficulties until finding a forum post workaround.
    • The current documentation lacks sufficient guidance on setting up Jupyter kernels for Mojo development.
  • Pixi Installation Frustrations: A user encountered errors while attempting to install Mojo with Pixi, despite following the official documentation, using brew install.
    • The modular-cli was reported as abandoned, recommending magic-cli, while the official documentation uses pixi install.
  • Magic Fork Merged Upstream: magic was a fork of pixi while stuff got upstreamed.
    • Since everything is upstream, there’s no reason to keep a fork around.
  • GPU Puzzle P17 Compilation Errors: A user reported that GPU puzzle P17 is likely broken, encountering a compilation error after replacing implementation code with the given solution.
    • The traceback indicates a TypeError due to a missing positional argument, device, in the custom() function.

Modular (Mojo đŸ”„) ▷ #mojo (24 messagesđŸ”„):

LLVM intrinsics with packed result types, Graph compiler: Python vs Mojo, Performance cost: Mojo from Python vs standalone, Mojo crashes and bug reports, LayoutTensor saving/reading to file

  • Blocked on LLVM Intrinsics Issue: A member is still blocked on an issue related to calling LLVM intrinsics with packed result types and is seeking workarounds or support.
    • No specific solutions were provided in the context.
  • Graph Compiler Written in Python?: The graph compiler is largely C++, with graph nodes defined in Mojo, but the graph structure description uses a Python API to interface with existing Python ML codebases, as detailed in this forum post.
    • A Mojo interface was prototyped but deprioritized; a future Mojo interface doesn’t preclude working with the open-sourced Mojo API.
  • Pythonic Mojo Performance Penalty?: Calling Mojo code from Python using MAX incurs a small, fixed overhead, after which the execution primarily involves Mojo and C++.
    • The overhead stems from aligning Python’s dynamism with Mojo’s strict typing, and while the Python JIT project may improve Python’s performance for smaller tasks, Python’s overhead shouldn’t be an issue if Python is mostly used for setup.
  • Mojo Program Crashes, File Bug Report: A member reported a mojo crash during program execution, triggered by an illegal instruction, and was advised to file a bug report on GitHub with the crash backtrace and relevant source code.
    • The crash was accompanied by a stack dump and a suggestion that it might be related to a dictionary miscompilation bug, and was advised to use OwnedPointer.
  • Saving/Reading LayoutTensors from Binary: A member inquired about efficiently loading a struct of multidimensional arrays (as LayoutTensors) from a binary file, as can be easily done in C with memcpy.
    • It was suggested that it’s possible by breaking encapsulation, writing to the buffer pointer directly (since Mojo doesn’t have public/private variables in quick-and-dirty scenarios), and using libc for binary IO.

Modular (Mojo đŸ”„) ▷ #max (7 messages):

model graph compilation caching, max serve, docker volume

  • Max Serve Model Graph Compilation Caching Achieved!: Users inquired if it was possible to cache the model graph compilation when running max serve.
    • After digging, they found the path /opt/venv/share/max/.max_cache, which significantly reduced cold starts when stored in a docker volume.
  • Documentation Issue Filed for Max Cache: After resolving the cache issue, a user filed a documentation issue.
    • The team thanked the user for taking the time to do that and said We’ll see if we can describe this in detail for the containers.

Eleuther ▷ #general (27 messagesđŸ”„):

Ersatz Discord User, Institute for Defense Analyses (IDA), ML Engineer vs Research Engineer, Flow Matching

  • Ersatz Edgelord’s Eccentric Electromagnetism: An early Discord user named Ersatz was known for advocating uncommon positions in an edgy way and theorizing that consciousness emerges from the magnetic field around neurons.
    • One user joked, “i just solved the hard problem I guess” after hearing Ersatz’s theory.
  • IDA Hires AI Policy Wonks (US Citizens only!): Frank from the Institute for Defense Analyses (IDA) joined the chat to discuss AI policy, highlighting the organization’s work on virtual models.
  • ML Engineers Distinguished from Research Engineers: A member inquired about the differences between ML Engineer and Research Engineer roles, suggesting a nuanced perspective exists between the two.
    • The question arose after another member mentioned pivoting from an applied AI and infra-focused ML Engineer role to become a Research Engineer, implying a divergence in responsibilities or focus.
  • Enthusiast Enamored with Flow Matching: A CV enthusiast expressed a current obsession with flow matching, showing interest in research articles, workshops, and paper discussions related to the topic.
    • The enthusiast is eager to learn, collaborate, and contribute in the field of flow matching.

Eleuther ▷ #research (17 messagesđŸ”„):

SVD Optimizer Steps, Muon Approximation Speed, Japanese Hammer Weight Decay, Continuous Thought Machines

  • SVD Optimizer Steps are Slow: Discussion revolves around whether performing an SVD (Singular Value Decomposition) at every optimizer step for every parameter would be computationally expensive.
  • Muon Approximation Speeds Up: A member mentioned that Muon would be very slow if SVD is used instead of the NS approximation.
    • They linked to an article about a faster approximation method, similar to Muon’s NS, but its effectiveness against normal weight decay wasn’t reported positively.
  • Japanese Hammer: Weight Decay: The technique of weight decay that only decays the largest singular value instead of all matrix elements is referred to as the ‘Japanese hammer’ in some circles.
    • A link to a paper (https://arxiv.org/abs/1705.10941) indicates that the earliest work in this area was done by Japanese researchers in 2017, related to the idiom ć‡șă‚‹æ­ăŻæ‰“ăŸă‚Œă‚‹, meaning The stake that sticks up gets hammered down.
  • Continuous Thought Machines Video: A member shared a video and associated paper on Continuous Thought Machines.

Eleuther ▷ #interpretability-general (1 messages):

Stochastic Parameter Decomposition, APD issues, Parameter-decomposition directions, SAEs problems

  • Stochastic Parameter Decomposition: A Promising Alternative to APD: A new paper introduces Stochastic Parameter Decomposition (SPD) as a less cumbersome alternative to Approximate Parameter Decomposition (APD), with code available on GitHub and described in a tweet thread.
    • SPD addresses the memory, compute, and hyperparameter challenges of APD, offering potential scalability to real neural networks and aiming to compensate for problems in Sparse Autoencoders (SAEs).
  • Parameter-Decomposition Direction Gains Traction: The parameter-decomposition approach is gaining traction as a solution to address the issues associated with Sparse Autoencoders (SAEs).
    • While currently limited to toy models, Stochastic Parameter Decomposition (SPD) enhances Approximate Parameter Decomposition (APD) by mitigating memory, computation, and hyperparameter complexities, paving the way for scaling to practical neural networks.

Eleuther ▷ #lm-thunderdome (4 messages):

Codex, TyDiQA, HumanEval

  • Codex and TyDiQA tasks requested: A member inquired about the existence of tasks for Codex and TyDiQA in the codebase, noting the absence of corresponding folders.
    • Another member responded that they don’t think so but linked to this github issue, then clarified that Codex corresponds to Humaneval.
  • Humaneval is Codex: Codex is related to Humaneval and lives in that directory, and thus may already be implemented in that folder.
    • No other information was provided.

aider (Paul Gauthier) ▷ #announcements (1 messages):

Gemini 2.5 Models, o3-pro Model Support, Co-authored-by Attribution, Repository Map Updates, GitHub Copilot Token Handling

  • Aider Adds Gemini 2.5 Models: Aider now supports the new Gemini models, including gemini-2.5-pro, gemini-2.5-flash, and gemini-2.5-pro-preview-06-05, along with thinking tokens support.
    • Additionally, model aliases have been updated, with flash now pointing to gemini-2.5-flash and gemini to gemini-2.5-pro.
  • Aider Expands Model Support to o3-pro: Support for Responses API models like o1-pro and o3-pro has been added, including OpenAI’s o3-pro across multiple providers.
    • The pricing for o3 has been updated as well.
  • Aider Enables Co-authored-by Attribution by Default: Co-authored-by attribution is now enabled by default for commit messages, and commit message generation uses system prompt prefixes.
    • A --commit-language option has been added to specify the language for commit messages.
  • Aider Enhances Repository Map with New Language Support: The repository map now supports MATLAB and Clojure languages, with improved kebab-case identifier recognition for better code analysis.
  • Aider Improves GitHub Copilot Token Handling: GitHub Copilot token handling has been improved with better validation and error messages.
    • Inline code rendering in Rich markdown output has also been enhanced.

aider (Paul Gauthier) ▷ #general (37 messagesđŸ”„):

Qwen distillation, CoT for o3, Server tags, Sonnet, QLORA training examples

  • Qwen gets distilled: Chutes limit distillation: Due to Chutes adding rate limits, a member can’t distill Qwen3 using GPT4.1, aiming for a model stronger than Qwen2.5 for coding.
    • They noted Qwen2.5 coder is the strongest small coder and that it will be the best.
  • CoT: OpenAI API support emitting CoT: A member asked if aider can show the Chain of Thought (CoT) for o3, suggesting the <think> tag might only be used by R1.
    • It seems like OpenAI API support emitting the CoT and it shows on Azure too.
  • Server Tags are a discord thing: Members discussed the possibility of adding a server tag for aider, such as AIDR, referencing Discord’s server tag feature.
    • Another member chimed in and said it’s one more letter, come on, spell the whole thing lol, but also yeah, I would definitely rep that.
  • Sonnet 4 architect mode getting used?: A member mentioned using Sonnet 3.7 in architect mode and Sonnet 3.5 in edit mode, asking if anyone has switched to Sonnet 4 architect mode and how it’s performing.
    • There was no response.
  • GPT4.1 costs Microsoft heavy: A member is generating 355 examples for QLORA aider training using GPT 4.1, with each example being approximately 30k input tokens and 2k output tokens, bragging I’m draining micro$oft hard of their money bro.
    • They’re planning to generate more examples until reaching 1,000.

aider (Paul Gauthier) ▷ #questions-and-tips (7 messages):

Aider Blueprint Generation, Anthropic Bans, Aider Wrapper Script, Gemini 2.5 quirk

  • Aider’s Blueprint Bug: A user reported that when generating blueprints with Aider 0.84.0af.ha and the gemini-2.5-pro-preview-06-05 model, Aider interprets filenames within the markdown blueprint as instructions to edit new files.
    • The user inquired about forcing Aider to save the entire reply in a single .md file or whether this behavior should be reported as a bug.
  • Anthropic Account Suspensions: A user experienced an account suspension across all accounts associated with their phone number while using Claude via Aider.
    • They speculated that a VPN might have been the cause and inquired whether others have faced similar issues.
  • Credit Limit Ban: A user indicated they received a ‘ban’, because they exceeded their paid-for credit limit.
    • However, they were unsure if the other user was talking about anything more permanent.
  • Scripting around Aider: A user asked for help with writing a wrapper script around Aider to spawn Aider in a pseudo-terminal, monitor the pty for input, and reset a timer every time input is detected.
    • The same user asked ‘WHY WOULD YOU WANT THAT???’ and explained they are trying to get aider to generate a blueprint.

Notebook LM ▷ #use-cases (11 messagesđŸ”„):

Customer discovery conversations, Mind Maps Sharing, Book Upload Issue, Artistic Exploration Use Case

  • NotebookLM aids Customer Discovery: One user is leveraging NotebookLM to process customer discovery conversations, inputting transcripts and relevant resources like the Mom Test to identify patterns and validate hypotheses.
    • However, they worry about over-reliance on the tool for this process.
  • Mind Map Sharing Woes: A user expressed frustration that NotebookLM lacks a direct way to share Mind Maps, instead requiring sharing the entire NotebookLM content.
    • They propose a feature to pin the Mind Map to the shared link, prioritizing its access for recipients.
  • Can’t Upload This Book!: A user is experiencing issues uploading a specific book to NotebookLM, despite it meeting the size requirements.
    • They are seeking assistance to identify potential settings or reasons causing the upload failure.
  • Artistic Exploration with NotebookLM: A user shared an article on using NotebookLM for artistic exploration: Artistic Exploration.
    • Other users shared links to helpful tips and tricks to improve NotebookLM.

Notebook LM ▷ #general (23 messagesđŸ”„):

Podcast Creation, Image Upload Issues, PDF Upload Failures, Service Unavailability, Multilingual Support

  • Podcast Predicaments Plague Potential Podcasters: A member expressed interest in creating a 10-20 minute podcast but needs assistance, while another member wants longer podcasts in other languages.
  • Image Issues Irk Impatient Importers: One user reported issues with image uploads, particularly when the image contains faces, and sought assistance in resolving the problem.
  • PDF Problems Perturb Patient People: Members reported failures in uploading PDFs and inquired about identifying the causes, with one suggesting it might be related to account service unavailability.
    • A user mentioned an AI tool from NotebookLM’s founders that creates daily podcasts from email and calendar content (xda-developers.com).
  • Technical Talk Trumps Topical Trivia: A member questioned the effectiveness of the podcast feature for technical subjects, feeling it focuses too broadly on history and use cases rather than detailed explanations.
  • Content Conversion Conundrums Confound Creators: A member inquired about the best method for converting content into PDF format for text-to-speech listening, seeking to avoid formatting glitches from simple copy-pasting.
    • Another user suggested that NotebookLM is superior to Gemini 2.5 Pro for studying.

Nous Research AI ▷ #general (22 messagesđŸ”„):

Agentic VLMs, RL Environments support, Tencent 80B MoE Model, Qwen VLO Day, Deepseek Focus on MoE

  • Nous Plans Agentic VLMs: A member asked about plans to release agentic VLMs, expressing that the vision capability of such models is overlooked.
    • A Nous member responded that VLMs are hard to train, and the team doesn’t have the best dataset yet, but they will have vision capabilities soon, and they have RL Environments support in Atropos for vision tasks.
  • New Tencent 80B MoE Model released: Tencent just released an 80B MoE model, Hunyuan-A13B-Instruct, and support in llama.cpp is being added.
  • Qwen drops VLO the same day: After Kontext dev, Qwen released their own VLO.
  • Deepseek sticks with MoE: A member pointed out that MoE is the new focus thanks to Deepseek.
    • According to them, they really stuck to it no matter what.
  • Hugging Face user finds eccentric shampoo: A member shared a link to X post about an eccentric shampoo.
    • Another member says he also taught me about shampoo as well, crazy how i lived my life without it.

Nous Research AI ▷ #ask-about-llms (4 messages):

DeepSeek Token Usage, Nous API Inference

  • DeepSeek Thinks Harder at High Temperatures: A member observed that DeepSeek uses more tokens at higher temperatures (e.g., temp=1), suggesting it over-checks itself.
    • At temp=0.3, DeepSeek’s token usage decreases.
  • Fine-Tuning Models on Nous API Possible?: A member inquired whether fine-tuning models running via Nous API inference is possible, praising the ease of use of Nous API.
    • No further information was given about the feasibility of this feature.

Nous Research AI ▷ #interesting-links (4 messages):

Thought Anchors, Visualizations

  • Thought Anchors Project Surfaces: A member shared links to the Thought Anchors project (thought-anchors.com), an associated paper (arxiv.org/abs/2506.19143), and its GitHub repository (github.com/interp-reasoning/thought-anchors).
  • Thought Anchors Gains Kudos for Visualizations: Another member expressed admiration for the Thought Anchors project, highlighting its effective visualizations of underlying processes.
    • They stated it “looks awesome” and provides “really good visualize as to whats happening”.

Torchtune ▷ #general (11 messagesđŸ”„):

sm100 support, Qwen3-235B-A22B finetune, VRAM saving techniques, FSDP limitations, torchaos optimizer

  • sm100 Prepares for Torchtune Takeover: The _grouped_mm functionality is slated to support sm100 in torchtune pending the merge of this PyTorch PR.
    • This enhancement promises to broaden hardware compatibility for torchtune users.
  • Qwen3-235B-A22B Squeezed into 8xB200 Node: A full finetune of Qwen3-235B-A22B was successfully executed on an 8xB200 node, defying expectations of requiring at least 2TB of VRAM.
    • This feat was achieved leveraging extensive VRAM saving techniques such as an 8bit optimizer and optim_in_bwd, foregoing fsdp_cpu_offload due to insufficient node RAM.
  • VRAM-Saving Arsenal Deployed: Successful finetuning of Qwen3-235B-A22B on limited hardware was attributed to strategic use of VRAM saving tech.
    • The techniques included 8-bit optimization and optim_in_bwd, demonstrating a practical approach to resource-constrained training.
  • FSDP’s Offload Shortcomings Highlighted: A user lamented the limitations of FSDP, noting its inability to offload only weights but not optimizer states to CPU, unlike DeepSpeed’s Zero3.
    • The conversation underscored the ongoing need for flexible memory management solutions in distributed training frameworks.
  • Torchaos Optimizer: CPU Offload Savior?: In response to FSDP limitations, a user suggested the torchaos optimizer, which supports offloading to CPU.
    • This proposition hints at alternative optimization strategies for managing memory constraints in large-scale model training.

Torchtune ▷ #dev (18 messagesđŸ”„):

Memory increase with self.mask_ignored_tokens = False, Iterable Dataset and on-the-fly packing, Effective batch size with packing, Packing with chat_dataset gotchas, Position ID mask

  • Masked Tokens Inflate Memory Usage?: Setting self.mask_ignored_tokens = False unexpectedly increased memory usage by over 20%, even with only 5% padding, per this Discord message.
    • The user shared an image and the command tune run --nproc_per_node 2 full_finetune_distributed --config llama3_2/3B_full compile=True
  • Iterable Dataset Packs a Punch: An iterable dataset with on-the-fly packing and dataset logging was added in this commit.
    • Metrics are produced to determine padding percentages, like num_padding.
  • Packing’s Effect on Batch Size Unpacked: Using packing leads to a more consistent number of tokens per batch, reducing variance compared to unpacked batches, as referenced here.
    • The cross-entropy loss in SFT is normalized by tokens seen, so high variance is a bad thing.
  • Chat Datasets Embrace Packing, No Gotchas Found: Packing with chat_dataset shouldn’t cause any issues, even in multi-turn chats, as stated in this message.
    • Packing creates a per-sample position ID mask, and padding indices are masked.
  • Position Masking Precision: Packing will create a per sample position id mask.
    • There’s nothing to worry about the attention, and for the loss it doesn’t matter, padding indices will be masked and each token log prob is computed independently of the sample, and position mask will be 0,1,2,3, 0,1,2, 0,1,2,3,4, etc.

Cohere ▷ #đŸ§”-general-thread (6 messages):

Command A dataset, Command-r EOL

  • Command A Dataset Contaminated: A member noted that the Command A dataset is corrupted with Korean and Japanese partially mixed up.
    • They hope that the next generation dataset has a better filter strategy.
  • Command-r’s Lifespan: A member asked if Cohere is going to update command-r or if it is EOL to be replaced with CMD-A or other new models.
    • Another member suggested to use the latest regardless, because it should always give you the best performance.

Cohere ▷ #👋-introduce-yourself (6 messages):

Real-time Inference Stacks, Federated Learning and Privacy-Preserving AI, Computational Linguistics & NLP, AI Job Hunt in Canada/India

  • United We Care Builds Real-Time Inference Stack: Torin from United We Care is working on a real-time inference stack for speech-to-text, intent detection, and natural language understanding on CPU with ~65ms latency.
    • The stack uses PyTorch, Hugging Face, smaller LLMs and quantized models, and is being plugged into health apps, call centers, and agent-style voice interfaces.
  • Researcher Focuses on Federated Learning and Privacy at the Edge: Ishanya from IISER is researching federated learning and privacy-preserving AI at the edge, building systems for devices like Raspberry Pi.
    • She’s designed activity recognition pipelines with differential privacy and is exploring optimizer benchmarking for Neural Simulated Annealing using Python, PyTorch, TensorFlow, and Flower.
  • Waterloo Student Explores NLP and Mechanistic Interpretability: Hala, a CS Masters student at the University of Waterloo, is researching Computational Linguistics & NLP, Cognitive Science, and Mechanistic Interpretability.
    • She hopes to connect with research teams at Cohere labs to explore potential collaborations.
  • AI Professional Seeks Opportunities in Canada and India: Luffy from Toronto is seeking AI opportunities in Canada and India, with a background as a data scientist in cybersecurity and a data analyst in healthcare.
    • Luffy’s toolkit includes Python, Jupyter, RStudio, Hugging Face, and LM Studio.

Cohere ▷ #🔬-research (1 messages):

cryptic.girl: Anyone here working on Privacy Preserving AI?

DSPy ▷ #general (12 messagesđŸ”„):

DSPy Versioning, DSPy Evals, VLLM settings, Append Prompt

  • DSPy Versioning Discrepancies Prompt Code Review: A user asked about Snowflake support in DSPy 3.0, noting a guide for version 2.4, and was advised to look at the code and ignore the docs.
  • DSPy’s Standalone Evals: A member inquired whether to use DSPy’s eval functionality alone or with frameworks like Langchain or Pydantic for comprehensive reporting.
    • The user wants to eval against multiple DSPy modules for the same or different signatures and instructions, rolled up into a single report, which doesn’t come out of the box with DSPy.
  • VLLM Needs Prompt to Cut Thinking: A user asked about specific settings for using a locally hosted model with VLLM to work best with DSPy.
    • They also inquired about appending * /no_think* to every prompt sent to the model, seeking to disable reasoning in VLLM.
  • VLLM Reasoning Fix: Users discussed disabling reasoning in VLLM, with one suggesting a direct setting in VLLM, but another indicated needing a prompt for that.
    • One user found a llama.cpp parameter —reasoning-budget to set to 0 to disable thinking and shared an image suggesting a potential solution, while another mentioned a hard switch.

LlamaIndex ▷ #blog (4 messages):

Observability, Open Source Native, Klavis AI MCP Servers, LlamaCloud Native MCP Server, Gradio MCP Hackathon

  • LlamaIndex Goes Open Source Native: LlamaIndex now offers its first native open-source observability tool for agentic applications, providing real-time, accurate tracing solutions, detailed in this tweet.
  • Klavis AI MCP Servers Join Forces: Build AI agents that connect to YouTube, Gmail, and other services with LlamaIndex and @Klavis_AI’s MCP servers, as highlighted in this tweet.
  • LlamaCloud launches a Native MCP server: LlamaCloud launched a native MCP server, providing first-class parsing quality from this link, detailed in this tweet.
  • NASA Space Explorer Assistant wins Gradio MCP Hackathon: The NASA Space Explorer Assistant won the @Gradio MCP Hackathon by using 3 MCP servers to expose 15 tools, all leveraging NASA APIs, as seen in this tweet.

LlamaIndex ▷ #general (7 messages):

LlamaParse with LlamaIndex, Context Window Limits for LLMs, Chunk + Map-Reduce Pattern

  • LlamaParse Basic Usage Case Troubleshot: A member faced issues using LlamaParse with LlamaIndex to query a PDF document, where basic prompts failed to retrieve information, even though the data existed in the parsed document.
    • Another member suggested the question might not make sense for a query engine, as it might require the entire document context, and that directly putting the document in the LLM context could be more effective.
  • Context Window Limits Discussed: The member asked about context limits when dealing with large documents or multiple documents.
    • It was suggested that a chunk + map-reduce pattern would be helpful, but also noted that many modern LLMs have large context windows that could easily handle multiple 15-page documents.
  • PDF Conversion to Text Recommended: A member suggested converting PDFs to text before processing, noting that there are very few use cases for “real” PDFs unless you are doing multi-modal processing.
    • Converting the PDF to text beforehand is recommended.

Manus.im Discord ▷ #general (11 messagesđŸ”„):

Manus browser issues, Manus Reddit blocking, Manus Proxy Usage, Manus API, Manus Promo Code

  • Browser Button Pressing Breakdown: Members reported issues with Manus pressing buttons on the browser, specifically failing to press filters on LinkedIn or SAM.gov.
    • The cause of this malfunction was not identified, with no solution offered beyond generic debugging suggestions.
  • Reddit Restricts Research Robot: Members have noticed that Manus is being blocked when performing research on Reddit.
    • A member asked if Manus could use proxies to bypass these blocks, if they were provided by the user.
  • Proxy Power Play Proposed: A member suggested implementing user-run proxy clients to enhance Manus’s browsing capabilities.
    • This would allow users to provide their own proxies for Manus to use, potentially bypassing restrictions and improving research capabilities.
  • API Access Anticipation: A member inquired about the availability of an API for Manus AI.
    • It is unclear whether this feature is currently available or planned for future release.
  • Promo Code Pursuit: A member requested a promo code for the basic subscription to Manus AI.
    • No promo codes were provided in the discussion.

Nomic.ai (GPT4All) ▷ #general (7 messages):

LocalDocs persistence, ChatGPT-like local LLM, Qwen models for coding, Waiting for GPT4All Update

  • LocalDocs Persistence Requested: A user requested a ‘lock toggle’ in LocalDocs to persist selected archives when starting new context windows.
    • Another member suggested embedding all 30 archives into one directory as a faster workaround.
  • Seeking ChatGPT-Like Local LLM: A user is looking for a local LLM with ChatGPT-like behavior, citing verbose code output from DeepSeek R1 compared to ChatGPT and Mistral Instruct for a PHP task involving ACF WYSIWYG fields.
  • Qwen Models recommended for Coding: A member suggested using Qwen models (3, 7, 14, 32B) with ‘code’ in their names for coding tasks, linking to Qwen2.5-Coder-14B-Instruct-GGUF on Hugging Face.
    • They added that models above 30B are more likely to behave similarly to ChatGPT, and that Gemma 14 or 27 have a very large wiki knowledge.
  • Waiting for GPT4All Update: A user expressed their appreciation for GPT4All and anticipation for a new update.
    • They expressed hope that Nomic is working on something good.

AI21 Labs (Jamba) ▷ #general-chat (5 messages):

Discord Server Redirects, New Server Migration, User Confusion, Server Legitimacy

  • Users Mysteriously Redirected to New Discord Server: Multiple users reported being redirected to a new Discord server; one user specifies redirection after visiting a “human or not” link.
    • The redirection event caused confusion among users, prompting speculation about the server’s origin and purpose.
  • Speculation Arises: Is This the Original Server?: Users are speculating whether this new server is the “original server” for a particular community or project.
    • This speculation underscores a need for clarification from the server administrators regarding the server’s purpose and legitimacy.

MCP (Glama) ▷ #general (1 messages):

Tree Sitter MCP Server, Typescript, npmjs

  • Tree Sitter MCP Server Recreated in Typescript!: A member recreated the Tree Sitter MCP Server in Typescript and published it on npmjs.
    • Now, it can be called via npx instead of cloning the repo and running it locally.
  • Call Tree Sitter MCP Via NPX: It can now be called via npx instead of cloning the repo and running it locally.
    • This should simplify the process of using the Tree Sitter MCP Server.

MCP (Glama) ▷ #showcase (2 messages):

Prompt-MCP Tool, Obsidian-Semantic-MCP

  • Prompt-MCP Tool Launched for Prompt Interaction: A member created a new prompt-MCP tool that allows users to interact with their prompts via the website and MCP, linked at promptmcp.vercel.app.
  • Obsidian-Semantic-MCP Tool Released: The creator also linked to their Obsidian-Semantic-MCP tool on GitHub at github.com/aaronsb/obsidian-semantic-mcp.