> AI Discords for 2/11-12/2024. We checked **20** guilds, **312** channels, and **6901** messages for you. Estimated reading time saved (at 200wpm): **589 minutes**.

We have long contended that the RAG style operations have been used for context (knowledge base, facts about the world) and memory (running list of facts about you) will diverge. The leading implementation was MemGPT and now it seems to have rolled out in both ChatGPT (with a weirdly roon-y tweet. more details from Joanne Jang) and LangChain.

OpenAI:

image.png

LangChain:

image.png

In some sense this is just a crossing over of something the LMstudio/Sillytavern roleplay people have had for a while now. Expectation is that it will mildly improve UX but not lead to a big wow moment since the memory modeling is quite crude at the moment, not humanlike, and subject to context limits.

Table of Contents

[TOC]

PART 1: High level Discord summaries

TheBloke Discord Summary

  • Unbounded Textual Contexts: Engineers are exploring new open-source large language models like the Large World Model, which boasts coherence with contexts up to 1 million tokens. Discussions include language support, as in Cohereā€™s aya model, which covers 101 languages, and challenges working with jax-based tools during model installations.

  • Nurturing Erotically Programmed Role Play: The community is dissecting performances of re-quantized Miqu models like MiquMaid-v2-70B, attuned for Erotic Role Play (ERP). Emphasis was on the impact of enhanced hardware, with a jump from 0.7t/s to 2.1t/s in tokens per second while using 12GB VRAM GPUs.

  • Instruct, Optimize, Repeat: Finetuning techniques explained include using Sequential Fine-Tuning (SFT) followed by Direct Preference Optimization (DPO) as improved RLHF/PPO, detailed on page 6 of a paper. Unsloth AIā€™s apply_chat_template is touted over Alpaca to train LLMs for multi-turn conversations.

  • JavaScript Meets Python in AI Development: Experimentation with JSPyBridge led to successful bridging of JavaScript and Python in expanding the SillyTavern project. This included addressing Windows-specific errors, like cpu_embedding=True to circumvent access violation issues and integrating Python classes asynchronously into JavaScript code.

  • Confounding Losses in Model Training: An engineer observed an unexplained variance in training loss when finetuning Mixtral 8x7b qlora, resulting in higher losses compared to Mistral 7b despite similar datasets and hyperparameters. The matter remains open for community input or similar experiences.

LM Studio Discord Summary

  • Large Models Crying for RAM: Users like @nonm3 and @theoverseerjackbright battled errors loading large models in LM Studio due to limited RAM and VRAM. Suggestions included trying smaller model quants, and some faced GPU detection issues with LM Studio, prompting restart crashes.

  • MedAlpaca Heads to the Clinic: Discussions on medical LLMs saw medAlpaca, a fine-tuned model for medical question answering, as a promising addition to @pepito92iā€™s medical project. Microsoftā€™s phi-2 modelā€™s absence from LM Studio was noted, with the possibility of it being converted to .gguf format by user TheBloke to use with llama.cpp.

  • GPU Matchmaking and Overclocking: Hardware enthusiasts like @luxaplexx questioned NVLinkā€™s role in memory cycling, ultimately suspecting cards like the 960 might not be NVLinked. Users discussed GPU upgrades for better performance with models, considering options like the RTX 3060 12GB. Others like @alastair9776 and @rugg0064 weighed the benefits and risks of overclocking for faster token generation.

  • Quant Leap Forward: Eager anticipation for IQ3_XSS support in LM Studio, with users like @n8programs and @yagilb expecting it in the next update. A GitHub pull request reflected the communityā€™s excitement over upcoming 1.5 bit quantization. Meanwhile, preparations were suggested for downloading forthcoming, as-yet-unsupported models like IQ3_XSS.

  • Beta Release Relief: @rafalsebastian ran into a stumbling block running LMstudio on CPUs with only AVX support. @heyitsyorkie provided hope by directing to the 0.2.10 AVX beta release for Windows that enables compatibility, while still recommending an upgrade to AVX2 for optimal performance and offered a helpful link.

  • Multi-Model Management Mystery: @alluring_seahorse_04960 sought advice on running dual models simultaneously on one machine to avoid repetition errors, using a Conda environment but steering clear of VMs. The nature of the repetition errors in question was humorously prodded by @wolfspyre, awaiting further details.

LAION Discord Summary

  • Magvit V2 Sparks Interest and Debate: Engineers delved into the technicalities of reproducing the Magvit V2 model, with discussions focusing on appropriate datasets, parameters for video compression and understanding, and the mention of experiments on the lfq side of Magvit2. The community also saw a surge in interest around MAGVIT, likely due to influencer mentions.

  • Scrutinizing Stable Cascadeā€™s Efficacy: Stability AIā€™s Stable Cascade model spurred intense conversations regarding its high VRAM requirements, optimization issues, and erroneous inference time graphs. Technical issues reported included challenges with text clarity in images and the inability to run models in float16, alongside performance evaluations on GPUs like the 3090.

  • Legal Frays in AI-Generated Imagery: The community engaged in a heated discussion about copyrights and the legality of AI-generated images, highlighting a TorrentFreak article about a court dismissing authorsā€™ copyright infringement claims against OpenAI.

  • Ethical Conundrums with AI and Adult Content: The conversation shifted to the role of adult content in driving technological progress, with some participants recognizing the historical pattern while others doubted its constructive impact on AI. Topics included the rise of non-consensual deepfake pornography, its market dynamics, and the potential ethical pitfalls plaguing the AI community.

  • Calls for Higher AI Image Standards: Discussions included technical insights into improving AI image generation, such as the viability of VAE encoder training. Members also reflected on the communityā€™s photorealism standards, expressing the need for better quality in AI-generated images.

Eleuther Discord Summary

  • Checksum Hunting Season Open: @paganpegasus provided checksums for The Pile zst shards and pointed to EleutherAIā€™s hashes and the Discord pins.

  • Image Content Classification Tools Discussed: OwlViT and CLIP models were recommended as tools for discerning the content of images and the concept of ā€œnothingā€ in imagery was discussed due to an inquiry by @everlasting_gomjabbar.

  • Paper Review in Collaborative Spirit: A user received appreciative feedback on a manuscript titled ā€œDonā€™t think about the paper,ā€ with the EleutherAI Discord community being credited in the paperā€™s acknowledgements.

  • Cloud Computing Resources Examined: GCP and Colab surfaced as favorable cloud resources for NLP classification model training, with discussions encompassing cost-benefit analyses of platforms like runpod and vast.ai.

  • Research Computing Power Up for Grabs: EleutherAIā€™s computational resources were said to be available for collaboration on a semi-custom LLM project, with the caveat of having a clear research agenda and collaborative value proposition.

  • Semantic Scholarā€™s Linking Logic Revealed: Arxiv papers are automatically linked to authors on Semantic Scholar, with room for manual corrections to ensure accuracy.

  • Fractal Fun with Neural Training Parameters: @jbustter shared fractals created from neural network hyperparameters, highlighting a blog by Jascha Sohl-Dickstein that correlates fractals with training convergence/divergence.

  • A Deep Dive into Data Presentation for ML: A discussion was sparked concerning active learning and methods for models to choose their own data presentation sequence.

  • Enriching Encoder-Decoder Models with Unsupervised Data: Strategies to employ unsupervised datasets effectively in encoder-decoder models were discussed.

  • New NLP Robustness Method Flies Off the Press: A paper focusing on test-time augmentation (TTA) to enhance text classifiersā€™ robustness was published, with the author thanking the community for support.

  • The Quest for Interpretability Insight: @jaimerv asked for updated resources on interpretability methods beyond the standard Representation Engineering paper.

  • Summoning Collaborators for Hallucination Leaderboard: A call for contributions to a hallucinations leaderboard was made, requesting assistance with tasks, datasets, metrics, and result evaluations.

  • Aligning Pythia with Practice: Concerns were aired about potential misalignments between training batches and checkpoints in the 2.8b size Pythia deduped suite, with follow-up discussions suggesting opportunities for a publication on Pythiaā€™s reliability.

LlamaIndex Discord Summary

LlamaIndex v0.10 Marks Major Milestone: LlamaIndex v0.10 has been released, presenting notable advancements including a new llama-index-core package and PyPi packages for every integration/template. Detailed information on migration is accessible through their comprehensive blog post and documentation.

Webinar on No-Code RAG with LlamaIndex: A webinar demonstrating the creation of no-code Retrieve and Generate (RAG) apps using LlamaIndex.TS is set up with Flowise co-founder Henry Heng. Registration for the Friday event is available here.

Troubleshooting LlamaIndex: Engineers faced challenges with migration following LlamaIndexā€™s update and were pointed to a Notion migration guide for assistance. Furthermore, for configuration queries like chunk_size post-ServiceContext depreciation, engineers are advised to refer to the new Settings documentation and relevant LlamaIndex GitHub resources.

RAG App Building with Dewy Tremendously Simplified: A comprehensive guide to building a full-stack RAG app using NextJS, OpenAI, and the open-source knowledge base Dewy has been shared. The tutorial is aimed at grounding language models in precise, reliable data and can be studied in detail here.

Handling Document Complexity and Enhancing Enterprise with LlamaIndex: Users engaged in discussions about filtering complex documents and integrating LlamaIndex to enhance enterprise efficiency with tools such as Slack, Jira, and GDrive. Also, creating multiple agents for merging different document sources was considered, referencing the possibility of using traditional indexing techniques instead of high-cost LLMs for dynamic filtering.

HuggingFace Discord Summary

  • Hugging Face Accelerates with Message API: Hugging Face launched a new Message API compatible with OpenAI, aimed at streamlining use of inference endpoints and text generation services with client libraries. Theyā€™ve also advanced their offerings with new releases like Datatrove on PyPI, Gradio 4.18.0, and tools like Nanotron and Accelerate 0.27.0 for 3D parallelism training. Additional partnerships and resources, such as a Codecademy AI course and a blog post on SegMoE, support the continuous learning and innovation in their community.

  • Search Engine Woes and Hosting Queries in Focus: Technical discussions spotlighted the difficulties in creating search engines with mentions of approaches like TF-IDF and BM25, and the use of spaCy for Part of Speech tagging. Other conversations pivoted to queries about hosting custom models and serverless inferencing solutions, as well as the practicality of running 100B+ parameter models on enthusiast-level hardware.

  • Template Talk and Model Deployment Discussions: Users addressed the need for a simple chatbot development prototype capable of database interaction and email API integration, featuring resources like Microsoftā€™s AutoGen on GitHub and the potential of AutoGen Studio. Challenges around deploying finetuned machine learning models such as mistarl_7b_gptq for fast inferences were raised, with emphasis on choosing the right platforms or libraries for the task.

  • Glimpse into Creator Innovations: Members of the community showcased their creative projects, including GIMP 3.0 plugins interfacing with Automatic1111, development of an automated image tagging model for diffusion tasks, and updates to tools like PanoramAI.xyz introducing a ā€œremixā€ mode for image transformations. Excitement built around AI-applications in fashion design as well, demonstrating the breadth of applications being pursued.

  • Analyzing S4 and Advancing NLP: The community shared their insights into the S4 architecture ideal for long-range sequences and sought clarity on its implementation. The paper on LangTest got introduced, which offers testing and augmenting capabilities for NLP models. Topics extended to extracting language identifiers from models like XLM-RoBERTa and converting natural language into formal algebraic expressions.

  • Enthusiasm for Diffusion and Emerging Models: Conversations sparked around facilitating multi-GPU training for diffusion model fine-tuning, with mentions of scripts such as train_text_to_image.py. The successful deployment of models like mistarl_7b_gptq for fast inference, and effective text generation with stable cascade were discussed. The buzz was palpable around the teased development of a new terminus model.

  • Complications in Computer Vision Explored: The channel delved into challenges like hierarchical image classification, with resource suggestions including an ECCV22 paper on the same. Members discussed requirements for Gaussian splats, industry-grade image retrieval systems and sought collaboration on multimodal projects.

Nous Research AI Discord Summary

  • LongCorpus Dataset Unveiled for Pre-Training: The new LongCorpus-2.5B dataset is released, featuring 2.5 billion tokens from various domains, specifically curated for long-context continual pre-training and designed to minimize n-gram similarity with training sets.

  • Coherence Preserved in Scaling Models: Scaling with ā€˜self-extendā€™ is considered superior over ā€˜rope scalingā€™ for maintaining coherence, as indicated by the implementation in llama.cpp, and offers the benefit of requiring no setup, fine-tuning, or additional parameters.

  • Persistence and Resistance in AI Agents and Models: LangGraph agents can persist their state across interactions, as shown in a YouTube demonstration, while the Gemini model shows resistance, with its refusal tendencies prompting comparisons unfavorable to GPT-4.

  • Multimodal AI Breakthrough with Reka Flash: Reka Flash, a new 21B fast multimodal language model, is introduced and now available in public beta, promising to measure up to major models like Gemini Pro and GPT-3.5. The initiative can be followed on Reka AIā€™s announcement page.

  • CUDA Pains and WAVeform Gains in AI Research: The ZLUDA project aimed to run CUDA on AMD GPUs can no longer be considered active, and a fresh perspective in AI research proposed in an arXiv paper, suggests wavelet transforms could enhance Transformers by addressing both positional and frequency details efficiently.

Mistral Discord Summary

  • Newbies Get Model Recommendations: Participants recommended instruct models for chat-GPT-like interactions to newcomer @nana.wav, with the clearer instruction-following focus as opposed to the more general autocompletion capabilities of other models.

  • RAG Setup and Model Debates Heat Up: A guide on integrating Mistral with RAG was shared, while the effectiveness of LangChain vs. LlamaIndex was debated; separately, DSPy was touted for leveraging LLMs for programming rather than chatting, adorned with a supportive Twitter link.

  • Deployment Dilemmas and Solutions: Docker deployment via ollama or vllm projects was suggested, while others discussed API alternatives and faced cloud quota barriers; meanwhile, success stories involved deploying Mixtral on HuggingFace despite the hiccups with AWQ quantization.

  • Fine-Tuning Finesse and RAG Revelations: Users discussed fine-tuning vs. RAG with insights into LLM base knowledge importance; guidance was given on input data structuring for LLM output enhancement and queries about prompt versioning tools surfaced.

  • Humans in Tech and AI Seek Touchpoints: French librarian (@maeelk) sought internship opportunities in psychology and AI; the cost of innovatively building audio-inclusive S2S models sparked discussions around budget constraints and investment needs.

  • Technical Troubles and Support Suggestions: @ingohamm faces hurdles with TypingMindā€™s API key and a suggestion was made to contact [email protected] for assistance with API and subscription issues.

Perplexity AI Discord Summary

  • Perplexity AI Outshines Rivals in Complex Query Handling: @tbrams tested Perplexity AI with a difficult question from the ā€œGeminiā€ paper and found it outperformed Googleā€™s Gemini service and OpenAI, answering more quickly. The test results from Perplexity AI are documented here.

  • Perplexityā€™s Potential in API Customization Highlighted: The PPLX API allows for custom search queries using parameters like "site:reddit.com OR site:youtube.com", as mentioned by @me.lk. However, several users have encountered issues with the API such as performance hiccups (@andrewgazelka) and nonsensical responses (@myadmingushwork_52332).

  • Perplexity AI Subscription and Renewal Queries Addressed: Users are seeking details on trial subscriptions and renewal processes for Pro subscriptions, with inquiries about token refresh rates also surfacing. There is currently no early access program for new Perplexity features as confirmed by @icelavaman.

  • Promising Enhancements and Community Collaborations: Perplexity AI is receiving community praise for tools like the pplx shortcut action (@twodogseeds). Meanwhile, @ok.alex is encouraging a community-driven effort to contribute to an alternative feed/newsletter Alt-D-Feed.

  • Seeking Direct Support Channel for Sensitive Data Issues: A user (@kitsuiwebster) has expressed the need for direct assistance with a sensitive company data issue, avoiding public disclosure while lacking response from support channels.

OpenAI Discord Summary

  • ChatGPT Remembers Your Favorite Color: OpenAI announced a new memory feature for ChatGPT, rolling out to select users, enabling ChatGPT to remember user preferences and details over conversations for a more personalized experience. Users can control what ChatGPT remembers and can switch off this feature.

  • AI-Assistants in Creative Process Paid Talks: A UK researcher, @noodles7584, is looking to compensate community members for a 30-minute discussion on AI use in creative workflows.

  • Performance Quirks in GPT Variants: The community reported fluctuations in GPT-4ā€™s task handling, and Abacus.AIā€™s Smaug-72B was noted for outperforming GPT-3.5, while ChatGPT-4 seems hesitant to generate full code snippets.

  • Fine-Tuning AI to Watch Videos? Not Yet: Discussion in #gpt-4-discussions clarified that while GPT can describe images from a video with its vision capabilities, it cannot yet be fine-tuned for video-specific knowledge or tasks.

  • Exploring and Perfecting Prompt Engineering: Good prompt engineering was highlighted as involving clear instructions and precision, with a focus on fostering simple storytelling in text-based AI adventures and recognizing differences between prompt engineering and API infrastructure development.

OpenAccess AI Collective (axolotl) Discord Summary

  • Axolotl Embraces MPS, Thanks to GitHub Heroes: Maxime added MPS support in the axolotl project via pull request #1264, referencing the importance of a PyTorch pull request #99272. Clarification on contributor identities highlighted the importance of collective recognition in open source.

  • Chat In The Time Of Datasets: The MessagesList standard for chat datasets proposed by @dctanner aims for cross-compatibility and is under discussion. The format might include conversation pairs, greetings, and assistant-initiated closures, with challenges noted in JSON-schema validation.

  • Axolotl Tokenized Right, Check the Debug Flag: Users are troubleshooting tokenization within axolotl, with advice to inspect the tokenizer configs and a recommendation to use a debug flag for verification.

  • Model Query Woes and Training Queries Grow: Queries about improving modelā€™s multilingual capabilities, LoRA adapter inferencing, and model parallelism were discussed, with solutions ranging from pre-training needs to updates in transformers and DeepSpeed Zero 3 configs for better functionality.

  • Fine-tune or Re-train? Duplicate Dataā€™s Pain: The impact of training data overlap and finetuning practices were questioned, highlighting concerns about reusing text that a model may have encountered during pretraining.

  • RunPod Image on Vast.AI, A Smooth Sail!: The Axoltl RunPod image was reported by @dreamgen to work seamlessly on Vast.AI, underscoring the inter-operability with cloud infrastructure providers.

LangChain AI Discord Summary

  • LangChain Unveils Memory Journaling App: @hwchase17 introduced a new journaling app featuring LangChain memory module, inviting feedback for the early version akin to OpenAIā€™s ChatGPT with memory feature. Try and give feedback using this journal app and watch the intro video.

  • LangChain Community Tackles Diverse Technical Challenges: Topics covered included the possibility of LangChainā€™s Android integration, pre-processing benefits for efficient embeddings, the search for a capable PDF parser, and calls for improved documentation structure. Additionally, a user faced dependency issues while updating Pinecone Database to v2 with LangChain, which was promptly addressed.

  • Scaling and Integration Enquiries in Langserve Channel: Discussions included questions about scaling Langserve and using Langsmith for deployment. There was a query about exposing a chain from a NodeJS app and an unaddressed issue regarding disabling intermediate steps in the playground. Connection issues with an OpenAI API call from a k8s cluster-based app were also described.

  • Dewy RAG Application with NextJS and OpenAI Guide Shared: @kerinin contributed a guide exploring a full-stack RAG application, utilizing NextJS, OpenAI API, and Dewy, focusing on reducing hallucinations and improving model response accuracy. The full guide is available here.

  • Quest for a Functional PDF Parser and Custom Calculator: Within the tutorials channel, the search for a superior contextual PDF parser to Adobe API, and guidance for building a Langchain-based calculator were topics of discussion, aiming for practical integrations and solutions in AI workflows.

DiscoResearch Discord Summary

  • Seeking Argilla Hosting Solutions: @drxd1000 requested advice for hosting an Argilla server capable of supporting multiple annotators with no clear resolution reached.
  • Layer Selective Rank Reduction in the Spotlight: @johannhartmann discussed an implementation of ā€˜Layer Selective Rank Reductionā€™ for mitigating continual training forgetting. The method targets statistically less significant layer parts, and a GitHub repository was mentioned.
  • Overcoming OOM With Mixtral: @philipmay encountered an Out of Memory error with a mixtral model, and @bjoernp suggested using multi-GPU support, mentioning that two A100s might alleviate the issue.
  • Cross-Language Toxicity Detection Dataset: @sten6633 sought a German toxicity evaluation dataset, considering the translation of ToxiGen from Hugging Face, which requires access agreement.
  • New Computational Technique Teased: @phantine teased a technique named ā€œUniverses in a bottleā€ with implications for the P=NP problem, linked to a GitHub page, but details were sparse.
  • BM25 Search Strategy Proves Effective: huunguyen reported success using BM25 with additional querying and reranking to enhance search capabilities, and successfully indexed the entirety of Wikipedia into an index under 3GB.
  • German AI Model Update Inquiry: thomasrenkert asked about the release timeline for version 2 of the German model or a Mixtral variant, but no additional details were provided.

CUDA MODE Discord Summary

  • CUDA Compatibility Crusade: Members discussed achieving CUDA binary compatibility on HIP/ROCm platforms, driven by the ZLUDA project on GitHub, which is a CUDA on AMD GPU initiative. Amidst technical emoji enthusiasm, there were musings about market monopolies and AGI, alongside personal experiences with Radeon hardware issues related to dynamic memory allocation.

  • Generative AI Jobs Galore: A Deep Tech Generative AI startup in Hyderabad is hiring ML, Data, Research, and SDE roles, with applications welcomed here. However, the legitimacy of the job posting was questioned, flagging the need for moderator attention.

  • Compute Shaders and Matrix Math Musings: Inquiries on educational materials for CUDA led to The Book of Shaders recommendation, while the discussion in the PMPP book channel debated the benefits, or lack thereof, of transposing matrices to reduce cache misses in multiplication, indicating varied opinions but no consensus on observed benefits.

  • Apple Chips Enter Monitoring Realm: @marksaroufim shared asitop, a CLI performance monitoring tool designed for Apple Silicon, likening it to top or nvtop in utility for engineers leveraging Appleā€™s technology.

  • GPU Experiments and Job Shuffling: An engineer successfully relocated an Asus WS motherboard to a miner setup, effectively running large quantized models on a NVIDIA 3060 GPU. This indicates a hands-on approach within the community towards custom hardware configurations.

Latent Space Discord Summary

  • Reka Enters the Model Arena: A new AI entity named the Reka model has sparked interest in the community following a tweet shared by @swyxio. The excitement is palpable with discussions around the tweet found here.
  • Investor Insights Meet AI: @swyxio spotlighted a VC podcast delving into AI, which could be of significant interest to engineering aficionados. The podcast episode is accessible here.
  • BUD-E Buzz: BUD-E, an empathetic and context-aware open voice assistant developed by LAION, could signal a new direction in conversational AI. More details are laid out on the LAION blog.
  • Pondering the Definition of Agents: The community exchanged views on defining ā€œagents,ā€ with @slono suggesting that they are goal-oriented programs that require minimal input from users, a concept significant in the realm of AI development.
  • Karpathyā€™s OpenAI Exit Raises Questions: The AI community is abuzz over the news of Andrej Karpathy leaving OpenAI, with @nembal pointing to an article from The Information and speculation about AGI influences. The article is accessible here.

LLM Perf Enthusiasts AI Discord Summary

  • Minding the Model Size for M2 Max: @potrock inquired about running Mistral model sizes on an M2 Max with 32GB, and @natureplayer advised that a 4GB model would be the feasible option, cautioning against an 8GB model and noting that 5GB models may be unstable.

  • GPT-5 Rumor Mill: @res6969 expressed humorous doubt about GPT-5ā€™s existence, suggesting that speculation on the modelā€™s upcoming release is overstated, with others joining the jest with emojis.

  • Enhanced Memory in ChatGPT: @potrock highlighted a new feature tested in ChatGPT, based on a blog post, where it can retain user preferences and information across sessions for more personalized interactions.

AI Engineer Foundation Discord Summary

  • Weekly Sync-Up Teases DĆ©jĆ  vu: @._z playfully announces the start of the weekly team meeting likening it to a recurring DĆ©jĆ  vu experience.
  • Member Bows Out from Meeting: @juanreds sends regrets for being unable to attend the weekly meeting, offering apologies to the team.
  • Call for AI Hackathon Co-Hosts: @caramelchameleon seeks collaborators to co-host an AI developers hackathon in tandem with game developers in the lead-up to the GDC.
  • Hackathon Offers Dual Attendance Modes: The hackathon mentioned by @caramelchameleon has options for attendance both online and onsite in San Francisco.
  • Hackathon Organizer Steps Up: @yikesawjeez shows eagerness to get involved in organizing the hackathon and highlights their expertise with Bay Area events.

Skunkworks AI Discord Summary

  • Direct Messaging Initiated: User @bondconnery has put out a request for a private message.
  • Exploring LLaVA Framework Integration: @CodeMan inquired about integrating the LLaVA framework with an SGLang server and SGLang worker, aiming for a potentially more specialized setup than a conventional model worker.
  • Off-Topic Video Share Ignored: A non-technical video link was shared, not relevant to the engineering discussions.

The Alignment Lab AI Discord has no new messages. If this guild has been quiet for too long, let us know and we will remove it.

The Datasette - LLM (@SimonW) Discord has no new messages. If this guild has been quiet for too long, let us know and we will remove it.

PART 2: Detailed by-Channel summaries and links

TheBloke ā–· #general (1460 messagesšŸ”„šŸ”„šŸ”„):

  • Exploring the Limits of Large Language Models: Users are discussing new open-source large language models capable of handling extremely long contexts, such as the Large World Model which claims to work coherently with contexts up to 1 million tokens. There are also mentions of the Cohereā€™s aya model that supports 101 languages.
  • The Quest for Efficient Multimodal AIs: Conversations focus on multimodality in AI with references to models handling visual inputs and potential outputs, indicating significant advancements beyond text-based models. The jax-based tools required to run the models are causing installation hiccups for some users.
  • Models Under Scrutiny: The community is very active in testing released models, mentioning issues such as TUX dependency problems and a ValueError during setup, indicating some challenges in getting the advanced models running smoothly.
  • Users Share Knowledge: Experienced users offer insights and assistance on how to handle models and UIs for various tasks, including long-context quantization in existing frameworks like ExLlama v2. Discussions also touch on the possibility of banishing stop tokens to encourage longer continuous outputs.
  • Towards Intelligent Role-Playing: There is a discussion on finding the balance between RP-oriented models and smarter generalized ones, with mentions of a Mixtral variant (BagelMIsteryTour) that might better fulfill user requirements for intelligent and adaptable model behavior.

Links mentioned:

TheBloke ā–· #characters-roleplay-stories (154 messagesšŸ”„šŸ”„):

  • Exploring Miqu Variants: @superking__ opened a discussion about the performance of the Miqu models, particularly after being re-quants from the original GGUFs (Googleā€™s Generative Unsupervised Feature extraction). @soufflespethuman mentioned MiquMaid-v2-70B, a variant specifically fine-tuned for ERP (Erotic Role Play), and provided sensitive content links to various versions on Hugging Face, which have been marked due to their nature.

  • Performance Gain with Better Hardware: @superking__ shared their experience on performance improvement from ā€œpainfully slowā€ to ā€œalmost usableā€ by upgrading their hardware to 12GB VRAM, which changed the given modelā€™s tokens per second from 0.7t/s to 2.1t/s.

  • Model Comparisons and Recommendations: In the context of roleplay and storytelling, users discussed various models. @spottyluck praised Nous Capybara Limarpv3 34B for its capabilities and provided a link to the model on Hugging Face. @wolfsauge shared a sketch about ā€œThe Continentalā€ featuring Christopher Walken and @eqobaba inquired about appropriate models and settings for engaging in NSFW ERP, mentioning a specification of 48GB VRAM and RTX A600.

  • Discussing Model Output Improvement: @neriss suggested using a higher temperature or lower minimum probability to reduce repetition and improve creativity in AI model outputs. The conversation highlighted variations in temperature settings, with @dercheinz suggesting higher temperatures, while @neriss advised lower ones, each to counteract repetitive or uncreative responses from models.

  • Dataset Cleaning Challenges and Strategies: @c.gato and @potatooff exchanged thoughts on cleaning datasets manually, with @c.gato seeking advice on how to perform ngram analysis to prevent overtraining on specific ngrams. @mrdragonfox recommended using Pythonā€™s pandas library for handling tabular or JSON data, sharing a gist for guidance.

Links mentioned:

TheBloke ā–· #training-and-fine-tuning (43 messagesšŸ”„):

  • Understanding Finetuning Techniques: @starsupernova explained that Mixtral ā€“ Instruct was trained using SFT on an instruction dataset followed by Direct Preference Optimization (DPO) on a paired feedback dataset, as detailed on page 6 of their paper. DPO is described as an optimized form of RLHF/PPO finetuning.

  • Unsloth AIā€™s Apply Chat Template: @starsupernova, likely the founder of Unsloth AI, highlighted the use of apply_chat_template instead of Alpaca for training an LLM on multi-turn conversation datasets. They also hinted at uploading a new notebook with all chat templates to simplify the process.

  • Augmentoolkit for Instruct-Tuning Datasets: In the conversation, @mr.userbox020 shared a link to a GitHub repository offering a toolkit to convert Compute and Books Into Instruct-Tuning Datasets. Although @starsupernova was not familiar with it, they suggested trying it out as it appeared promising.

  • Anticipation for Updated Training Resources: @avinierdc is awaiting an updated notebook from @starsupernova for fine-tuning Mistral on multi-turn conversation datasets. @starsupernova assured they would ping when itā€™s available on the Unslothā€™s Discord server.

  • Unexplained Variation in Training Loss: @dreamgen reported observing a 2x higher training and evaluation loss when fine-tuning Mixtral 8x7b qlora compared to Mistral 7b, despite using the same dataset and similar hyperparameters, and inquired if others had seen something similar.

Links mentioned:

GitHub - e-p-armstrong/augmentoolkit: Convert Compute And Books Into Instruct-Tuning Datasets: Convert Compute And Books Into Instruct-Tuning Datasets - e-p-armstrong/augmentoolkit

TheBloke ā–· #coding (8 messagesšŸ”„):

  • Interest in Collaboration Sparked: @_b_i_s_c_u_i_t_s_ expressed interest in an unspecified topic, potentially around chatbot implementation, which was well received by @mr_pebble, finding it motivating to progress on implementing various chat methods.

  • Bridging JavaScript and Python: @spottyluck experimented with expanding the SillyTavern project to use a JavaScript-Python bridge, utilizing JSPyBridge to potentially adapt and enhance functionalities. They shared how it enabled testing of Microsoftā€™s LLMLingua, despite some issues with prompt mangling.

  • Using Python Classes in JavaScript: @spottyluck provided code examples illustrating the ease of creating Python classes within JavaScript using JSPyBridge, along with an asynchronous function, compressPrompt, which demonstrates the interaction between languages to compress prompts.

  • Modifications to Handle Windows Errors and Devices: In their continued development, @spottyluck modified Intelā€™s BigDL.llm transformer to support specific requirements, like cpu_embedding=True on Windows due to access violation errors, and dealing with model device allocation issues using model.to().

  • Compression Process Integration into Routing: @spottyluck explained integrating prompt compression into their web service by adding a flag to the /generate router post and using conditional logic to process the prompt through the bridge, demonstrating how Python can operate as if it were a JavaScript class.

Links mentioned:

GitHub - extremeheat/JSPyBridge: šŸŒ‰. Bridge to interoperate Node.js and Python: šŸŒ‰. Bridge to interoperate Node.js and Python . Contribute to extremeheat/JSPyBridge development by creating an account on GitHub.

LM Studio ā–· #šŸ’¬-general (202 messagesšŸ”„šŸ”„):

  • Struggles with Large Models: Users like @nonm3 encountered errors while loading large models in LM Studio due to insufficient RAM and VRAM, with suggestions to try smaller model quants. Others like @theoverseerjackbright faced issues with LM Studio not detecting GPUs correctly and crashing post-restart.

  • Software Seekers and Recommendations: @tvb1199 was in search of client software that can interact with LM Studio for RAG capabilities, and was pointed towards AGiXT, while @pierrunoyt and others discussed Nvidiaā€™s ā€˜Chat with RTXā€™ with RAG features as a potential game-changer.

  • Compatibility Inquiries: Several users such as @wizzy09 had trouble installing or opening LLM Studio on unsupported platforms like a 2014 MacBook Pro, with clarifications from users like @heyitsyorkie explaining that LMStudio does not work on Intel Macs.

  • Nvidiaā€™s Chat with RTX Triggers Interest: The community showed a keen interest in Nvidiaā€™s ā€˜Chat with RTXā€™. Users like @hypocritipus were intrigued by the RAG feature, hoping for a similar easy-install, no-dependency RAG feature in LM Studio.

  • LM Studio Usage and Model Discussions: Users like @bigboimarkus expressed satisfaction with LM Studio for tasks such as proofreading, whereas @mr.stark_ queried about models that learn on the fly. Conversations included the functionality and integration with other tools like Ollama and Automatic1111.

  • General Community Assistance and Banter: Throughout, community members engaged in sharing tips, offering troubleshooting advice, including suggestions for alternatives or downgrading versions, and occasionally joked about AI capabilities such as predicting lottery numbers.

Links mentioned:

LM Studio ā–· #šŸ¤–-models-discussion-chat (75 messagesšŸ”„šŸ”„):

  • MedAlpaca for Medical LLMs: User @heyitsyorkie suggested medAlpaca, a model fine-tuned for medical question answering, for @pepito92iā€™s project on LLMs in the medical field.
  • Phi-2 Model Discussions: @.dochoss333 inquired about the absence of the official ā€œmicrosoft/phi-2ā€ model in LM Studio, and @heyitsyorkie clarified that itā€™s not a GGUF model and thus wonā€™t show up. @hugocapstagiaire_54167 mentioned user TheBloke might have transformed it into a .gguf for usability with llama.cpp.
  • LLama.cpp and Model Support: @jedd1 puzzled over why some models wouldnā€™t load, and @heyitsyorkie pointed out that the Yi-VL models are unsupported in the current build of llama.cpp, requiring an update for compatibility.
  • LM Studio Assistant Functionality Inquiry: User @edu0835 inquired about the possibility of creating an assistant in LM Studio with the ability to utilize PDFs or books for a medical assistant application, without a direct response provided at this time.
  • Model Performance Comparisons Engage Community: Users like @kujila and @heyitsyorkie engaged in comparisons between different language models, with discussions on model specificity, ethical behavior of AI models, and suggestions to try out models like Deepseek Coder Ins 33B.

Links mentioned:

LM Studio ā–· #šŸŽ›-hardware-discussion (140 messagesšŸ”„šŸ”„):

  • NVLink and Memory Cycling Queries: @luxaplexx asked if GPUs were NVLinked and how memory cycles through in a multi-GPU setup. The consensus, including from @heyitsyorkie, is that they likely arenā€™t NVLinked due to potential CUDA issues, especially with older cards like the 960. Users are considering whether different generations of NVIDIA GPUs like the 1080 and 1060 6g can work together effectively.

  • Discussions on Upgrading to Better GPUs: Several users, including @crsongbirb and @heyitsyorkie, discussed upgrading their GPUs for improved performance in LLM tasks, with a suggestion to look at the RTX 3060 12GB as a viable option for running LLMs locally.

  • Risks and Rewards of Overclocking: In a discussion initiated by @alastair9776 about overclocking for better performance, @rugg0064 and @crsongbirb noted that overclocking VRAM/RAM can lead to a notable increase in token generation speed, although caution is advised due to potential hardware stress.

  • Combining GPUs and Threadripper Dreams: Conversation ensued about the feasibility and costs of using multiple high-performance GPUs, with users like @nink1 and @quickdive. debating if a beefy CPU is necessary when having multiple powerful GPUs, and the logistics of housing such a setup.

  • CUDA on AMD and Other Hardware Convos: @666siegfried666 shared news about the ZLUDA project allowing CUDA apps to run on AMD hardware and this sparked a brief discussion on the relevance and future potential of such a feature. Users such as @addressofreturnaddress and @joelthebuilder also discussed their own rig setups and potential upgrades, highlighting personal preferences and value assessments.

Links mentioned:

LM Studio ā–· #šŸ§Ŗ-beta-releases-chat (21 messagesšŸ”„):

  • Awaiting the Next Update for IQ3_XSS Support: @n8programs inquired about IQ3_XSS support in the latest release, to which @yagilb responded that it will be included in the next update.
  • Elevation of 1bit Quantization on the Horizon: @drawless111 shared excitement about upcoming 1.5 bit quantization, posting a GitHub pull request link indicating progress. This elicits reactions with @heyitsyorkie anticipating a sweet next beta with the new quant sizes.
  • Model Benchmarking Induces Awe: @drawless111 expressed amazement at the latest benchmarks for 1bit quantization, stating ā€œ70B model on 16 GB card. WOOF.ā€ and pointing out a ā€˜70Bā€™ model posted on Hugging Face that can offload on VRAM effectively.
  • Preparations for Incompatible Model Downloads: Users, including @epicureus, are advised to download models like IQ3_XSS even if theyā€™re not supported yet, with @fabguy humorously suggesting ā€œSave the model, save the world!ā€
  • Hugging Face Hub Features Multiple New Models: @drawless111 shared an update, revealing the availability of 5 IQ1 models on Hugging Face that work with various VRAM sizes, nonchalantly noting an increase to 10 by the end of the conversation.

Links mentioned:

LM Studio ā–· #avx-beta (2 messages):

  • No AVX2, No Cry: @rafalsebastian expressed concerns about not being able to run LMstudio on CPUs with only AVX (version one) after getting the message that their processor doesnā€™t support AVX2. They wondered if they should switch machines for running local LLMs.
  • LM Studio Beta for the Rescue: @heyitsyorkie responded with a solution, mentioning that LM Studio can indeed run on CPUs with only AVX support by downloading the 0.2.10 AVX beta release for Windows. They also recommended upgrading to a CPU with AVX2 for optimal results and provided a link to beta releases and terms of use.

Links mentioned:

LM Studio Beta Releases: no description found

LM Studio ā–· #crew-ai (3 messages):

  • Looking for Dual Model Deployment Tips: @alluring_seahorse_04960 wonders how to run two models on the same machine without facing repetition errors. The user mentions using a Conda environment on Ubuntu and avoids VMs for their slowness.
  • Humorous Clarification Request on Repetition: In response to @alluring_seahorse_04960, @wolfspyre jokes about the nature of the repetition errors, questioning whether they pertain to looping outputs or tasking issues within worker processes.

LAION ā–· #general (361 messagesšŸ”„šŸ”„):

  • Magvit V2 Reproduction Inquiries: @.lostneko sought technical guidance for reproducing Magvit V2. Discussions circled around the ideal datasets and parameters for video compression and understanding, with @chad_in_the_house mentioning experiments on the lfq side of Magvit2.

  • Mysterious Buzz around Magvit: @pseudoterminalx and others in the chat noticed sudden interest in MAGVIT, speculating about a recent influencer mention given the two mentions within a short time frame.

  • Stable Cascade Discussions Heat Up: Focus shifted to Stability AIā€™s Stable Cascade model, with dialogues highlighting its hefty VRAM requirements, misleading inference time graphs, and concerns about the model being poorly optimized and full of bugs. @pseudoterminalx shared examples of its capabilities, including issues with text clarity in image outputs.

  • Evaluating AI Models and Copyright Concerns: Conversations touched on the usage and legality of AI-generated images. Users @vrus0188 and @kenjiqq debated AI image model copyrights, commercial use, and the implications of research-only model licenses.

  • Hardware and Performance Perspectives: A technical dialogue ensued over Stable Cascadeā€™s heavy VRAM use and optimization problems, as @pseudoterminalx reported issues like inability to run models in float16 and @kenjiqq provided details about inference time on consumer GPUs like the 3090.

Links mentioned:

LAION ā–· #research (48 messagesšŸ”„):

  • Discussion on Impact of Adult Content on AI: @vrus0188 and others discuss the historical contributions of adult content to advancing technology, juxtaposing it with AI developments. Some users like @twoabove acknowledge the pattern of adult industries driving tech advancements, while others like @SegmentationFault doubt if the focus on adult content leads to meaningful progress in AI.

  • Concern Over Explicit AI-Generated Content: @thejonasbrothers shares a news article highlighting the misuse of AI in creating non-consensual pornography, noting the challenges it poses and its high visibility. This leads to a discussion on the broader implications and controversies surrounding AIā€™s use in adult content.

  • Observations on the Pornography Market and AI: Users like @chad_in_the_house and @freon discuss the profitability and market saturation of NSFW content, contemplating the economical and ethical risks involved in this space.

  • Debates Over the Merits of AI-Powered Erotic Roleplay: @SegmentationFault expresses frustration over the preference for low-effort erotic content in AI communities, arguing that this hinders meaningful developments in AI models. Others like @mfcool and @.undeleted echo these sentiments, criticizing the quality stagnation in AI-generated adult imagery.

  • Technical Discussion on AI Image Quality: @drhead delves into technical aspects of AI-generated images, mentioning the NovelAI model and discussing the viability and impact of VAE encoder training for improved image generation. There is a communal reflection on the standards of ā€œphotorealismā€ within the community and how they could be improved.

Links mentioned:

Eleuther ā–· #general (179 messagesšŸ”„šŸ”„):

  • Checksums for The Pile Data Located: @paganpegasus provided @hailey_schoelkopf with the checksums for The Pile zst shards, linking both the Discord pins and the EleutherAIā€™s hashes.

  • Tools to Determine Image Content: @everlasting_gomjabbar inquired about tools to discern if an image is of an object/location versus ā€˜nothingā€™ like a blurry shot. @paganpegasus described the complexity of defining ā€œnothingā€ in images, while @rallio. recommended using models like OwlViT or CLIP.

  • Manuscript Review and Editing in Progress: @wonkothesensible, through a series of messages, provided meticulous feedback on a paper draft provisionally titled ā€œDonā€™t think about the paperā€, focusing on clarifying language and grammar. @hailey_schoelkopf expressed gratitude and indicated credits to the EleutherAI Discord in the paperā€™s acknowledgements.

  • Cloud Resources for NLP Classification Discussed: In response to @pxxxl seeking advice on cloud resources for training NLP classification models, @ad8e recommended GCP and Colab, with various participants chiming in about the costs and features of various platforms like runpod and vast.ai.

  • Inquiries About EleutherAI Computing Resources: User @vidava asked about the guidelines and requirements for accessing EleutherAIā€™s computational resources for a semi-custom LLM project featuring architectural adjustments and fine-tuning adapters. @stellaathena indicated openness to collaboration but highlighted the need for clarity on the research agenda and proposed a collaborative value proposition.

  • Semantic Scholar Paper-Author Linking Mechanism: Regarding whether Semantic Scholar automatically links Arxiv papers to authors, _inox clarified that the process is automatic but allows for manual intervention or suggested changes if errors occur.

Links mentioned:

Eleuther ā–· #research (208 messagesšŸ”„šŸ”„):

  • Fractal Analysis of Neural Network Hyperparameters: @jbustter shared visualizations of fractals generated from neural network hyperparameters, with red indicating diverging training and blue for converging. Jascha Sohl-Dicksteinā€™s blog post showcases the concept, correlating fractal patterns with the learning rates of network layers and the networkā€™s weight offset.

  • Discussing Convergence and Divergence in Training: The conversation, involving users like @Hawk, @genetyx8, and @mrgonao, discussed the means for determining if neural network training is converging or diverging, debating the presence of ā€œdiverging to infinityā€ and the nature of boundaries within fractal visualizations, with suggestions that NaNs may denote divergence.

  • Active Learning and Data Presentation Order in ML: @rybchuk inquired about research on models choosing the order of data presentation, leading to a discussion about active learning. @thatspysaspy mentioned the subfieldā€™s existence, noting its lack of success, and @catboy_slim_ added that it could halve training requirements by using smaller models to filter data for larger modelsā€™ training.

  • Leveraging Unsupervised Data in Encoder-Decoder Models: The question of how to utilize large unsupervised datasets effectively in encoder-decoder models for tasks such as audio to text was brought up by @loubb. Suggestions and discussions ranged from training components separately to integrating cross-attention during pre-training.

  • Release of an NLP Robustness Paper and Test-Time Augmentation: @millander announced the publication of their lead author paper on improving text classifiersā€™ robustness through test-time augmentation (TTA) using large language models. They thanked the community for support and shared the arxiv link to their work.

Links mentioned:

Eleuther ā–· #interpretability-general (1 messages):

  • In Search of Interpretability Guidance: @jaimerv reached out to the channel asking for a more current overview of approaches to interpretability than the paper they referenced on Representation Engineering. They are seeking assistance for potentially better or newer resources on the topic.

Eleuther ā–· #lm-thunderdome (4 messages):

  • Contributors Wanted for Hallucinations Leaderboard: @pminervini shared a call to action for contributions to the hallucinations leaderboard, adding that there are several new hallucination-oriented tasks to work on within the Harness leaderboard space.
  • Enthusiastic Response to Collaboration: Following the announcement, @baber_ expressed interest and asked what specific help was needed.
  • Call for Specific Assistance: In response, @pminervini mentioned they need help with task definitions, proposing/adding new datasets and metrics, and assistance in determining which results to re-compute following recent updates to the harness.

Eleuther ā–· #gpt-neox-dev (4 messages):

  • Potential Misalignment in Pythia Deduped Data: @pietrolesci has raised concerns about a possible misalignment between training data batches and checkpoints specifically for the 2.8b size Pythia deduped suite. Other models, including the smaller versions and 6.9b, seem well-aligned.
  • Response to Data Alignment Query: @hailey_schoelkopf acknowledged @pietrolesciā€™s query about the alignment issue and stated they will follow up on this matter.
  • Interest in Pythia Research and Suggestion for Publication: @stellaathena expressed excitement about the potential for a blog post or workshop paper demonstrating the reliability of Pythia, which they would extensively cite.
  • Openness to Writing About Pythia: In response to @stellaathena, @pietrolesci appreciated the suggestion about creating a post regarding their findings on Pythia, considering it a good short project post-ACL deadline.

LlamaIndex ā–· #announcements (2 messages):

  • LlamaIndex v0.10 Released: @jerryjliu0 announced the release of LlamaIndex v0.10, which is the most significant update to date, featuring a new llama-index-core package and splitting integrations/templates into separate PyPi packages. The llamahub.ai is also being revamped, theyā€™ve deprecated ServiceContext for better developer experience, and encourage the community to explore the blog post and documentation for detailed info on migration and contributing.
  • Celebrating Team Achievement: Big thanks were given to <@334536717648265216> and <@908844510807728140> for leading the effort on the latest LlamaIndex update, which is a step towards making it a production-ready data framework.
  • Tweet about LlamaIndex v0.10 Launch: LlamaIndex shared a tweet highlighting key updates in LlamaIndex v0.10, including the creation of hundreds of separate PyPi packages, the refactoring of LlamaHub, and the deprecation of ServiceContext.
  • Webinar Announcement with No-Code RAG Tutorial: Flowiseā€™s co-founder, Henry Heng, will feature in a LlamaIndex Webinar to demonstrate building no-code Retrieve and Generate (RAG) applications using their new integration with LlamaIndex.TS. The webinar is scheduled for Friday 9am PT and interested individuals can register here.

Links mentioned:

  • LlamaIndex Webinar: Build No-Code RAG Ā· Zoom Ā· Luma: Flowise is one of the leading no-code tools for building LLM-powered workflows. Instead of learning how to code in a framework / programming language, users can drag and drop the componentsā€¦
  • Tweet from LlamaIndex šŸ¦™ (@llama_index): šŸ’« LlamaIndex v0.10 šŸ’« - our biggest open-source release to date, and a massive step towards production-readiness. šŸš€ āœ…Ā Create a core package, split off every integration/template into separate PyPi ā€¦
  • LlamaIndex v0.10: Today weā€™re excited to launch LlamaIndex v0.10.0. It is by far the biggest update to our Python package to date (see this gargantuan PR)ā€¦

LlamaIndex ā–· #blog (5 messages):

  • LlamaIndex Hits v0.10 Milestone: LlamaIndex announces its biggest open-source release, v0.10, signaling a shift towards production-readiness. A core package has been created and hundreds of integrations split off into separate PyPi packages as highlighted in their Twitter post.

  • Tutorial on Multimodal Apps with LlamaIndex: @ollama and LlamaIndex co-present a tutorial for building context-augmented multimodal applications on a MacBook, including smart receipt reading and product image augmentation, shared via this tweet.

  • DanswerAI Enhances Enterprise with LlamaIndex: DanswerAI leverages @llama_index to offer ChatGPT functionalities over enterprise knowledge bases, integrating with common workplace tools such as GDrive, Slack, and Jira to boost team efficiency as announced in the Twitter announcement.

  • Upcoming No-Code RAG Webinar with FlowiseAI: @llama_index teams up with @FlowiseAI for a webinar on building no-code RAG (Retrieval-Augmented Generation) workflows with LlamaIndex.TS and Flowise, details in their recent tweet.

  • Define Research Workflow with RAG-powered Agent: A notebook by @quantoceanli outlines a process to establish a scientific research workflow, harnessing LlamaIndex to operate with resources like ArXiv and Wikipedia for an innovative RAG-powered agent, showcased in this tweet.

LlamaIndex ā–· #general (303 messagesšŸ”„šŸ”„):

  • LlamaIndex Import Troubles: Users like @ddashed, @bhrdwj, @lhc1921, and @cheesyfishes discuss issues with the latest LlamaIndex update. Users were advised to start with a fresh venv or container and pointed towards a migration guide and package registry for reference.

  • Complex Document Filtering Challenges: User @_shrigmamale sought assistance in filtering large directories of complex documents based on keywords, dates, and file types. Another user, @qingsongyao, suggested traditional indexing techniques over expensive LLMs like GPT-4 for dynamic file filtering.

  • Efficient Handling of Multiple Document Sources: Users like @nvmm_, @whitefang_jr, and @.saitej engaged in discussions about handling and merging private user-uploaded documents with public indexed documents using LlamaIndex and the potential for creating multiple agents for individual documents.

  • Configuring Chunk Sizes and Testing Performance: @sgaseretto asked about where to specify chunk_size now that ServiceContext is deprecated in favor of Settings. @cheesyfishes provided the new way to configure chunk size globally or by passing the node parser/text splitter into the index.

  • Handling Changes with Chat Memory Buffer: @benzen.vn inquired about experiencing non-relevant responses when using a ChatMemoryBuffer. @whitefang_jr suggested that off-topic conversations might degrade the relevancy of queries and pointed to parts of the LlamaIndex source code for explanation.

Links mentioned:

LlamaIndex ā–· #ai-discussion (1 messages):

  • Super-Easy Full Stack RAG App Building Guide Released: @kerinin has shared an article about building a Retrieval-Augmented Generation (RAG) application using Dewy, a new open-source knowledge base. The guide entails using NextJS, OpenAI API, and Dewy to create a RAG application that improves the accuracy of language model responses by grounding them in specific, reliable information. Read the guide.

Links mentioned:

Building a RAG chatbot with NextJS, OpenAI & Dewy | Dewy: This guide will walk you through building a RAG application using NextJS for the web framework, the OpenAI API for the language model, and Dewy as your knowledge base.

HuggingFace ā–· #announcements (1 messages):

  • Hugging Face Launches Message API: šŸš€ Hugging Face introduces a new Message API compatible with OpenAI, enabling the use of OpenAI client libraries or third-party tools directly with Hugging Face Inference Endpoints and Text Generation Inference. Learn more from their announcement here.

  • New Open Source Releases and Features: šŸ¤— Datatrove goes live on PyPI, Gradio updates to 4.18.0 with an improved ChatInterface and more, and thereā€™s a launch of Remove Background Web for in-browser background removal. Additionally, Nanotron for 3D parallelism training and new features in Hugging Face Competitions were announced. Accelerate 0.27.0 was released, boasting a PyTorch-native pipeline-parallel inference framework.

  • Product Innovations at Hugging Face: HF introduces LoRA Studio with a dedicated UI on the Hub, incorporates 2FA support, releases a Mask Generation task page, and announces the arrival of models trained with Axolotl.

  • Partnerships and Learning Resources Expansion: Hugging Face announces a partnership with Codecademy for a new free AI course on transformers and publishes a blog post about SegMoE, which enables model merging on text-to-image models.

  • Optimizing Model Performance: Thereā€™s a technique to load pre-trained PyTorch models approximately 2x faster using Accelerate, detailed in a user guide by @RisingSayak.

Links mentioned:

HuggingFace ā–· #general (192 messagesšŸ”„šŸ”„):

<ul>
  <li><strong>Search Engine Development Struggles</strong>: <code>@spidy___</code> discussed challenges in developing a search engine and extracting keywords with <code>@vipitis</code>, <code>@cubietom</code>, and others. The conversation explored the limitations of NER and alternatives like keyword extraction, TF-IDF, BM25, and the use of spaCy for Part of Speech tagging.</li>
  <li><strong>Hosting and Inferencing Challenges</strong>: Users like <code>@sullynaj</code> and <code>@ram1428</code> enquired about hosting custom models and whether serverless inferencing is available, with pointers to server-less or affordable solutions discussed.</li>
  <li><strong>Tackling Model Scale</strong>: Conversations with users like <code>@zorian_93363</code> and <code>@xacer_</code> revolved around the feasibility and usefulness of running very large models (100B+ parameters) on typical "open source enthusiast" hardware.</li>
  <li><strong>Valentine's Day Vibes</strong>: <code>@not_lain</code> spread love and joy on Valentine's Day, encouraging the community to hug their loved ones.</li>
  <li><strong>Discussion on Running Models Locally</strong>: <code>@aj_0003</code> asked about running machine learning models locally while <code>@pierrunoyt</code> discussed using Hugging Face to clone and run a model.</li>
</ul>

Links mentioned:

HuggingFace ā–· #today-im-learning (9 messagesšŸ”„):

  • Simple Chatbot Development Blueprint: @wilbert.comicho is looking to create a simple chatbot to gather five specific details from a user and send them via email. They are seeking a template to handle database querying, user prompting/saving data and calling an API for email sending.

  • AutoGen as a Starting Point: @dwb7737 suggested using Microsoftā€™s AutoGen for chatbot development and pointed to GitHub for detailed use cases and Jupyter Notebooks. Additionally, highlighted that OpenAI is preferable to open-source LLMs when utilizing AutoGen.

  • Starting Small with AutoGen Studio: In a follow-up, @dwb7737 recommends getting to grips with the basics before diving into AutoGen Studio due to possible behavioral discrepancies and bugs, advocating for an understanding of the underlying processes. They provided a link to AutoGen Studio samples.

  • wilbert.comicho: Confirms they will be checking out the recommended resources.

  • Video Guide for Ollama Models: @dwb7737 shared a YouTube video as an excellent resource for learning how to use Ollama open source models in conjunction with LangChain and Autogen.

  • Google Sheets Merge Pitfalls: @lunarflu is engaged in merging two Google Sheets and cautions the importance of handling duplicate records and maintaining unique records to prevent issues.

  • Creating Transformers with FP8: @neuralink has progressed to mastering 99% of doremi reproduction and have advanced their training with end-to-end FP8 in 3D parallelism.

  • Switching from AI to Academia: @sardarkhan_ shares their shift from reading about diffusors and transformers to focusing on their upcoming mid-semester exams.

Links mentioned:

HuggingFace ā–· #cool-finds (8 messagesšŸ”„):

  • Back to ML After a Hiatus: @charlweed is diving back into Machine Learning by working on a GIMP 3.0 plugin that connects to Automatic1111, currently facing challenges with posting image data for Image2Image functionality via API.
  • Digging into the Dirt for Energy: @Gordo Stoli shared a research study on Soil Battery, a potential advancement in energy technology.
  • MoE Security Vulnerabilities Exposed: @osanseviero introduced a paper demonstrating how Mixture of Experts (MoE) models are susceptible to adversarial attacks affecting the outputs of benign queries.
  • Understanding MoE Risks and Mitigations: @osanseviero also wrote detailed notes on potential mitigation strategies for the vulnerabilities described in the DeepMind paper, suggesting batch order randomization among other methods, available here.
  • Questions about MoEā€™s Future Stability: @meatfucker highlighted the potential future threat of the reported MoE attack strategy and considered the implications for systems using large batches, which may inadvertently affect output quality.

Links mentioned:

  • [@osanseviero on Hugging Face: ā€œMixture of experts: beware šŸ›”ļøāš”ļø

New paper by DeepMind: Buffer Overflow inā€¦ā€](https://huggingface.co/posts/osanseviero/980907000007376): no description found

HuggingFace ā–· #i-made-this (10 messagesšŸ”„):

  • Quiz Generation Anticipation: @lunarflu suggested the addition of a loading screen or bar for the quiz generation process, mentioning an issue with just waiting for the quiz to appear without any indication.
  • Automated Image Tagging Model Deployed: @not_lain announced an automated model for tagging images pertinent to diffusion tasks and gave instructions for use, along with a link to their discussion. They also mentioned the modelā€™s implementation improvements in refs/pr/2.
  • Model Supports Various Image Formats: @not_lain highlighted that their tagging model accepts input as a string (path), a PIL image, or a numpy array, showcasing flexibility in handling images.
  • AI for Anime Data Set: @not_lain expressed intentions to use their image tagging model to annotate an anime dataset, while @imcoza1915 commented on the coolness of the tool.
  • New ā€œRemixā€ Mode for Image Transformation: @matthieulc shared an update to PanoramAI.xyz, introducing a ā€œremixā€ mode with ControlNet technology for better structure preservation in image transformations. Users are reminded they can navigate the tool using arrow keys.
  • From Sketch to Fashion with AI: @tony_assi unveiled a project Sketch to Fashion Design with great pride, which has received positive feedback as an AI able to understand designs, as @chad_in_the_house implied.

Links mentioned:

HuggingFace ā–· #reading-group (32 messagesšŸ”„):

  • S4 Architecture Gets Annotated: @ericauld shared a resource on ā€œThe Annotated S4ā€ asking for feedback and pointing out its usefulness for understanding the S4 architecture, which excels in modeling very long-range sequence tasks. They indicated that reading it may help clarify the model before their upcoming talk on Mamba/S4.
  • Seeking Clarity on S4 Implementation: @austintb. expressed desire for clarification on the S4 architectureā€™s implementation and computational complexity details. @chad_in_the_house echoed the sentiment, requesting intuitive explanation of concepts and prior work such as the hippo codebase, later suggesting a focus on intuition and coding for ericauldā€™s main talk.
  • Mamba/S4 Talk Schedule and Content Preferences: @ericauld proposed scheduling the Mamba/S4 talk for Friday at 10am California time and suggested potential content for the primary and secondary (math-focused) sessions based on community feedback.
  • LangTest Paper Makes Its Debut: @prikfy announced the publication of their LangTest paper in the Software Impacts journal, a tool for testing and augmenting NLP models. The paper and the GitHub repository for LangTest were shared, with @ryzxl contributing further context on its comprehensive testing capabilities and how to get started using the library.

Links mentioned:

HuggingFace ā–· #diffusion-discussions (10 messagesšŸ”„):

  • Multi-GPU Training Inquiry: George is looking for advice on adapting the train_text_to_image.py script for multi-GPU usage, mentioning previous experience with nn.DataParallel.

  • Deployment Options for finetuned models: @lokendra_71926 finetuned the mistarl_7b_gptq model and is seeking recommendations for a library or platform suitable for fast inference deployment.

  • Success with Stable Cascade: @isidentical asked if anyone achieved good text generation with stable cascade, similar to the examples in the readme and confirmed getting 50% success on arbitrary words with the right prompting strategy.

  • HuggingFaceā€™s Inference Engine Suggestion: @chad_in_the_house suggested that HuggingFace has an inference engine that could potentially serve for llms deployment and also mentioned that the discussion might be more appropriate in another channel.

  • Terminus Model Anticipation: @pseudoterminalx teased that a new terminus model is still in the development phase.

HuggingFace ā–· #computer-vision (7 messages):

  • Hierarchical Image Classification Challenge: @cropinky described the issue of hierarchical image classification and advised that the complication level depends on the quality and amount of data. They suggested checking out an ECCV22 paper and related datasets on paperswithcode for further research.

  • In Search of Gaussian Splats: @aeros93 inquired about resources or pre-trained models for creating Gaussian splats from point clouds or images. No specific resources were provided, but @johko990 redirected the query to another channel that could potentially help.

  • Quest for Multimodal Project Insights: @joee2711 is working on a multimodal project and sought clarification on the difference between Q-former / MLP connector and if MLP connectors and adapters are the same. They also expressed an interest in connecting with others working on similar projects.

  • Enhancing Image Retrieval Systems: User @femiloye is developing an image retrieval system akin to person reidentification and is looking for methods to improve match accuracy beyond using model embeddings. They are currently utilizing a custom deit transformer trained with reid loss for this purpose.

HuggingFace ā–· #NLP (4 messages):

  • Fine-tuning Mistral for Deployment: @lokendra_71926 fine-tuned mistarl_7b_gptq model on custom data and is seeking recommendations for a library or platform for deployment to achieve faster inference.

  • Language Identification with XLM-R: @_michaelsh inquired about how to extract the language from xlmr after reading a HuggingFace post which explains that XLM-RoBERTa does not require language tensors to understand the language being used.

  • From Natural Language to Algebraic Representations: @_david_valente_ is looking for research or work that has focused on translating natural language into algebraic representations such as LEAN.

  • Voice Simulation and Language Transformation with Transformers: @mentrass asked about methods to simulate oneā€™s voice and alter the language using transformer models.

HuggingFace ā–· #diffusion-discussions (10 messagesšŸ”„):

  • Multi-GPU Adaptation Inquiry: @George is looking for an easy way to adapt the train_text_to_image.py script for multi-GPU usage, noting past experience with nn.DataParallel.
  • Deployment Platform for finetuned model: @lokendra_71926 has finetuned the mistarl_7b_gptq model and is inquiring about a library or platform for fast inference deployment. @chad_in_the_house suggests looking at Hugging Face inference engine for LLMs.
  • Text Generation with Stable Cascade: @isidentical questions whether anyone has been able to achieve text generation with stable cascade as showcased in the modelā€™s readme, later confirming a 50% success rate with good prompting.
  • Inference Optimization Discussion Redirected: @chad_in_the_house points out that discussions regarding inference optimization should move to a different channel titled <#1019883044724822016>.
  • Anticipation for New Terminus Model: @pseudoterminalx indicates that a new terminus model is currently being developed.

Nous Research AI ā–· #ctx-length-research (3 messages):

  • DAMO-NLP-SG Releases Vast Long-Context Dataset: @giftedgummybee shared the LongCorpus-2.5B dataset which contains 2.5B tokens collected from various domains for long-context continual pre-training. The datasetā€™s composition is inspired by Long-Data-Collections, and its selection criteria ensures a low n-gram similarity with the training set to exclude QA and Summarization data.

  • Scaling Models with ā€˜ropeā€™ vs ā€˜self-extendā€™: @blackl1ght highlighted that scaling models with ā€˜self-extendā€™ can preserve coherence better than ā€˜rope scalingā€™, even at larger scaling factors, referring to the implementation in llama.cpp.

  • Ease of ā€˜self-extendā€™ Implementation: @blackl1ght noted the benefits of ā€˜self-extendā€™ including no need for setup, fine-tuning, or extra parameters like those required in the ā€˜gguf configurationsā€™ for quants.

Links mentioned:

DAMO-NLP-SG/LongCorpus-2.5B Ā· Datasets at Hugging Face: no description found

Nous Research AI ā–· #off-topic (8 messagesšŸ”„):

  • Discussing LangGraph Agentsā€™ Perseverance: @pradeep1148 shared a YouTube video titled ā€œLangGraph Agents Persistence,ā€ highlighting that LangGraph agents can be set up to retain their state across interactions.

  • Geminiā€™s Resistance Frustrates Users: @llmaniac1000 expressed disappointment with Geminiā€™s frequent refusal tendencies, seeking othersā€™ experiences with it. @n8programs chimed in, stating itā€™s not amazing and implying GPT-4 outperforms Gemini.

  • Mark Zuckerbergā€™s Image Transformation: @nonameusr shared a Twitter post suggesting that Zuckerberg has transitioned from villain to savior in the context of AI and VR.

  • A Touch of Humor with GIFs: @error.pdf reacted to previous discussions using humor by sharing a GIF from Tenor, without providing further commentary or context.

Links mentioned:

Nous Research AI ā–· #interesting-links (17 messagesšŸ”„):

  • Mesmerizing Mandelbrot Beauty Shared: @gabriel_syme posted a stunning visualization of the Mandelbrot set. @_3sphere added that the setā€™s focus on divergence contributes to its sense of complexity and order.

  • Crowdsourcing AI with ā€˜Marvā€™ Chatbot: @.dvs13 praised a crowdsourcing project and noted ambiguity in the term ā€œprompt.ā€ The project involves a chatbot named Marv, which answers questions with sarcasm.

  • Reka Introduces Multi-Modal AI Models: @metaldragon01 highlighted the launch of Reka Flash, a 21B fast multimodal language model, alongside its smaller counterpart Reka Edge. Reka Flash boasts competitive performance to major models like Gemini Pro and GPT-3.5 and is available in public beta.

  • Pursuing CUDA Compatibility with AMD: @leontello shared a GitHub project, ZLUDA, which aims to run CUDA on AMD GPUs. Unfortunately, the project is no longer actively pursued as detailed by @adjectiveallison, who quoted the projectā€™s lead expressing itā€™s effectively abandoned.

  • Wavelets Meets Transformers in AI Research: An arXiv paper shared by @euclaise suggests that wavelet transforms could enhance Transformers by capturing both positional and frequency information with linear complexity. The paper details Wavelet Space Attention (WavSpA) and has been tested on the Long Range Arena. Find the paper here.

Links mentioned:

Nous Research AI ā–· #general (180 messagesšŸ”„šŸ”„):

  • New Model Training Begins: @n8programs excitedly shares the start of training a new model, mentioning terms like dachshund, neuralbeagle-dpo, and expressing the process as randomly throwing stuff together genetic algorithm-style.
  • Playful Banter About Model Merging: @teknium humorously notes the metaphorical alignment between dog breeds and model merging, while @leontello likens the mixing methods to evolutionary strategies, and @n8programs reports a horrifying outcome of his merging experiment.
  • Typo Alert in Model Card: @everyoneisgross reports a typo in Hugging Faceā€™s model card for 70B llama, which was swiftly corrected by @teknium, leading to expressions of congratulations on the model launch.
  • Quantization Quest: Discussion about post-training quantization methods, with @stellaathena sharing a link to a new quantization method, and @nruaif jokingly looking forward to even lower bit-precision.
  • AI Activation Additions: A deep dive into activation hacking is mentioned, with @filipvv referencing an external article and @mihai4256 discussing their plans to refine their approach, while @proprietary voices interest in the work.

Links mentioned:

Nous Research AI ā–· #ask-about-llms (38 messagesšŸ”„):

  • DeepSeekMath Merges into the Conversation: User @yxzwayne inquired about the integration of newly introduced deepseekMath in merging strategies, indicating interest in its application.

  • Finetuning for Dummies Guide Discovered: @nemoia was searching for straightforward instructions on how to finetune Mistral 7B and create their own datasets and later shared a helpful Medium guide that provides detailed examples and explanations on the process.

  • Forced FA2 Line Causes Memory Issues: In response to a question about FA2 not being enabled, @bloc97 clarified that the problem was related to an attempt to create a large attn_weights matrix, indicating the line of code causing memory issues can be seen here.

  • Secondary Options for Coding Models: @natefyi_30842 was looking for a less expensive alternative to GPT-4 for a coding model, and @teknium suggested trying out the deepseek coder, which is hosted by ā€œtogether.ā€

  • MIQU Modelā€™s Pretraining and SFT Clarified: @teknium explained to @yxzwayne that the MIQU model was first pretrained on the Llama-2 70b and then underwent SFT (Supervised Fine Tuning), focusing specifically on instruction-focused data.

Links mentioned:

Nous Research AI ā–· #collective-cognition (3 messages):

  • Project Downfall Due to Chat GPT Update: @adjectiveallison inquired if a project was still active after encountering issues accessing the site. @teknium responded, clarifying that the website broke due to the new Chat GPT update with various modes, leading to the original team being unable to maintain it.
  • Sympathies for the Broken Project: @adjectiveallison expressed disappointment upon learning that the project was no longer maintained following the complications with the new Chat GPT update.

Mistral ā–· #general (43 messagesšŸ”„):

  • Model Selection Advice for Beginners: Newcomer @nana.wav inquired about the best models to use, and @afriendofmaurice recommended instruct models for chat-GPT-like interactions. @mrdragonfox clarified that instruct models are more focused on instruction following, whereas others are akin to raw autocomplete.
  • Integration with Visualization Libraries: @carnivore5 asked if anyone had experience integrating Mistral functionalities with GraphViz or similar visualization libraries, leading to a clarification by @mrdragonfox about Mistralā€™s lack of inherent function-calling ability.
  • Chat vs. Completion Endpoints: @i_am_dom and @mrdragonfox discussed the difference between Mistralā€™s /chat/completion and a wished-for raw /completion endpoint, with most usage currently gravitating towards the chat endpoint.
  • Internship Struggles with Mistral: @nana.wav shared struggles with learning how to use downloaded models and intentions to fine-tune them, leading @mrdragonfox to advise starting with simpler steps. The conversation included sympathy and reminiscence from others, highlighting the common intern experience with overwhelming tasks.
  • Mistral API Latency Issues: @justinmann. reported inconsistent latencies when using the Mistral API, with response times varying drastically from under a second to over a minute. @sublimatorniq suggested contacting support for assistance.

Mistral ā–· #models (20 messagesšŸ”„):

  • RAG Guide for Mistral: @ethux shared a helpful guide explaining how Mistral works with RAG (Retrieval-Augmented Generation), including steps on retrieval and generation with examples from Mistral, LangChain, and LlamaIndex.
  • Debate on LangChain vs. LlamaIndex: @sublimatorniq sparked a discussion on the effectiveness of LangChain vs. LlamaIndex, with @rabdullin expressing skepticism about their use in serious LLM-driven products.
  • DSPy Advocacy: @mrdragonfox advocated for DSPy as a powerful framework, citing that it uses LLM as a ā€œdeviceā€ and not a ā€œchatā€ interface and linked to a Twitter post exemplifying its strength.
  • Mistral-7b Training Dataset Inquiry: @kushagra_67246 inquired about the datasets on which Mistral-7b is trained, receiving humorous and vague responses indicating a mixed variety of internet sources ā€” from @tom_lrd describing it as ā€œTop secret magic soupā€ to @gamerboi0129 listing textbooks and Wikipedia among other comprehensive sources.
  • Clarification on Raw Pretraining Checkpoint: @nofreewill42 asked for an open-sourced checkpoint of the Mistral model right after raw text pretraining, expressing that mistralai/Mistral-7B-v0.1 seemed too interactive to be raw.

Links mentioned:

Mistral ā–· #deployment (46 messagesšŸ”„):

  • Docker Deployment Recommendations: @rusenask suggested checking out the ollama or vllm projects for APIs that can be run through Docker for different use cases.
  • Quota Troubles in the Cloud: @gridinoc experienced difficulties deploying Mixtral with SkyPilot as AWS, Google Cloud, and Azure either denied quota increases or did not respond to requests.
  • Alternatives to Self-Hosting: @mrdragonfox discussed options for deployment, suggesting cheaper API offerings such as direct mistral or together.ai, despite the current GPU shortages and quota issues faced by @gridinoc.
  • AWQ Quantization Hitches with MoE: Multiple users, including @mrdragonfox and @casper_ai, discussed issues with the AWQ quantization method and Mixtral models, with @casper_ai recommending an alternative working repository hosted on Hugging Face.
  • Success with HuggingFace Deployment: @ethux pointed to an instance of Mixtral deployed on HuggingFace.co/chat, offering an alternate route to those facing cloud service barriers.

Links mentioned:

Mistral ā–· #finetuning (76 messagesšŸ”„šŸ”„):

  • Fine-Tuning vs RAG Explained: Users in the channel debated the merits of fine-tuning versus using Retrieval-Augmented Generation (RAG), with @rabdullin advising to focus on prompt engineering and @mrdragonfox highlighting the importance of base knowledge in a Large Language Model (LLM) when using RAG. @tom_lrd and @mrdragonfox outlined that RAG acts as middleware to provide relevant context for the LLM and has its own complex underlying processes.
  • Onboarding the New AI Enthusiast: In response to @1mbc seeking resources for understanding AI core concepts, @mrdragonfox and @tom_lrd provided insights into how RAG and GPTs work and suggested platforms like Medium for further learning. No specific resources were linked.
  • Chatbot Integration Strategies Shared: The conversation delved into the technicalities of feeding an LLM with personalized data, with @mrdragonfox and @tom_lrd describing how data can be pre-processed and turned into a structured format that enriches the LLMā€™s output, specially when using an LLM as a middleware to process user input.
  • Clarifying Misconceptions on LLM Data Storage: @mrdragonfox corrected some misconceptions about how an LLM ā€˜learnsā€™ from new data, such as the functions of GPTs and the significant complexity behind embedding and search before data becomes usable context for an LLM.
  • Prompt Versioning Tools Inquiry: @khandelwaal.ankit inquired about tools for prompt versioning during fine-tuning experiments, noting a lack of support for Mistral models in some existing tools like PromptLayer; however, no solutions were specifically endorsed or detailed in the discussion.

Mistral ā–· #showcase (2 messages):

  • Limits on Code Modification: @ethux expressed skepticism about the possibility of making a certain change, suggesting that it might not be possible without altering some code.

Mistral ā–· #random (15 messagesšŸ”„):

  • French Librarian Seeks Internship Opportunities for Student: User @maeelk, a French librarian, is promoting AI use and looking for an internship for a student studying psychology and AI, referring to the Masterā€™s program at the University of Savoie Mont Blanc. Interested parties can reach out for collaboration via [email protected].
  • Mistralā€™s Fan Quiz: User @akshay_1 challenges @maeelkā€™s Mistral fandom by asking them to list the weights of the 7b model. Another user, @ethux, responds humorously, implying the difficulty of listing such technical details.
  • Building Audio-Inclusive S2S Models on a Shoestring Budget: @akshay_1 shares a clientā€™s request to build an S2S model with a persona, fine-tuned with an audio dataset on a budget of $1,000. Several users, like @ethux and @mrdragonfox, react to the insufficient budget, implying that much more would be required.
  • The Price of Innovation: @skadeskoten inquires about the competitive budget for creating a specialized S2S model, to which @mrdragonfox responds that the cost greatly depends on the extent of architecture needed.

Links mentioned:

Ergonomie socio-cognitive des systĆØmes intelligents - Classique et alternance - Ametys Campus - UniversitĆ© Savoie Mont Blanc: no description found

Mistral ā–· #la-plateforme (2 messages):

  • API Key Confusion for TypingMind: @ingohamm reported issues with using the API key for TypingMind, despite having a subscription and payment method in place. He mentioned that trying after a wait or deleting the API key prompted a message about no active subscription, and questioned the status of his account or subscription.
  • Seek Support from Mistral: In response to the issue, @sublimatorniq suggested that @ingohamm reach out to [email protected] for assistance with his API key and subscription concerns.

Perplexity AI ā–· #general (149 messagesšŸ”„šŸ”„):

  • Seeking Support for Company Data Issues: User @kitsuiwebster expressed that sending emails for support got no response and preferred not to disclose the data-related issue publicly. Instead, they wished to contact directly for help with a company-related problem.
  • Debating the Merits of Perplexity vs. Phind: User @ludwig_von_mises_fan opened a discussion about the effectiveness of Phind over Perplexity for coding and general search, while @gooddawg10 and @brknclock1215 defended Perplexityā€™s search capabilities, with no preference for coding.
  • Experiencing Technical Difficulties with Perplexity: Users @yellephen, @luke_____________, and @chenlieong reported issues with the Perplexity chatbot, such as endless loading for answers and service unavailability; @dima_shliugaev from the team acknowledged the issue and it was confirmed to be back online by @vova_at_pplx_ai.
  • Model Performance and Usage Discussions: Users shared their experiences with different AI models for tasks such as code debugging (@matheusgnhr), tic-tac-toe (@noremac258), and PDF reading (@reader7904); queries regarding specific model details (@hzpd and @unknownuser787) and API usage (@pilotgfx) were also seen.
  • Subscription Details and Model Information Inquiry: Users @stocktown and @ewaathescientist sought clarification on trial subscriptions and the renewal of Pro subscriptions, while @voidfulness inquired about token refresh rates and was informed by @me.lk that tokens refresh 24 hours after use.

Links mentioned:

Perplexity AI ā–· #sharing (13 messagesšŸ”„):

  • Perplexity AI Tackles Tough Test Question: @tbrams was impressed with how quickly Perplexity AI handled a complex question from the ā€œGeminiā€ paper, a task that Googleā€™s Gemini service and OpenAI took longer to address. Details of this successful test are available on the Perplexity AI platform.
  • Community Contributions and Creations: @twodogseeds gave a shoutout to Perplexity for the pplx shortcut action, which supports their Farm Friend research agent. No further details were shared in the message.
  • Exploring Diverse Perspectives with Bryan Johnson: @ok.alex shared a link to a summary of Bryan Johnsonā€™s perspectives via Perplexity AI, while @brknclock1215 offered an alternative angle for scientific summarization. Links to these summaries are found at Bryan Johnson Summary and Scientific Summary respectively.
  • Engage with the Alt-D-Feed: @ok.alex invited the community to contribute to an alternative feed/newsletter, suggesting it as a collaborative project to curate together. Interested individuals can like and share this initiative.
  • Summarizing Documents in Seconds!: @aykbl expressed enthusiasm for Perplexity AIā€™s capability to summarize documents swiftly, emphasizing its speed with a smiley face. The content linked or specificity of documents was not mentioned.

Links mentioned:

Discord - A New Way to Chat with Friends & Communities: Discord is the easiest way to communicate over voice, video, and text. Chat, hang out, and stay close with your friends and communities.

Perplexity AI ā–· #pplx-api (24 messagesšŸ”„):

  • Custom Search Queries Available: User @me.lk clarified that by using search parameters such as "site:reddit.com OR site:youtube.com" in prompts, one can specify multiple content sources when using the API.
  • Performance Issues with Online API: @andrewgazelka reported performance problems with pplx-70b-online, but noted that removing the system message in the code seemed to fix the issue.
  • PPLX API Fails with Nonsensical Responses: @myadmingushwork_52332 raised a concern with the API returning random and nonsensical replies involving a mix of numbers and characters when online searching is required.
  • Reference Provision Under Development: @dvrshil expressed a desire for Perplexity to provide references in API responses, to which @mares1317 responded, stating that the development team is working on this feature.
  • No Early Access Program Yet: @icelavaman indicated that early access to new Perplexity features is not available at this moment; announcements for new features will come at a later date.

Links mentioned:

OpenAI ā–· #annnouncements (1 messages):

  • ChatGPT Gets a Memory Boost: @abdubs announced a new memory feature for ChatGPT which allows it to remember user preferences and details across conversations, thereby enhancing future interactions. This feature is rolling out to select Free and Plus users, with control options available at ChatGPT Memory Features.

  • Youā€™re the Boss of ChatGPTā€™s Memory: Users have the power to tell ChatGPT what to remember, ask it to recall information, and instruct it to forget things conversationally or through settings. The memory feature can also be turned off completely if preferred.

  • Memory Feature Rolling Out Gradually: OpenAI is currently deploying the memory upgrade to a limited user base and plans to gather feedback to gauge its usefulness. Further announcements regarding a broader rollout will be made soon.

Links mentioned:

Memory and new controls for ChatGPT: Weā€™re testing the ability for ChatGPT to remember things you discuss to make future chats more helpful. Youā€™re in control of ChatGPTā€™s memory.

OpenAI ā–· #ai-discussions (83 messagesšŸ”„šŸ”„):

  • Discovering the Secrets of SEO: @spidy___ sought insights on how to autonomously tag webpages with relevant keywords like web crawlers do, finding limitations in NER for keyword extraction. @light.grey.labs advised examining SEO files, as web builders often embed a variety of keywords into these for search relevance.

  • Seeking Creative Minds for AI Research: @noodles7584, a UK researcher, invited community members to discuss how AI is used in creative processes, offering compensation for the 30-minute discussions.

  • The Quest for the Ultimate Chatbot: Chat explored the challenges with current chatbots, including the inability of GPT models to meet all individual needs, voiced by @jaicraft. @lumirix and others discussed workarounds, like combining bots or leveraging chatbot integrations with services like Google Docs.

  • ChatGPT Accused of Laziness: @pigondrugs and others commented on GPTā€™s difficulty retaining context, with growing complaints after context capacity increased. In contrast, @drinkoblog.weebly.com argued that higher context limits reduce perplexity, leading to better performance.

  • AI Model Rivalry Heats Up: @cassofthenight spotlighted Abacus.AIā€™s Smaug-72B model outperforming GPT-3.5 and expressed concerns over ChatGPT-4ā€™s reluctance to produce complete code snippets, suggesting that the AI dodges detailed scripting in favor of pseudo code.

Links mentioned:

Discord - A New Way to Chat with Friends & Communities: Discord is the easiest way to communicate over voice, video, and text. Chat, hang out, and stay close with your friends and communities.

OpenAI ā–· #gpt-4-discussions (49 messagesšŸ”„):

  • GPT 4 Turbo Cost Queries Clarified: @jeremy.o directed @koukyo to the OpenAI pricing page for GPT 4 cost details and noted that GPT 4 Turbo is the top/cheapest model, costing 2 cents less than other versions, with similar or slightly worse quality depending on use.

  • GPT 4 Sometimes Slacks Off?: @rodney.leonardo reported a decrease in GPT 4ā€™s intelligence in basic tasks, like summarizing a PDF. Community members including @blckreaper confirmed observations of performance issues, and discussions on the topic are collected in a separate channel: <#1047565374645870743>.

  • Still Waiting for @mentions: Users including @pax0086 and @ancryp discussed the gradual rollout of the @mention feature in GPT, with @darkninjaforever reminding that OpenAI often does gradual feature rollouts, indicating some users are still awaiting access.

  • Trying to Push the Boundaries of GPTā€™s Vision: @flokyhuan inquired about using videos for fine-tuning language models and was informed by @solbus that fine-tuning is currently only available for text models, and while the GPT vision feature can describe images from a video, it canā€™t be fine-tuned for specific knowledge like sports rules.

  • ChatGPT Memory Feature Rollout Progresses: @lumirix confirmed that the ChatGPTā€™s feature for remembering past conversation details is being rolled out to both free and Plus users but noted that itā€™s only available to a small portion of users at this time.

Links mentioned:

OpenAI ā–· #prompt-engineering (23 messagesšŸ”„):

  • Prompt Engineering Basics Explained: @eskcanta outlined that good prompt engineering involves using precise language, giving clear instructions to the AI, and careful review of the AIā€™s output. When instructing the AI, focus on what to do instead of what not to do, avoiding conflicting instructions.
  • AI Text Adventures Streamlined: @drinkoblog.weebly.com advised @stealth2077 to use custom instructions like ā€œFocus on simple storytelling and character dynamicsā€ to keep narratives straightforward and avoid complexity, which the AI tends to default to in text adventures.
  • Navigating Platform Confusion: @beanz_and_rice humorously attempted to engage with ChatGPT on the Discord server, prompting @toror to respond with amusement at the unsuccessful effort.
  • API Infrastructure vs. Prompt Engineering: @darthgustav. clarified the difference between prompt engineering and API infrastructure to @kate.yanchenka, suggesting that the latterā€™s queries about automated budget calculations and dynamic data handling were related to software development rather than prompt engineering.

Links mentioned:

OpenAI ā–· #api-discussions (23 messagesšŸ”„):

  • Newbie Seeks Prompt Engineering Wisdom: @zzavior sought advice for getting started with prompt engineering. @eskcanta provided an extensive guide focusing on using clear and precise language, checking the output, and ensuring not to trigger conflicts with the AIā€™s capabilities or training.
  • Library Queries for Prompt Engineering: @kate.yanchenka inquired about libraries for prompt engineering to manage budgets, fit dynamic data, and handle AI model fallbacks. @darthgustav. clarified that the topic was more about AI software development than prompt engineering.
  • Conversation Assistance Request Goes Unnoticed: @beanz_and_rice attempted to initiate an interaction using Discord slash commands but failed, followed by a comedic outcry that prompted a reaction from @toror.
  • Crafting Lightweight Text Adventures: @stealth2077 asked for tips on making a text adventure less deep and thematic. @drinkoblog.weebly.com suggested using custom instructions to guide the AI towards simpler storytelling.
  • Joke Generation Confusion: @lisabkk45_48614 requested a joke, but @solbus directed them to use the official ChatGPT website instead of the Discord channel.

Links mentioned:

OpenAccess AI Collective (axolotl) ā–· #general (66 messagesšŸ”„šŸ”„):

  • MPS Support Acknowledgment and Clarification: @caseus_ expressed gratitude for MPS support in the axolotl project thanks to a GitHub pull request #1264 by Maxime. Confusion arose about the contributorā€™s Discord identity, and @yamashi (the right Maxime) clarified their involvement, noting the dependency on transformers merging changes and the PyTorch pull request #99272 as crucial for further development.
  • Tinkering with Yi-34b Training and eBay Finds: Users @le_mess, @cookiesowns, and @c.gato discussed various AI and non-AI topics ranging from slow loss decrease during Yi-34b training to an eBay link for an old tech product.
  • Exploring Model Adaptation and Enhancements: @yamashi suggested the potential benefits of porting models to Keras for wider hardware support, and @dreamgen and @c.gato discussed error handling and fixes related to Hugging Face checkpoint saving, in light of a pull request #1414 and a related issue #1452.
  • Queries on Cheapest LLM Endpoint Services: @le_mess inquired about affordable LLM endpoint services with responses pointing to local options like llamacpp, external services such as Together AI, and OpenRouterā€™s cost-effectiveness. Users mentioned JSON serialization issues with Basten and the need for custom configurations.
  • Discussion of Various Challenges Using LLMs: Issues like JSON serialization (@dangfutures), challenges with FP32 slowness (@yamashi), and need for additional documentation were discussed providing snapshots of technical hurdles and collaborative problem-solving occurring in the AI community.

Links mentioned:

OpenAccess AI Collective (axolotl) ā–· #axolotl-dev (8 messagesšŸ”„):

  • Converging on Chat Dataset Formats: @dctanner is coordinating with Hugging Face to standardize a chat dataset format named MessagesList, to streamline the various dataset formats emerging for fine-tuning chat models. They shared a link to the MessagesList proposal discussion and suggested creating a GitHub org and dedicated page for documentation.

  • Naming Conventions Matter: @dctanner emphasized the importance of a universal format name like MessagesList thatā€™s not tied to a specific app like ShareGPT or ChatML, which can be confused with the template rather than the JSON format itself.

  • Validation Challenges for MessagesList: @faldore acknowledged that although they like the idea of MessagesList, it poses challenges in validation because the concept of a ā€œconversation pairā€ is not easily described by JSON-schema.

  • The Ideal MessagesList Schema: @faldore proposed an ideal schema for the MessagesList format that includes optional system messages, tools/functions, source metadata, and a greeting message, ensuring user and assistant messages are paired, and the last message is from the assistant.

  • Benefits of the Suggested Schema: @faldore advocates for the proposed schema, arguing that it is more manageable, verifiable, and space-efficient, and enforces structured message pairing in datasets.

Links mentioned:

@dctanner on Hugging Face: ā€œAs the amount of datasets for fine tuning chat models has grown, thereā€™s beenā€¦ā€: no description found

OpenAccess AI Collective (axolotl) ā–· #general-help (26 messagesšŸ”„):

  • Tokenization Troubles in Axolotl: User @nafnlaus00 enquired about a method to verify that axolotl is tokenizing as expected. @dreamgen recommended inspecting the tokenizer config in the output directory, while @nanobitz pointed to a debug flag in the axolotl repository.

  • Transformers Update Might Fix Inferencing Issue: @thierry_lama reported a device error while trying to infer on a trained model using runpodā€™s GPU. @nanobitz suggested that it could be due to an issue with transformers and recommended updating.

  • Multilingual Capabilities Enhancement Attempt: @sadaisystems asked about improving a modelā€™s capabilities in a language other than English, receiving a response from @le_mess that pre-training is necessary for significant improvement beyond what LoRA can offer.

  • Inferencing with LoRA on the Fly: @wizmak sought a way to add LoRA adapters to a base model in real-time during inferencing, and @nanobitz confirmed that with Hugging Face, you can load the peft model, but was unsure of the command to unload it.

  • Model Parallelism with DeepSpeed Zero 3: User @mihai4256 sought assistance for a working deepspeed zero 3 config for model parallelism, noting that existing ones from the repo werenā€™t functioning as expected for this particular use case.

Links mentioned:

GitHub - OpenAccess-AI-Collective/axolotl: Go ahead and axolotl questions: Go ahead and axolotl questions. Contribute to OpenAccess-AI-Collective/axolotl development by creating an account on GitHub.

OpenAccess AI Collective (axolotl) ā–· #datasets (1 messages):

  • Duplicate Dilemma in Dataset Finetuning: @_rxavier_ inquired about identifying if a text has been previously used to train a model. They asked about techniques for determining model familiarity with a text, possibly by examining the modelā€™s response to an articleā€™s introduction.

  • The Impact of Training Data Overlap: Additionally, @_rxavier_ questioned the implications of finetuning a model using data that may overlap with its pretraining dataset. They pondered the potential negative effects of such overlap on the finetuning process.

OpenAccess AI Collective (axolotl) ā–· #runpod-help (1 messages):

  • Axoltl RunPod Compatibility with Vast.AI: User @dreamgen successfully used the Axoltl RunPod image on Vast.AI, reporting that it worked out of the box.

LangChain AI ā–· #announcements (1 messages):

  • LangChain Introduces Journaling App with Memory: @hwchase17 shared an early version of a journaling app that incorporates memory, using the LangChain memory module. The app is in an early stage and feedback is welcomed; it remembers information about users for future interactions, akin to the memory feature announced by OpenAI for ChatGPT today. Test the app here and check out the introductory video.

Links mentioned:

LangChain AI ā–· #general (59 messagesšŸ”„šŸ”„):

  • Seeking Assistance on LangChain with Android: User @grindmaster2512 inquired about the integration of LangChain with an Android application, and followed up with @812757029260230658 to seek solutions to this query.
  • Efficient Chunk Pre-processing for Embeddings: @swastikk asked whether chunk pre-processing (like removing white spaces) is necessary before creating embeddings. @johnny2x2 confirmed that removing superfluous text aids the process, especially with email data.
  • PDF Parser Search, Alternatives to Adobe API: @dejoma requested recommendations for a PDF parser that can split contextually, expressing dissatisfaction with Adobe APIā€™s limitations and seeking effective PDF API alternatives.
  • Calls to Improve LangChainā€™s Documentation Structure: @b0otable provided feedback to the LangChain team suggesting the improvement in documentation structure by reducing example redundancies and updating syntax to avoid inefficient navigation for users.
  • Dependency Issues with Pinecone and LangChain: User @segmentationfault. experienced dependency resolution errors when trying to update Pinecone Database to v2 with a LangChain dependency, prompt response and solutions were provided by @jacoblee93, a maintainer of LangChain.

Links mentioned:

LangChain AI ā–· #langserve (8 messagesšŸ”„):

  • Clarification on Langserve Scaling: @kjoth_00356 inquired about scaling Langserve to multiple instances and asked about the difference between hosted Langserve and Langserve. @veryboldbagel hinted at a deployment via hosted Langserve, leading to further clarification from @dachsteinhustler who pointed to Langsmith as part of the solution hosted at Langchain Platform, which is in early testing and might require an invite code.

  • In Search of NodeJS and Chain Integration: @_mauricemoss is looking for a way to expose a chain from a NodeJS app for use in a RemoteRunnable, but no solution has been provided within these messages.

  • Disabling Intermediate Steps in Playground: @dachsteinhustler expressed a need to disable the intermediate steps in the Langchain playground to prevent browser crashes caused by large base64 strings, resulting in a workaround that involves using RunnableLambda.

  • Connection Issues with k8s Cluster App: @ezelanza. described an issue where a connection is refused when attempting to invoke the OpenAI API through a k8s cluster-based application, mentioning that direct invocations to the back end work, but requests from the front end (React) fail, even with curl.

Links mentioned:

LangSmith: no description found

LangChain AI ā–· #share-your-work (1 messages):

  • Introducing Dewy and RAG with NextJS and OpenAI: @kerinin shared their contribution towards Dewy, an OSS knowledge base, along with a post detailing how to build a full-stack RAG application. The guide includes using NextJS, OpenAI API, and Dewy, aimed to minimize hallucinations and ensure accurate language model responses. Check out the blog post here.

Links mentioned:

Building a RAG chatbot with NextJS, OpenAI & Dewy | Dewy: This guide will walk you through building a RAG application using NextJS for the web framework, the OpenAI API for the language model, and Dewy as your knowledge base.

LangChain AI ā–· #tutorials (2 messages):

  • Seeking a Superior PDF Parser: @dejoma is looking for a PDF parser that can split contextually. Expresses discontent with Adobe API due to its low usage cap and lack of ā€˜pay-as-you-goā€™ option; is open to suggestions for robust PDF APIs.
  • Langchain Calculator Quest: @sougata is building a calculator using Langchain that interprets multiplicative operations as mul(a,b). Requests guidance on how to integrate a custom Python library for calculation with the modelā€™s augment function.

DiscoResearch ā–· #general (29 messagesšŸ”„):

  • Inquiry on Argilla Hosting Experience: User @drxd1000 is seeking advice on hosting a server for Argilla that can support multiple users for annotation, but there was no resolution provided in the messages.
  • Layer Selective Rank Reduction Methodology Discussed: @johannhartmann referenced their own implementation of ā€˜Layer Selective Rank Reductionā€™ to address continual training without forgetting, noting that ā€œThey basically figure out the statistically less relevant parts of the layers and use them as lora targets,ā€ and considering it more efficient than continual approaches. A related GitHub repository was mentioned but not detailed in the conversation: laserRMT.
  • Out of Memory Issue with lm-evaluation-harness: @philipmay faced an OOM error evaluating a mixtral model and was advised by @bjoernp to utilize multi-GPU support provided by lm-evaluation-harness, indicating two A100s might resolve the issue.
  • Search for German Toxicity Eval Dataset: User @sten6633 inquired about a German toxicity evaluation dataset and pondered the utility of translating ToxiGen, a dataset available on Hugging Face for implicit hate speech detection. The dataset mentioned can be found on Hugging Face, but requires agreement for access: ToxiGen.
  • Novel Computational Technique Teased: User @phantine hinted at a new method excluding MoE, briefly titled ā€œUniverses in a bottleā€ and hinted at a potentially radical claim: ā€œP=NP.ā€ A GitHub link associated with @phantineā€™s work was shared, but no specific details regarding the technique were provided: LargeWorldModel/LWM.

Links mentioned:

DiscoResearch ā–· #embedding_dev (5 messages):

  • BM25 + Query + Rerank Combo Wins: User huunguyen highlighted their effective use of BM25 with additional querying and reranking steps for search purposes, and reported that this method ā€œworks pretty good.ā€
  • Wikipedia in a Nutshell: huunguyen managed to index the entirety of Wikipedia, excluding non-essential content, and compacted the BM25 index into a sleek size of under 3GB.
  • In Search of BM25 Tools: sebastian.bodza inquired about the specific library huunguyen is using to implement the BM25 algorithm for their search index.

DiscoResearch ā–· #discolm_german (1 messages):

thomasrenkert: Is there an ETA for v2 of the German model? Or for the Mixtral variant?

CUDA MODE ā–· #general (1 messages):

  • GPU Shuffling for Experimentation: @joseph_en reported successful relocation of the Asus WS motherboard to the miner and is awaiting 16x PCI extenders. Theyā€™ve utilized older GPUs for their experiments and have transitioned the minerā€™s motherboard into the case, noting it handles 7B or 13B quantized models with a single 12G NVIDIA 3060 with ease.

CUDA MODE ā–· #cuda (9 messagesšŸ”„):

  • Cross-Compatibility Quest: @iron_bound kicks off a discussion about achieving binary compatibility for CUDA to run on HIP/ROCm platforms, referencing a Phoronix article on Radeon CUDA - ZLUDA.
  • CUDA for AMD GPUs? Meet ZLUDA: @muhtasham shares a GitHub link to ZLUDA, a project that aims to make CUDA run on AMD GPUs, sparking interest and a request for user experiences by @marksaroufim.
  • Emoji Enthusiasm: @muhtasham invokes the spirits of the tech world through well-selected emojis of Jensen Huang and Lisa Su.
  • Market Monopolies and AGI Speculations: @andreaskoepf humorously suggests that Microsoftā€™s purchasing strategy and a borked chip market could leave antitrust agencies unequipped against an AGI future.
  • Real-World Radeon Trials: _tvi_ shares their experience with Radeon VII and a Ryzen APU, including struggles with dynamic memory allocation causing kernel crashes when handling large PyTorch data chunks.

Links mentioned:

CUDA MODE ā–· #algorithms (3 messages):

  • Multidimensional Gated Recurrences Have Limitations: User @euclaise mentioned a constraint in multidimensional gated recurrences, stating that they require a DxCxN attention matrix which is quite prohibitive in cost, even with a small value for C.
  • Beyond Simple Linear Recurrences: @euclaise pointed out that prefix-sum-like scans have applications beyond computing simple linear recurrences, opening up a broader range of computational possibilities.
  • Twitter Insights on Computational Techniques: @euclaise shared insights on computational methods, including the use of maximal scans for sequences (y[t]=max(y[t-1], x[t])), by providing links to their Twitter posts: Tweet on computational methods and Tweet on maximal scans.

CUDA MODE ā–· #jobs (3 messages):

  • Generative AI Startup Hiring in Hyderabad: @gradman33 shared a job opportunity at an early stage Deep Tech Generative AI startup in Hyderabad, India, seeking talents in ML/Data/Research/SDE. Interested candidates can apply here.
  • Potential Spam Alert in Jobs Channel: @pudding0377 flagged a post by @gradman33 as possibly irrelevant or spam, calling for the attention of moderators.

Links mentioned:

no title found: no description found

CUDA MODE ā–· #beginner (9 messagesšŸ”„):

  • New Member Alert: @cs_os_05101 mentioned that they have a 4060 Ti.
  • Search for Engaging CUDA Books: @euclaise inquired about fun books related to CUDA, sparking a conversation about educational resources.
  • Shader Book Recommendation: @marksaroufim shared The Book of Shaders, a gentle guide to Fragment Shaders, as a possible fun read on a topic adjacent to CUDA.
  • Understanding User Expertise: After citing familiarity with shader programming, @euclaise clarified theyā€™re looking for materials directly related to compute shaders or CUDA, rather than frag shaders.
  • Looking for Fun in Learning: Both @marksaroufim and @euclaise concurred that defining literature as ā€œfunā€ can be subjective, but @marksaroufim suggested PMPP as the best educational resource on CUDA, albeit not necessarily fitting the ā€œfunā€ criterion.

Links mentioned:

The Book of Shaders: Gentle step-by-step guide through the abstract and complex universe of Fragment Shaders.

CUDA MODE ā–· #pmpp-book (7 messages):

  • Matrix Transposition Debate: @eporat asked if transposing one matrix in a multiplication could lead to fewer cache misses and thus faster computation. @andreaskoepf responded, advising that while sequential memory access could be advantageous, the benefits might be negligible compared to tiled access.
  • Practical Test Yields No Benefits: Responding to the query about transposing matrices to speed up multiplication, @jeremyhoward recounted his experience stating that transposing during tile creation had no observable effect on performance.
  • In-Depth Discussion on Transposition: @eporat clarified that an inplace transpose isnā€™t necessary; sometimes, one only needs to adjust indice ordering in the inner loop, suggesting an alternative to transposition.
  • Further Clarification Sought: @andreaskoepf questioned @eporatā€™s suggestion, implying that matrix elements are read transposed by default during multiplication, indicating a misunderstanding or need for further explanation on what @eporat meant by adjusting loop indices.

CUDA MODE ā–· #smol-hw (1 messages):

  • Apple Silicon gets its own ā€˜topā€™: User @marksaroufim shared a link to asitop, a performance monitoring CLI tool for Apple Silicon. It was compared to existing tools like top or nvtop, tailored specifically for Appleā€™s custom chips.

Links mentioned:

GitHub - tlkh/asitop: Perf monitoring CLI tool for Apple Silicon: Perf monitoring CLI tool for Apple Silicon. Contribute to tlkh/asitop development by creating an account on GitHub.

Latent Space ā–· #ai-general-chat (24 messagesšŸ”„):

  • Reka Model Announcement: @swyxio shared a link to a tweet about a new Reka model, creating a buzz in the community. The tweet can be found here.
  • Favorite VC Podcast Meets AI: @swyxio expressed enthusiasm for a VC podcast discussing AI topics, providing a link to the episode and highlighting its relevance to the community.
  • Exploring the BUD-E Voice Assistant by LAION: @swyxio discussed a new fully open voice assistant named BUD-E, developed by LAION, which is aimed to improve conversational experiences by being empathetic and context-aware. Details are available on the LAION blog.
  • What is an Agent?: In search of a definition for ā€œagents,ā€ @kaycebasques asked the community for insights. @slono described them as programs that aim to achieve goals with minimal user input.
  • Karpathy Leaves OpenAI: @nembal spotlighted news from The Information about Andrej Karpathyā€™s departure from OpenAI, stirring curiosity about the implications for the AI field. Background on the development of an AI product for automating tasks mentioned by @slono vaguely referenced AGI as a possible factor in the context of the departure.

Links mentioned:

LLM Perf Enthusiasts AI ā–· #opensource (2 messages):

  • Choosing the Right Mistral Model Size: @potrock asked about the appropriate Mistral model size to run locally on an M2 Max with 32GB, seeking community input.
  • Safe Model Sizing Advice: @natureplayer suggested that 4GB is a safe size for local execution on the mentioned hardware, while 8GB will not work, and 5GB might be possible but is not guaranteed.

LLM Perf Enthusiasts AI ā–· #openai (4 messages):

  • GPT-5 Speculation Quelled: User @res6969 humorously noted that the rumors of GPT-5 have been greatly exaggerated, indicating skepticism about its existence or imminent release.
  • Laughter is the Best Medicine?: Both @res6969 and @potrock shared lighthearted reactions with custom emoji and laughing-to-tears emoji, respectively, contributing to a jovial environment on the topic at hand.
  • A Memory Upgrade for ChatGPT: @potrock shared a blog post discussing new memory features being tested in ChatGPT that allow the model to remember user preferences and details across conversations, which users can manage conversationally or through settings.

Links mentioned:

Memory and new controls for ChatGPT: Weā€™re testing the ability for ChatGPT to remember things you discuss to make future chats more helpful. Youā€™re in control of ChatGPTā€™s memory.

AI Engineer Foundation ā–· #events (6 messages):

  • Weekly Meeting Kick-Off with a Sense of Humor: @._z announced the start of the weekly meeting with a playful note: šŸ˜„ DĆ©jĆ  vu.
  • Meeting Attendance Update: @juanreds informed they could not attend the weekly meeting, apologizing to the team.
  • Invitation to Co-host an AI Hackathon: @caramelchameleon asked if anyone is interested in co-hosting an AI developers hackathon, hinting at a collaboration opportunity with game developers before GDC this year.
  • Chance to Join Hackathon Online or Onsite: @caramelchameleon mentioned the possibility of attending the hackathon both online and onsite in San Francisco.
  • Eager Organizer Jumps In: @yikesawjeez expressed interest and requested to be contacted as they specialize in organizing hackathons, especially those associated with events in the Bay Area.

Skunkworks AI ā–· #general (2 messages):

  • Private Message Prompt: User @bondconnery requests a direct message with a simple ā€<@1117586410774470818> DM sirā€.
  • LLaVA Framework Inquiry: @CodeMan is seeking insights or experiences on integrating LLaVA with an SGLang server and SGLang worker, as opposed to using a standard model worker.