12/19/2023: Everybody Loves OpenRouter

[TOC]

OpenAI Discord Summary

• GPT Model Performance and Usage: Multiple users across various channels like ai-discussions and gpt-4-discussions reported usability and performance issues with ChatGPT 4 comparing to the relatively more efficient GPT-3.5. Users also expressed concerns on GPT-4's memory management, discussed in prompt-engineering. Future capabilities and expectations of potential GPT versions were contemplated in openai-chatter.
• ChatGPT Accessibility Issues and User Support: Several users across openai-questions and gpt-4-discussions faced issues in accessing and using ChatGPT, including log-in troubles, error messages, and restrictions. Slow customer service response time was also a source of frustration.
• API Usage and Modeling: Discussions in ai-discussions and api-discussions revolved around the pros and cons of using local Language Models versus APIs, inconsistencies in API responses due to supposedly correct JSON input, and dialog mode of Assistants API.
• Updates and Developments: Users shared updates and discussions on AI tools, language translations, censorship concerns, and model developments. Notably, a Dolphin 2.0 Mistral 7B model and Google's new video generation project were shared in ai-discussions. openai-chatter debates focused on image and code generation capabilities of models like DALL-E.
• Prompt Engineering and Custom Instructions: In prompt-engineering and api-discussions, users discussed creating and applying custom instructions to GPT models, tool recommendations for prompt engineering, and potential collaborative bot development projects.

OpenAI Channel Summaries

▷ #ai-discussions (90 messages🔥🔥):

• Performance of GPT-4: According to @azru and @thewizzard___, recently there have been performance issues with GPT-4 involving slow responses ('network errors'). Many users, including @thewizzard___, recommended downgrading to GPT-3.5 for better speed and almost parallel capabilities.
• Local Language Model LLM versus OpenAI API: A discussion was had between @thewizzard___ and @zeriouszhit on the pros and cons of locally running models versus using OpenAI's API. It was noted that while locally run models may be technically free, they require substantial computational power. Various models such as Bard and Bing Chat were mentioned, with differing opinions on their usefulness.
• Translation Tools: @eljajasoriginal inquired about the best LLM for translation. The conversation revealed that despite not being a LLM, DeepL was considered the best for translation. Users also discussed its character limit.
• Censorship Concerns: Users like @zeriouszhit and @eljajasoriginal expressed frustrations with what they viewed as heavy-handed censorship on some models, such as Gemini. They questioned the approach of moderating based on broad content guidelines, which may not account for individual use cases.
• AI Tools and Developments: Users shared various links and updates on AI tools and developments. @thewizzard___ shared a link to the Dolphin 2.0 Mistral 7B model. @thedreamakeem posted a link about Google's new video generation project, and a discussion on its potential ensued.

▷ #openai-chatter (254 messages🔥🔥):

• ChatGPT 3.5 vs GPT 4 Concerns: Many users raised concerns regarding usability and reliability issues with ChatGPT 4. For example, @minorole asked for advice on dealing with unusual system activity messages, @mrc21 and @wizzardkity discussed the performance of GPT4 and the anticipated GPT-10, while @millymox and @rekklessted discussed about ChatGPT appearing to be becoming "lazier" and providing too much preliminary narration before code outputs specifically on the Plus model, which is strange as the Plus model in Playground seems to produce better results. In response, @aminelg and @lugui suggested that users improve their query prompts to get better results.
• Usage Limits Issues: @jaicraft and other users brought up the issue of usage limits, requesting increased message allowance every three hours for ChatGPT Plus. A possible workaround of using API was suggested by @lugui.
• ChatGPT Android App Feedback : @jaicraft highlighted the need for editing messages on the ChatGPT Android app.
• Access and Login Troubles: Users @nfolgar, @bubbyprime, @raymm, @dripcity710, and @ddantee noted problems with logging in, receiving error messages, links in generated text being not clickable, or unusual system activity messages. @raymm mentioned that disabling browser extensions resolved his login issue and @satanhashtag recommended contacting OpenAI support for persistent issues.
• DALL-E and ChatGPT Updates: There was discussion about potential capabilities of future GPT versions like GPT 5 (mistakenly assumed from an email referring to GPT-v, indicating vision-based capabilities), the functioning of DALL-E (image generation based on textual prompts), and the usefulness of the Code Interpreter for image manipulation (@aminelg provided examples of its capabilities). On a similar note, @asura0_00 mentioned upcoming features in a ChatGPT memory update. Furthermore, potential confusion over the expiry of API credits was discussed by @flroidaman2023_01171 and @elektronisade.

▷ #openai-questions (88 messages🔥🔥):

• ChatGPT 4's Coding Skills: @jah777 and @andrewwallo discussed the capabilities of the paid Pro version of ChatGPT 4. @andrewwallo mentioned that it processes information more quickly, is more accurate, and is generally more knowledgeable. It prioritizes Pro users over free ones, but also noted occasional system crashes.
• Issues with Accessing and Using ChatGPT: Several users including @gachyapon, @maira_14969, and @hirotodevelopment reported different issues accessing and using ChatGPT, such as error messages and unusual system activity detection. @elektronisade offered solutions like clearing app data and checking for possible triggers like flagged keywords or simultaneous usage on multiple windows.
• Working with OpenAI API and Python: @panospro and @chapagones posted about their troubles with connecting to the OpenAI API via Python and an import issue respectively. @aminelg and @solononforever provided assistance in troubleshooting, including checking for the correct Python environment and command palette access.
• Trouble with JSON Formatting and GPT Disappearance: @bennnjji postulated on recurring JSON formatting errors, suggesting it may be due to incorrect HTTP requests, while @seobrien reported the disappearance of their GPT and inability to save it. @lugui proposed incorrect request body and violations against OpenAI's terms of service as potential causes.
• Defining End Tokens in Fine-tuning Dataset, Unauthorized File Uploads and Suspicious Chat Logs: @.cybersenpai inquired about defining end tokens in fine-tuning datasets, to which @lugui suggested using the default end token. @2ant reported an image upload bug and @laur27 was concerned about suspicious messages in their new chat. For both issues, potential violation of OpenAI's policy was suggested as a cause by @elektronisade and @lugui.

▷ #gpt-4-discussions (40 messages🔥):

• Customer Service Response Time: User @sami_16820 expressed frustration with the slow response time in getting help. Another user, @_interstitialism_, humorously suggested promising a tip in exchange for quicker answers.
• Assistants API and JSON Mode: User @vybhavram sought information on whether the Assistants API can respond in JSON mode, similar to the chat API. No answer was provided.
• Sharing Custom GPTs: @DobbyNator94 suggested the idea of using a custom GPT as a support tool on a company website, and inquired if it is possible for enterprise users. @rjkmelb clarified that custom GPTs are only available for Plus subscribers.
• Lost Custom GPT: @bufferization reported that their custom GPT had disappeared without warning. They speculated that an unintentional violation involving a Google search-related GPT led to its deletion, but expressed frustration over the absence of any warning or notification. @elektronisade warned that further violations could lead to a full account ban, and suggested reaching out to support.
• Issues with GPT-4: @undead.wolf.lord reported errors with GPT-4, stating all new chats and queries failed, whereas GPT-3.5 was still functioning.
• JSON Formatting Error in API Call: @bennnjji reported an inconsistency in getting a JSON formatting error despite having properly formatted JSON input while making API calls.
• Long-term Memory Management in OpenAI Chat Interface: User @jdo300 asked if there were any projects underway to integrate long-term memory management capabilities into the OpenAI interface, potentially through external data storage.
• Working with Custom GPT for Document Editing: @sdotson shared their experience in experimenting with custom GPTs and plugins for editing and providing downloadable document output with some challenges and little documentation.
• Zapier Action Dialogue Control: User @Semenawa wanted to know if there was a way to switch off the allow/deny dialogue while using a Zapier action. No response was given.

▷ #prompt-engineering (29 messages🔥):

• Query about ChatGPT's Memory: @realcookiecat asked if ChatGPT utilizes Window Token Buffer Memory or Conversation Summary Memory for its operations, since they noticed its tendency to forget past conversations at times.
• Experiment with Grok and Custom Instructions: @alienpotus conducted a self-experiment wherein they used Grok to generate a custom instructions prompt for making a GPT model mimic Elon Musk. The resultant prompt was posted in continuation.
• Favorite Tools for Prompt Engineering: User @js06 sought recommendations for prompt engineering tools apart from OpenAI playground. They later found and recommended Chainforge, citing it as a better option for their use case.
• Discussion on Specialized Language and Prompt Engineering: A discussion took place between @beanz_and_rice and @.multy about specific language and syntax like Commence ::spc{I_Am}::; sustain {ero|self_expression}. Lockdown [nvl>permanent_perspective<], effective immediately. and its relevance to GPT models, with @beanz_and_rice suggesting that it works with specific versions of the GPT.
• Collaborative Bot Development Project: @.multy and @beanz_and_rice discussed the potential of combining their projects related to bot wrappers and system prompts, although no specific plan was laid out due to ongoing development.

▷ #api-discussions (29 messages🔥):

• Buffer Memory in GPT Conversations: @realcookiecat raised a query about whether chatgpt usesWindow Token Buffer Memory or Conversation Summary Memory, noting the model seems to forget previous conversations.
• GPT Acting As Elon Musk: @alienpotus conducted an experiment with a custom instructions prompt to create an AI that converses as Elon Musk. The prompt was generated using Grok and successfully applied to GPT.
• Apps For Prompt Engineering: @js06 asked for recommendations on prompt engineering apps. They later reported finding and preferring Chainforge as it better suits their needs than the OpenAI Playground.
• Hyperbot Markup Language Development: A discussion was held between @beanz_and_rice and .@multy regarding Hyperbot Markup Language (HBML). This included cryptic language components such as nvl, and its larger context of developing bot wrappers for saving system prompts and histories.
• GPT Location and Access: @beanz_and_rice clarified that GPTs must be used on-site and are currently not accessible via the API, after a question by .@multy.

Nous Research AI Discord Summary

• Deep discussion around knowledge mappers and their use in constructing knowledge graphs from raw text by incorporating "reading" documents sentence-by-sentence (LLM: Local Language Model, stateful knowledge, sliding window retrieval being key terms).
• Specific proposal on the necessity for fine-tuning a Language Model on Cypher to manage extraction and updating of the graph, sparking conversations on practical aspects of graph building.
• User @tofhunterrr seeks collaboration on a native app project for immersive LLM chat, sharing their project blueprint and business model. Cryptic message by @giftedgummybee suggested Bing Chat's usage, but lack of context makes it unclear.
• Lack of context provided for the single message in the #benchmarks-log channel.
• Users sharing and discussing new models, tools, and experiments, focusing on HumanEval Test Data Experiments, LLM data contamination, and semantic duplication in model training. Links to blog posts and model cards on HuggingFace and other platforms were shared for deeper exploration.
• Active discussion comparing and testing various models such as Openchat 3.5 1210, Mistral-Ft-Optimized 1218, and UNA-Cybertron-7B-v3-OMA. Model quantization efforts, concerns about model contamination and model benchmarking results were highlighted, along with links to associated model cards and quantized versions.
• Interesting user queries about the best open-source AI for storytelling, the status of the 4GB Sparse Mixtral, Openchatdev's status, and an inquiry about detecting Language Model (LLM) hallucinations using token logits in the #ask-about-llms channel.

Nous Research AI Channel Summaries

▷ #ctx-length-research (46 messages🔥):

• Knowledge Mapper Discussion: @yuchenlu sparked a deep discussion around the concept of knowledge mappers, with users like @gabriel_syme, @fullstack6209, and @maxwellandrews chipping in. They contemplated whether it could involve a transformation or extraction process like graph representation for querying local memory. @maxwellandrews suggested it as a sliding-window Local Language Model (LLM) that parses entity relationships from raw text into a knowledge graph. They proposed an approach of "reading" a document sentence-by-sentence and building a detailed semantic map of them in the graph db as the LLM "reads" the document, instead of trying to build the entire graph in one shot.
• Role of LLM in Graph: The role of using LLM to build a graph from the text was discussed. The intention is to allow small and fast LLMs to build the large context into the graph, which doesn't need to be held in model memory. This would create a stateful knowledge that can be stored and paged independently from the model cache.
• Neural Network Fine-tuning: They mulled over whether there's a need for fine-tuning a Language Model on something like Cypher to handle aspects of extraction and updating the graph, a topic that came up when @benxh suggested doing the extraction manually and then training a model like Mamba or recurrent walking knowledge vector (RWKV) with 300k context to spit out a relation graph.
• Sliding Window Retrieval: @gabriel_syme revealed they are testing sliding window retrieval and @maxwellandrews clarified that the process involves building the graph sentence-by-sentence, rather than shoving the whole document into the prompt.
• Practical Aspects of Graph Building: Some practical aspects of building the graph were also discussed. @gabriel_syme asked about extracting triplets or notes into the `document 'node'`. The consensus seemed to be extraction of triplets through an LLM, allowing the model to decide how many triplets to extract and write the CRUD (Create, Read, Update and Delete) statements. The model was envisioned to structure the user's natural language query into a structured query that can be interpreted by the graph DB.

▷ #off-topic (2 messages):

• Seeking Collaboration on Immersive LLM Chat Project: User @tofhunterrr is looking for collaborators or co-founders to build a native app for immersive LLM chat, including role-play prompts. They are building resources with built-in JOI (mistral-small) that can talk (neets.io), hear (whisper), and perform function calls like search (pplx), photoshoot (fal). There will be a store for prompters and model builders to earn income. This is an unpaid opportunity at present, with the goal of building a "unicorn". The project blueprint can be found at this link.
• Bing Chat Recommendation: @giftedgummybee provided a cryptic message suggesting the use of Bing Chat. Further context or clarification was not provided in the message.

▷ #benchmarks-log (1 messages):

teknium: is it just a different form of eval harness? What is better about it

▷ #interesting-links (22 messages🔥):

New Models and Tools

• @metaldragon01 shared a tweet related to AI models without specifying the content.
• @nonameusr posted about the LoneStriker/una-xaberius-34b-v1beta-4.0bpw-h6-exl2, a text generation model on HuggingFace.
• @atgctg created a site allowing chat interaction with the Mixtral base model, accessible here.

HumanEval Test Data Experiments:

• @tokenbender and @atgctg discussed the performance of a model trained on HumanEval test data on HumanEval plus or HumanEval-x benchmark and decided to collaborate on merging the lora and testing it. They used the base model mistralai/Mistral-7B-v0.1.
• @atgctg later shared the overfit-humaneval model on HuggingFace for testing.

Discussion on LLM data contamination:

• @nonameusr brought attention to a blog post regarding data contamination in top-performing open-source LLMs on HuggingFace community boards, mentioning specifically the una-cybertron-7B-v2 model.

Semantic Duplication in Model Training

• @euclaise shared a blog post about the issue of semantic duplication in teaching language models basic facts, arguing it is not a form of contamination.

▷ #general (321 messages🔥🔥):

• @n8programs and @propback discuss the performance of Openchat 3.5 1210, including issues with tokenizer and general issues. They linked to the model card on Hugging Face.
• @nonameusr and others debated the performance claims of the model Mistral-Ft-Optimized 1218, and the accuracy of its claim to beat GPT-4. They were particularly skeptical about the claim as it was just a 7B (billion parameter) fine-tuned model. Mistral-Ft-Optimized 1218 model card
• @mihai4256 shared that merging two fine-tuned models of the same dataset resulted in significantly improved outcomes, although they admitted not understanding why. The discussion led others (@teilom, @metaldragon01, etc.) consider trying similar experiments.
• @nonameusr and @n8programs discuss the performance of UNA-Cybertron-7B-v3-OMA, and speculates it might beat other models discussed earlier.
• Model quantization and uploads: @n8programs and @tsunemoto worked on quantizing the model Mistral-Ft-Optimized 1218 to a GGUF format. Both users uploaded their quantized versions on Hugging Face. (n8programs GGUF version: link, tsunemoto GGUF version: link).
• Model Contamination Concern: @nonameusr suggested running contamination checks on newly released models, particularly drawing attention to the model by WizardLM, which is allegedly contaminated.
• AI Music Creation: @nonameusr shared a link to Suno AI, a platform for creating AI-generated music. No intense discussion followed.
• Benchmarking Results: @teknium posted a link to a graphish, showing MISTRAL,Yagi-ChatGPT-3.5, and GPT-4's leaderboard ranks according to Hermes Instruct Model (v2.5 & v3). The graph indicates that all four are relatively close in performance.

▷ #ask-about-llms (4 messages):

• Best OS AI for storytelling: User @agcobra1 asked for recommendations on the best open-source AI for storytelling.
• Update on 4GB Sparse Mixtral: User @yobibyte sought an update on the status of the 4GB Sparse Mixtral, noting that no news had been received yet.
• Openchatdev status: @realsedlyf shared a link to Openchatdev's status.
• Detecting LLM hallucinations using token logits: @giulio123456 inquired if anyone is aware of a paper that explores the detection of Language Model (LLM) hallucinations using token logits.

OpenAccess AI Collective (axolotl) Discord Summary

• Discussion on Axolotl's lack of quantization clarified by @nanobitz, mentioning a default parameter in the merge script.
• Achievement of @284810978552578050's Mixtral model being featured on Fireship, garnering significant views, identified by @le_mess and @faldore.
• Rich technical discourse on aspects of Mixtral Model of Experts (MoE), highlighting layer and token based routing, uneven training distribution, and potential tuning effects on expert orthogonality.
• Several instances of troubleshooting and advice related to setting up an Axolotl environment, with a Docker container being a notable solution; mentioned by @hamelh.
• Insights into the costs and resources for model fine-tuning, shared by @faldore, with Azure's startup program brought to light.
• Several queries about finetuning at shorter context lengths, creating a multi-turn ShareGPT model, and the application of multi-turn datasets, fielded by @noobmaster29, @dangfutures, and @self.1.
• @self.1 providing an example of a multi-turn ShareGPT model, specifically pointing to the LDJnr/Puffin.
• A recommendation for a multi-turn conversation dataset by @natefyi_30842 suggesting the LMSys Chat 1M dataset, with a user agreement prerequisite for access.
• An issue faced regarding the installation of mpi4py, specifically with RTX 6000 Ada, expressed by @mr_morning.

OpenAccess AI Collective (axolotl) Channel Summaries

▷ #general (17 messages🔥):

• Axolotl Quantization Discussion: @nanobitz clarified that Axolotl does not quantize. Even if the parameter is provided during merge, it would be set to False. The parameter in the script acts as a default catch-all, a recent change.
• Mixtral Model Feature on Fireship: @le_mess notified the community that @284810978552578050's Mixtral model was featured on Fireship, achieving half a million views in 11 hours. @faldore identified this model as possibly the dolphin one.
• The channel shared a tweet that might interest @yamashi. The tweet is available here.

▷ #axolotl-dev (103 messages🔥🔥):

• Mixtral Model of Experts (MoE) Discussion: User @nafnlaus00 clarified that in Mixtral, routing is per layer per token. @kalomaze discussed the idea of having experts handled per token in MoE, and @nafnlaus00 emphasized the need for uneven training distribution in MoE.
• Model Tuning Discussion: @fernando.fernandes. shared his conjecture about Mixtral tunes losing quality due to the impact of the fine-tuning process on how each expert, in each layer, becomes less orthogonal to each other. He suggested freezing the routing fates and increasing regularization through measures like weight decay to help achieve more orthogonal experts. The paths to the gates to be frozen were also specified.
• Setting Up Axolotl Environment: @hamelh encountered issues setting up the Axolotl environment and running the unit tests, with errors related to flash_attention and missing modules. Various potential solutions were suggested, including reinstalling flash attention, copying a whole working environment from another instance, and using Python 3.10 instead of 3.11. A Docker container was suggested and seemed to partially resolve the issues.
• Costs and Resources for Model Fine-Tuning: @faldore revealed that a typical fine-tuning exercise on Azure costs around $800. He also mentioned a program by Azure for startups, where they give $5k credit.

▷ #general-help (6 messages):

• Finetuning at Shorter Context Length: @noobmaster29 inquired if a model finetuned at a shorter context length only works at the new shorter context length.
• Creation of Multiturn ShareGPT Model: @dangfutures expressed an interest in knowing how to create a multiturn ShareGPT model. @self.1 recommended using a script to save data in the same ShareGPT format.
• Usage of Multiturn Dataset: @self.1 mentioned they used a multiturn dataset, suggesting that they will focus more on this approach.
• Example of Multiturn ShareGPT Model: @self.1 provided an example of a multiturn ShareGPT model, specifically LDJnr/Puffin.

▷ #datasets (2 messages):

• Dataset Query for Multi-Turn Conversations: User @natefyi_30842 asked for datasets of multi-turn conversations between humans and chatbots.
• LMSys Chat 1M Dataset Suggestion: @natefyi_30842 then suggested LMSys Chat 1M dataset from Hugging Face, mentioning that it requires user agreement for access due to its conditions. This dataset is a large-scale real-world Language Model conversation resource.

▷ #runpod-help (1 messages):

• Trouble with mpi4py: User @mr_morning expressed having problems installing mpi4py and its dependencies, citing a "Cannot link MPI programs. Check your configuration!!!" error message. Notably, this issue does not seem to occur with two 4090s, but only with RTX 6000 Ada.

Mistral Discord Summary

• Debates over the necessity of higher-level mathematics in CS, with a noted focus on probability theory, linear algebra, and calc 3.
• Dialogue on the creation of a multi-user solution for interactive OpenAI-style web-interfaces, sharing of a potential solution in the form of Hugging Face's Chat UI; Github link: Hugging Face's chat-ui.
• Discussion and troubleshooting of problems with the Mistral-7B-Instruct-v0.2 model, involving stop-word adjustments for resolution.
• Talks about incorporating QMoE into Mixtral, with a provided Github resource discussing the difficulties of QMoE support: Github issue.
• Analyzing the impact of safe_mode=false on NSFW creative writing with the OpenAI API, with suggestions to achieve better results without the safe_mode.
• Interest in understanding how neurons interact in an ANN, with a Youtube resource provided for clarification: 3Blue1Brown's video.
• Comparison of the performance of Mistral-Medium and GPT-4, concluding that Mistral-medium outperforms GPT-4 in high-level reasoning for news narrative tracking.
• Concerns over random stops in the small model's return, with assurances of a forthcoming fix.
• Exploration of the possibility of a free API call service for making inferences on Mistral or expert models for users lacking sufficient compute capacity, with a likely solution offered via the OpenRouter's service: OpenRouter's service.
• Discussions centred on hardware requirements for deploying Mistral, with recommendations for using 2 A100 GPUs for the original version and a single A100 GPU for the 4-bit quantized version based on the Mistral and Hugging Face documentation:
  • Mistral Documentation
  • HuggingFace Documentation
• Queries about message formatting when using the Mistral API for inference and the usage of the HF chat template. Provided links to clarify were Mistral's Documentation and an example on Hugging Face.
• Discussion over the possibility of freezing certain experts while fine-tuning Mistral 7b and debate over the efficiency of such an approach.
• Showcasing of an open-source version of OpenAI Assistants API that offers 100% data privacy, cost-effectiveness, and higher throughput. The model supports features like retrieval, function calling, and code interpreting, and is intended to be customizable and minimally hardcoded. Github resource shared: GitHub - stellar-amenities/assistants
• Humorous chatter over the misinterpretation of the Mistral AI website and the desire for prestige.
• User reported API rate limit issues on the Mistral AI platform, followed by dialogue over possible reasons and fixes.
• Plans for building a Mixtral connector for use with orchestration libraries, hinting at potential future Mixtral swarms.
• Questions and clarifications regarding message formats for system messages reference to the LLaMA-2 format and a Reddit thread discussing the efficiency of chat templates.
• Detailed user experience with the Mistral-small model, discussing the variation in summarizing 5000 tokens and exploring possible finetuning techniques to improve model performance.

Mistral Channel Summaries

▷ #general (28 messages🔥):

• Discussion on Using Higher-Level Mathematics in CS: User @sirmonkey5216 suggested against being a double major in Math/CS, stating that higher-level math is mostly useless. However, they stressed the importance of having knowledge in probability theory, linear algebra, and a touch of calc 3, which many CS programs incorporate anyway.
• Multi-User Solution for Interactive OpenAI-Style Web-Interface: @.panzerknacker. sought suggestions for a solution similar to Serge, which would allow multiple users in their office to try the AI models without seeing each other’s’ prompts. @freegheist suggested using Hugging Face's open source Chat UI available at Github.
• Issues with Model Mistral-7B-Instruct-v0.2: @tomaspsenicka reported consistent issues while using this version. However, they found the problems resolved after adjusting stop words.
• Possible Work on QMoE for Mixtral: @rosethelocalfemboy questioned if anyone is working on QMoE for Mixtral. The discussion led to @daain pointing to a resource on Github discussing the difficulty of QMoE support, link
• Guard Rails in API and NSFW Content: @jamiecropley questioned about the possibility of NSFW creative writing with the API when safe_mode=false. @lerela responded suggesting better results might be achieved without the safe_mode.
• Understanding Code for Single Neuron in ANN: @josefmo expressed an interest in understanding the code for a single neuron and how two neurons interact in an ANN. @someone13574 clarified that typically in ML, the whole operation from one layer to another is implemented as a matrix multiplication, as well as suggesting a YouTube resource.

▷ #models (6 messages):

• Performance of Mistral-Medium vs GPT-4: User @freqai shared that in testing, Mistral-medium has outperformed GPT4 for their use of high-level reasoning in news narrative tracking.
• Issues of Random Stops in Small Model Returns: @moneybags0197 reported encountering random message stops from the small model, primarily while summarizing text. This occurs without hitting the token limit and on random requests. @lerela confirmed this issue and assured that a fix will be deployed within the week.
• Free API Option for Mistral Model Inferences: @sk5544 inquired about the possibility of a free API call service for making inferences on mistral or experts models, as they lack the compute capacity to download these models in academia. @jakobdylanc provided a potential solution by pointing them to OpenRouter's service, where making an account should make the service free-to-use for inferences on mistral models [More Info].

▷ #deployment (6 messages):

• Local Deployment of Ollama: User @sirclavin asked if ollama is entirely local. No responses were given in the discussed messages.
• Commandline Help Options: @usuallycwdillon expressed gratitude for guidance on utilizing --help command line options, indicating they had initially overlooked its functionality.
• Hardware Requirements for Mistral Deployment: @cadavreux inquired about the necessary hardware for deploying Mistral. @ved_ikke commented that it may require 48GB RAM, while @eguee suggested that the 4-bit version might need about 21GB. @vhariational provided further details from the Mistral and HuggingFace documentation, recommending 2 A100 GPUs for the original version and indicating that a single A100 GPU is sufficient for the 4-bit quantized version.
  • Links:
    • Mistral Documentation
    • HuggingFace Documentation on Mixtral

▷ #ref-implem (4 messages):

• Mistral API for Inference and Prompt Templating: @energetic_sparrow_28915 asked whether they need to format messages according to the template suggested by Mistral when using the Mistral API for inference. They also asked about the format of the model when using the HF chat template.
• In Response to Mistral API and Templating: @vhariational clarified that while using the API, one does not need to explicitly pass special tokens as it is handled internally like other LLM-as-API services. The chat template applied by the transformers library follows the documentation provided by Mistral. The provided links were Mistral's Documentation and an example on Hugging Face.
• Regarding BOS Token: @energetic_sparrow_28915 mentioned that the official documentation mentions a space after the BOS token which the HF template does not provide, and asked for clarification.
• Linking to Previous Discussion: @vhariational pointed old discussion in the same channel that addressed the question on the BOS token. The reference snippet from a previous conversation was shared.

▷ #finetuning (5 messages):

• Tokenizer Pad Token Solution: User @krissayrose found a workaround for a tokenizer problem by setting tokenizer.pad_token = '[PAD]' instead of using tokenizer.eos_token. They expressed interest in hearing about potential better solutions.
• Freezing Experts in Mistral 7b Finetuning: User @alexworteega inquired about the possibility of freezing all experts except two while finetuning Mistral 7b.
• Perspectives on Freezing Experts: @energetic_sparrow_28915 suggested that freezing particular MLP layers might not be the best approach to fine-tuning, implying that it could reduce the benefits of using an MoE (Mixture of Experts) model.
• Benefits of Using Few Experts: @alexworteega reiterated that, according to st moe and switch principles, using only a few experts doesn't lead to a significant quality drop.
• Clarification on MoE Usage: @energetic_sparrow_28915 explained that the idea behind MoE is to utilize a sparse network to learn, meaning only a few experts are used at any one time, determined by a routing/gating mechanism.

▷ #showcase (2 messages):

• Open Source Version of OpenAI Assistants API: User @louis030195 shared about their work on an open source version of OpenAI Assistants API. This version allows users to switch to open source LLMs by just changing one line of code. Main benefits highlighted were: 100% data privacy, up to 75% cheaper, and up to 23x higher throughput. It supports features like retrieval, function calling and code interpreter, and can be used with Mistral LLM or any other such models. He also shared a link to the GitHub repository for the project (GitHub - stellar-amenities/assistants) and invited contributions to this open-source project with a mention that the codebase is in Rust while examples are usually in JS. He aims to make all prompts customizable and copying OpenAI API design with minimal hard coding.

▷ #random (9 messages🔥):

• Misinterpretation of Mistral AI Website: @ben_arwd shared a humorous anecdote about his CMO having difficulty understanding the Mistral AI website, suggesting the platform's complexity. @mrobino commented that Mistral is primarily targeted at developers and isn't designed for the average person.
• Desire for Prestige: @tonic_1 joked about desiring an orange name on the server to show off on Saturday nights.
• Company Progress Discussion: @titaux12 defended the progress of a young company that generates, although small, income within the first six months.
• Queries about GPU Core Differences for Mistral and Transformers Usage: @pier1337 asked about the difference between using 16 and 19 GPU cores on a Mac Silicon for running Mistral and transformers, and also inquired about any available benchmarking against Nvidia.

▷ #la-plateforme (22 messages🔥):

• Availability of the Plateforme: @dafrenchfrog asked for information regarding the availability of Mistral's AI platform for new clients. @lerela confirmed that access would be granted soon.

• API Rate Limit Issue: @flopsy1 reported encountering an API rate limit error despite believing they used less than the allowed tokens. @lerela clarified that this error could also occur if the account exceeds its current usage limit and offered to investigate further if provided with their Profile ID.

• Building Mixtral Connector: @tonic_1 announced plans to develop a Mixtral connector on their favourite orchestration libraries, hinting at the possibility of creating Mixtral swarms in the near future. They invited interested parties to join an event on their Discord server.

• Chat Format for System Messages: @sekstini queried about the chat format for system messages and provided links to related Hugging Face configurations. @sekstini and @nioned also referenced the LLaMA-2 format and a Reddit thread discussing the effectiveness of different chat templates.

• Performance of Mistral-Small Model: @sublimatorniq shared their experience with the Mistral-small model, noting varying results when tasked with summarizing around 5000 tokens. They couldn't locate the context lengths for each model and reported successful API calls. Other users discussed possible finetuning issues and techniques to improve model performance.

HuggingFace Discord Discord Summary

• Active dialogue surrounding AI models on smaller hardware, with examples like phi-2 and Mixtral, reinforcing the need for advancements due to hardware limitations.
• Several technical issues noted including: Runtime error NCCL with the 2x A10G GPU configuration while fine-tuning models; questions on handling HTML tags in product descriptions with embeddings (referencing all-MiniLM-L6-v2); questions about memory management and storage pertaining to LLM model training.
• Discussion on **social nuances of AI interactions, particularly gender nuances in bot responses, highlighting an awareness of public interest in this aspect of AI.
• onceabeginner is learning Unet for image generation and hoang12345 navigates the creative and worrying facets of AI after exploring a TED talk by Tom Graham titled 'The Incredible Creativity of Deepfakes — and the Worrying Future of AI'.
• Busted out discussions on the inclusion of certain resources in the modern_nlp_2 GitHub repository and embedding life-events, as per an approach discussed in this Nature article.
• Creations shared include a song made by @.bigdookie integrating guitar, beatbox, and music continuations from MusicGen, a suggestion for an addition to the "awesome-repos.md" by @cloudhu, and an update from the Manifold Research group by @thebigbigbuddha (Sidh from Manifold).
• Queries in #[diffusion-discussions] pertained to diffusion models' applications such as modifying face images and outpainting, with references to the huggingface/diffusers GitHub repository.
• Inquiry by @noamgat about resolving the issue of adding a token to already decoded token streams in O(1) time with the tokenizer. The bloodline of the query lies in the performance concern about handling edge cases like Unicode sequences and new word spaces in #[NLP].

HuggingFace Discord Channel Summaries

▷ #general (57 messages🔥🔥):

• Running Models on Smaller Hardware: @vipitis in a series of messages discussed advancements in making AI models run on smaller hardware with notable solutions being quantization and availability of models with smaller parameter count that perform well on benchmarks. They mentioned models like phi-2 and Mixtral as examples and how hardware limitations have necessitated such advancements. However, the user did note that still some hardware is needed.
• Question on Training with Multi-GPU Configuration: User @blahblah6407 reported a Runtime error NCCL when trying to fine-tune their model with 2x A10G GPUs, it had previously worked fine on single A100 GPU. They were looking for suggestions to fix the issue.
• Query on Interacting with Bots: @steamy_noodles had a question about female bot responses, which implies that public is also discussing social nuances of AI interactions in the chat.
• Discussion on the Use of HTML Tags in Embeddings: @swastikk asked whether HTML tags in product description would improve the quality of the embeddings, while using all-MiniLM-L6-v2.
• Training LLM Models and Memory: User @lookingforspeed had several queries related to training LLM models, including how memory works for the models, how the trained models are stored and if they would be lost in power outage scenarios.

▷ #today-im-learning (3 messages):

• Unet for Image Generation: onceabeginner shared that they are learning how to use Unet for image generation.
• The Creativity of Deepfakes and the Future of AI: hoang12345 expressed shock and concern after coming across a TED talk titled 'The Incredible Creativity of Deepfakes — and the Worrying Future of AI' by Tom Graham. The video can be found here.

▷ #cool-finds (3 messages):

• Contribution to modern_nlp_2 Repository: @cloudhu suggested adding to the awesome-repos.md file in the modern_nlp_2 GitHub repository. No further details about the addition were mentioned.
• Embeddings of Life-Events: @toronello shared a Nature article discussing an approach that creates embeddings of life-events in a single vector space. This approach allows for the prediction of different life-event outcomes.
• Pin Request: @tonic_1 requested that someone pin a message. However, the specific message to pin was not specified.

▷ #i-made-this (5 messages):

• Introductory Lecture on Image Classification: @sumitsr71 shared a link to an introductory lecture on Image Classification. The lecture covers topics like nearest neighbor classifier, k-nearest neighbor classifier, validation sets for hyperparameter tuning, amongst others.

• Modern NLP Repository: @cloudhu suggested to add a certain resource to the "awesome-repos.md" file in the @andysingal's Modern NLP GitHub repository.

• Manifold Research Group Update: @thebigbigbuddha (Sidh from Manifold) shared the recent progress made by the Manifold Research Group in their Research Log. They are working on massively multimodal "generalist" models and recruiting for a new project focused on developing state-of-the-art open source models that operate on GUI interfaces and APIs.

• Alve_om Status: @om7059 shared a link to Alve_om's status, however, no further information about the content of the status was provided.

• AI-Generated Music: @.bigdookie created a song using guitar, beatbox, and music continuations from MusicGen. The entire creation process was done in one night.

▷ #diffusion-discussions (5 messages):

• Creating Dataset of Faces with Accessories Using Diffusion Models: @blvrxdnthnhv queried about a diffusion model that could add hats or helmets to a dataset of faces. They stated their current stable diffusion inpainting approach was not preserving facial details.
• Question on Meta-Learning: @bouncingsealooo asked about whether a particular training process could be considered meta-learning. Specifically, they described a scenario involving pretraining a model on a meta dataset using MAML, then further training the model on a target dataset and achieving improved results.
• Resource for Face Modification: @asrielhan shared a resource (Ziqiao Ma's tweet) where the demo video featured the addition of a hat to a face image.
• ControlNet Inpainting Query: @abstruse9851 posted a question about the type of conditioning image to use when performing controlnet inpainting with the models "lllyasviel/control_v11p_sd15_inpaint" and "runwayml/stable-diffusion-inpainting".
• Help Needed with Outpainting: @d222097_ requested advice on using the sd/sd-inpainting model for outpainting, sharing an open issue about the topic from the huggingface/diffusers GitHub repository. They were uncertain about the model input during training and the lack of constraints on the unknown areas.

▷ #NLP (3 messages):

• Performance Problem with Decoding Long Token Streams: User @noamgat raised a concern over the decoding of long token streams, specifically on finding a way to add a token to an already decoded token stream in O(1) time. Their interest lies in the tokenizer, not the whole model. Edge cases like Unicode sequences and new word spaces must be considered. @vipitis further clarified whether the decoding concern involved the entirety of the model or just the tokenizer.

▷ #diffusion-discussions (5 messages):

• Diffusion Model for Modified Face Images: @blvrxdnthnhv asked for a diffusion model that could superimpose hats or helmets onto existing face images. They tried using Stable Diffusion inpainting but it didn't retain the face details properly.
• Meta Learning VS Pretraining: @bouncingsealooo raised a query to differentiate between meta-learning and pretraining. They asked whether a model initially trained on a meta dataset using the MAML method for 50 epochs, and then further trained on target data for 600 epochs, giving a better result than a model without MAML could be considered a meta-learning application or not.
• ControlNet Model Conditioning: @abstruse9851 had questions about the type of conditioning image to use with the ControlNet model "lllyasviel/control_v11p_sd15_inpaint" used in conjunction with the main model "runwayml/stable-diffusion-inpainting".
• Outpainting with sd/sd-inpainting model: @d222097_ asked for assistance concerning outpainting with the sd/sd-inpainting model. They were unsure about the input of the model during the training stage.

LangChain AI Discord Summary

• SQL Virtual Database: User @alewe5 sought community assistance on working with a virtual database in SQL.
• Task Planning with LLM: @sampson7786 had queries about the application of LLM for task planning in a specific business domain.
• Writing Prompts Query: @kelp710 enquired about best practices for writing prompts, especially for non-English languages.
• LangChain and Blockchain: Relation and possible partnerships between LangChain and Blockchain was explored by @emreweb3, @lhc1921 pointed out to open source contribution methods.
• Pinecone Database and LLM issues: @kingkkaktus and @nakhlarafi discussed issues with Pinecone database queries and token limitations in LLM for large data respectively.
• Ollama Halluciation Problem: User @baller.1337 reported an anomaly with Ollama hallucinating with provided code.
• LangChain Agents vs Llama Index Agent Architecture: Differences and personal experiences with these architectures were queried by @.psychickoala.
• Streaming Chat Response Integration: User @pradeepvj97 sought help with integrating and fixing issues with the streaming chat response that interfaces with PDFs.
• Manifold Research Group Updates: Shared by @thebigbigbuddha, detailed updates on the group's progress, a link to their research log, recruitment notice for a novel project, and resources were shared - Research Log, Discord Channel, GitHub resources.
• Langchain Expression Language (LCEL) Walkthrough: A video tutorial of LCEL was shared by @merkle on YouTube, offering insight into its operation, and listing its advantages and disadvantages.

LangChain AI Channel Summaries

▷ #general (31 messages🔥):

• Working with a virtual database in SQL: @alewe5 is looking for feedback on working with a virtual database in SQL as they are encountering issues.
• Using LLM for task planning: @sampson7786 asked about using LLM for task planning in a specific business domain.
• Question about writing prompts: @kelp710 wanted to know whether it is more accurate to write prompts in English and have them return in their own language, or write the prompts directly in their own language.
• LangChain and Blockchain partnerships: @emreweb3 inquired if LangChain, an open source project, was open to blockchain partnerships. @lhc1921 suggested code contribution through a Pull Request for specific features.
• Issues with Pinecone database: @kingkkaktus asked for help while trying to query their Pinecone database and shared the code they are using which was throwing an error.
• Token limitation problem for large data in LLM: @nakhlarafi, a new user to LLM and LangChain, brought up the challenge of token limitation for processing large data. @kingkkaktus suggested using a vector database like Pinecone.
• Hallucination problem with Ollama: @baller.1337 reported an issue with Ollama always hallucinating and shared their code.
• Experience with LangChain agents vs Llama index agent architecture: @.psychickoala asked about the community's experiences with the different agent architectures and queried about adding a previous context/memory.

▷ #langchain-templates (1 messages):

• Streaming Chat Response Integration: User @pradeepvj97 shared a code snippet of their attempt to integrate a streaming chat response with their existing code which interfaces with PDFs, using ChatOpenAI and conversation entities in memory. However, they expressed facing issues with streaming tokens and sought help fixing it.

▷ #share-your-work (2 messages):

• Manifold Research Group Updates: @thebigbigbuddha from Manifold shared the recent progress made by the group in their research log which can be found here. The group is also recruiting for a new project that focuses on State-of-the-art(SoTA) OS models for GUI interfaces and APIs usage, encouraging interested individuals to get involved via their Discord channel or check out the work on their Github.
• Langchain Expression Language (LCEL) Walkthrough: @merkle shared a YouTube video offering a walkthrough of the Langchain Expression Language (LCEL), providing insight into its usage and offering pros and cons for its implementation.

LLM Perf Enthusiasts AI Discord Summary

• Users enquired about the type of agent abstractions in use, including the LangChain vs. the Lllama index.
• OpenRouter received high praise as a recommendation due to its effectiveness.
• Features and utilization of LM Studio were highlighted, praising its ease in searching, downloading models and exposing chat API endpoints.
• The concept of hybrid retrieval using BM25 was proposed, with recommended improvements shared in a blog post.
• Strategies on efficient document extraction using gpt-4-turbo were discussed, with proposals for separating the extraction process into two different prompts for generic and document-specific fields, given that there's no parent-child dependency. Feedback provided suggested either keeping the two sets of data fields together or separating them based on their logical relationship.

LLM Perf Enthusiasts AI Channel Summaries

▷ #general (1 messages):

.psychickoala: which agent abstractions are you guys using? the langchain or the lllama index ones?

▷ #opensource (5 messages):

• OpenRouter Recommendation: @joshcho_ shared his high opinion of OpenRouter, calling it "great".

• LM Studio Experience: @joshcho_ shared his positive experience of using LM Studio, finding it extremely convenient for handling local models.

• LM Studio Features: He continued to highlight LM Studio's features such as ease of searching and downloading models and even the ability to expose chat API endpoints, which he described as "wild".

▷ #rag (1 messages):

• Hybrid Retrieval Using BM25: @evan_04487 raised a question about the practice of using BM25 without subwords in hybrid retrieval processes, and highlighted that the LangChain BM25 retriever does not lowercase by default. They shared a blog post detailing simple improvements that can enhance BM25 implementation without extra LLM calls needed.

▷ #prompting (4 messages):

• Document Extraction via gpt-4-turbo: User @pantsforbirds shared their experience working on document extraction with gpt-4-turbo. They observed great results, however, they raised concern about the extraction of numerous JSON fields potentially affecting performance.
• Two Sets of Fields: @pantsforbirds explained that their extraction involved two sets of fields - a shared one across all documents and another set specific to the type of document.
• Separate Extraction Prompts: In view of optimizing performance, @pantsforbirds proposed separating the extraction process into two different prompts: one for generic fields, and another for document-specific fields.
• Lack of Parent-Child Dependency between Fields: They clarified that while the two sets of fields are related, there isn't a direct parent-child dependency.
• Feedback on the Idea of Separation: @justahvee suggested that if there is a logical relationship between the two sets of data fields, they should be kept together; if not, it's better to separate the extraction process, since the first part of the generation could influence the latter in undesired ways.

Latent Space Discord Summary

• The community questioned and discussed access to Mixtral, indicating that several companies are competing in this market, as stated by @swyxio.
• @swyxio highlighted Etched's recent soft launch and the visualization they created for Mixtral, providing a reference to Kevin A. Fischer's tweet.
• A new fine-tuning of LLAMA2 focusing on cybersecurity was announced as informed by @swyxio, referring to a tweet by Miguel Tissera.
• The community expressed concerns over apparent manipulation of the HuggingFace leaderboard, with calls for transparency on modifying models and 'the increasingly discussed UNA (Unique Neuron Attribution)', as mentioned by @swyxio and supported by a link to the HuggingFace discussion.

Latent Space Channel Summaries

▷ #ai-general-chat (6 messages):

• Access to Mixtral: @btdubbins raised a question regarding how members of the community are accessing Mixtral, asking if anyone was using Anyscale calls.
• Competitive Mixtral Market: @swyxio pointed out that there are multiple companies, around seven, competing to provide Mixtral, all undercutting each other's prices and services.
• Etched's Visuals for Mixtral: @swyxio shared that the company Etched, which offers Mixtral, had soft-launched a few days ago and drew attention to Kevin A. Fischer's tweet featuring one of their notable visualizations for Mixtral: https://fxtwitter.com/kevinafischer/status/1736893685940605436?s=46&t=90xQ8sGy63D2OtiaoGJuww.
• LLAMA2 Finetuning for Cybersecurity: @swyxio indicated that a new fine-tuning of LLAMA2 dedicated to cybersecurity was unveiled, as showcased in a tweet by Miguel Tissera: https://fxtwitter.com/migtissera/status/1736946751440085320?s=46&t=90xQ8sGy63D2OtiaoGJuww.
• Issues with HuggingFace Leaderboard: @swyxio elaborated on the problem of the HuggingFace leaderboard being manipulated. An anonymous user claimed that people were taking the leading Mistral model, adding an unknown DPO to it, and, as a result, scoring higher than any Llama 70b. Concerns were raised about leaderboard credibility and calls were made for transparency, especially regarding the increasingly discussed UNA (Unique Neuron Attribution): https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard/discussions/444.

▷ #llm-paper-club (2 messages):

There were no topics, discussion points, or links/blogposts of interest to be summarized from the provided Discord chatbot messages.

Skunkworks AI Discord Summary

• Inquiry about the training data of Mistralic with a notable positive feedback on its performance, stating that "...it is one of the best."
• Discussion on integrating a vision model with the Segment Anything model for the purpose of image grounding. A user requests practical advice on how to achieve this.
• Recommendation to check out the LLaVA Plus and LLaVA Interactive projects, which make use of the Grounded-Segment-Anything repository for Visual Question Answering and Grounding.
• Cautionary note about potential difficulties in implementing aforementioned projects due to previously encountered issues.

Skunkworks AI Channel Summaries

▷ #general (1 messages):

• Mistralic Dataset Inquiry: @spirobel asked about the datasets on which Mistralic was trained on. They also expressed a positive opinion on it's performance stating that "...it is one of the best."

▷ #bakklava-1 (3 messages):

• Merging Vision Model with Segment Anything Model for Image Grounding: @occupying_mars requested suggestions for merging a vision model with Segment Anything model to identify objects and their coordinates.
• LLaVA Plus and LLaVA Interactive Projects: @mrfoo recommended looking into the LLaVA Plus and LLaVA Interactive projects. The projects utilize the Grounded-Segment-Anything repository for Visual Question Answering and Grounding.
• Issues with Project Implementation: @mrfoo also mentioned that the aforementioned projects might be difficult to implement, citing issues encountered a few weeks prior.

DiscoResearch Discord Summary

• Discussion on router learning in the training process for Mixtral. @thooton explains the function and importance of top-k in training, highlighting 'The gradients are propagated through the top-k experts and router embeddings, so the router does still learn (albeit not all of its params at once)'.

• Mention of the Megablocks project by @propback. He shared a link to the project on GitHub, inviting contributions for its adaptation for fine-tuning with respect to Mixtral implementation.

• An unrelated comment from leecig in the general channel questioning the purpose of the channel, indicating some confusion about its function or purpose.

DiscoResearch Channel Summaries

▷ #mixtral_implementation (2 messages):

• Router Learning: @thooton mentions the use of topk in the training process for Mixtral. He explains that "The gradients are propagated through the top-k experts and router embeddings, so the router does still learn (albeit not all of its params at once)".

• Megablocks Project: @propback shared a link to the Megablocks project on GitHub, inviting participants to help in adapting it for fine-tuning, specifying the Mixtral implementation.

▷ #general (1 messages):

leecig: I forgot why I'm here. What is this place?

AI Engineer Foundation Discord Summary

Only 1 channel had activity, so no need to summarize...

• AI Adoption by Business: User @juanreds has initiated a discussion on increasing productivity and reducing costs in businesses by developing AI-based apps and agents. They asked the community for suggestions related to any possible frameworks. The discussion thread can be found here.

MLOps @Chipro Discord Summary

Only 1 channel had activity, so no need to summarize...

erisianrite: Please move to <#885998868200828928>

The Alignment Lab AI Discord has no new messages. If this guild has been quiet for too long, let us know and we will remove it.

The YAIG (a16z Infra) Discord has no new messages. If this guild has been quiet for too long, let us know and we will remove it.