Topic: "sparse-mixture-of-experts"

Dec 11, 2023
12/11/2023: Mixtral beats GPT3.5 and Llama2-70B
Show details
mixtral-8x7b gpt-4 gpt-3.5-turbo llama-3 openhermes-2.5 llava-v1.5-13b-gptq mistral-ai openai huggingface sparse-mixture-of-experts fine-tuning quantization gpu-hardware transformers model-deployment open-source coding-datasets
Mistral AI announced the Mixtral 8x7B model featuring a Sparse Mixture of Experts (SMoE) architecture, sparking discussions on its potential to rival GPT-4. The community debated GPU hardware options for training and fine-tuning transformer models, including RTX 4070s, A4500, RTX 3090s with nvlink, and A100 GPUs. Interest was expressed in fine-tuning Mixtral and generating quantized versions, alongside curating high-quality coding datasets. Resources shared include a YouTube video on open-source model deployment, an Arxiv paper, GitHub repositories, and a blog post on Mixture-of-Experts. Discussions also touched on potential open-source releases of GPT-3.5 Turbo and llama-3, and running OpenHermes 2.5 on Mac M3 Pro with VRAM considerations.