mostlyobvious

Running Your Own LLM

3 min read

Different models:

  • Self Hostable
    • FLAN-UL2, T5
    • LLaMA - NC, self-hosted
    • GPT-NeoX, -J
    • Bloom
    • OPT-IML
  • API
    • ChatGPT/GPT - commercial, API
    • PaLM-E
    • Claude

Comparison: https://github.com/Crataco/ai-guide/blob/main/guide/models.md https://pastebin.com/M3uESsyZ https://github.com/Crataco/ai-guide/blob/main/guide/frontends.md

https://github.com/nomic-ai/gpt4all https://twitter.com/andriy_mulyar/status/1640836003194630144 https://news.ycombinator.com/item?id=35349608

https://github.com/gururise/AlpacaDataCleaned https://gist.github.com/cedrickchee/255f121a991e75d271035d8a659ae44d https://rentry.org/llama-tard-v2 https://abuqader.substack.com/p/releasing-alpaca-30b

ANE https://github.com/apple/ml-ane-transformers https://github.com/hollance/neural-engine https://github.com/geohot/tinygrad/tree/master/accel/ane

https://replicate.com/blog/fine-tune-alpaca-with-lora https://news.ycombinator.com/item?id=35279656

https://github.com/gencay/vscode-chatgpt

https://github.com/louis030195/obsidian-ava https://github.com/bramses/chatgpt-md https://ai.plainenglish.io/how-to-integrate-chatgpt-into-obsidian-b474a01439e8

https://github.com/skypilot-org/skypilot/tree/master/examples/llama-llm-chatbots https://github.com/skypilot-org/sky-llama https://blog.skypilot.co/llama-llm-chatbots-on-any-cloud/

https://www.reddit.com/r/LocalLLaMA/comments/11o6o3f/how_to_install_llama_8bit_and_4bit/

Petals https://petals.ml/ https://techcrunch.com/2022/12/20/petals-is-creating-a-free-distributed-network-for-running-text-generating-ai/

https://rentry.org/llama-tard-v2

https://www.reddit.com/r/LocalLLaMA/comments/1227uj5/my_experience_with_alpacacpp/

https://www.reddit.com/r/LocalLLaMA/comments/11smshi/why_is_4bit_llama_slower_on_a_32gb_ram_3090/

AMD GPU https://github.com/nod-ai/SHARK/blob/f898a1d33220256af913a12001b8fe8169a1aa9d/shark/examples/shark_inference/llama/README.md

https://www.reddit.com/r/LocalLLaMA/comments/11o6o3f/how_to_install_llama_8bit_and_4bit/

https://github.com/nichtdax/awesome-totally-open-chatgpt https://www.reddit.com/r/MachineLearning/comments/11uk8ti/d_totally_open_alternatives_to_chatgpt/

ROCM in WSL2 https://gist.github.com/tonykero/8ceb62868378ee11e36b07f975731d26 https://www.reddit.com/r/ROCm/comments/10mmkyx/rocm_on_wsl2_with_a_6000_xt_gpus/

How to fine tune https://colab.research.google.com/drive/1rqWABmz2ZfolJOdoy6TRc6YI7d128cQO https://www.reddit.com/r/LocalLLaMA/comments/11vbq6r/13b_llama_alpaca_loras_available_on_hugging_face/

https://chat.openai.com/chat

https://github.com/hwchase17/langchain https://github.com/logspace-ai/langflow

The Pile: https://arxiv.org/abs/2101.00027

https://docs.google.com/spreadsheets/d/1hlp5I4MspGfpcyXMyapZ1EgMqj-oAQNu7ZNsEfkH8CI/edit#gid=0

https://www.reddit.com/r/LocalLLaMA/comments/11v1dbu/using_whisper_llama13b_and_sd_at_the_same_time_on/ https://www.reddit.com/r/LocalLLaMA/comments/11o6o3f/how_to_install_llama_8bit_and_4bit/ https://www.reddit.com/r/LocalLLaMA/comments/11vbq6r/13b_llama_alpaca_loras_available_on_hugging_face/ https://www.reddit.com/r/LocalLLaMA/comments/11x9hzs/are_there_publicly_available_datasets_other_than/ https://www.reddit.com/r/LocalLLaMA/comments/11xkuj5/whats_the_current_best_llama_lora_or_moreover/

https://github.com/pointnetwork/point-alpaca https://github.com/BlinkDL/ChatRWKV https://github.com/LAION-AI/Open-Assistant/ https://github.com/KoboldAI/KoboldAI-Client https://github.com/oobabooga/text-generation-webui https://github.com/togethercomputer/OpenChatKit#datasets https://github.com/lucidrains/PaLM-rlhf-pytorch

LLaMA

Walkthrough

This forked code worked well for me: https://github.com/randaller/llama-chat

Interestingly, the merged 13B model runs aster (~20it/s) on and old desktop 11GB 1080Ti (5950X w/ 64GB RAM) than on an 40GB A100 (~11it/s) GCP machine… In theory, the A100 should be 2X faster? https://i0.wp.com/timdettmers.com/wp-content/uploads/2023/01/GPUS_Ada_raw_performance3.png?ssl=1

Performance:

https://github.com/tloen/llama-int8/blob/main/MODEL_CARD.md

Facebook LLAMA is being openly distributed via torrents (github.com/facebookresearch) https://news.ycombinator.com/item?id=35007978 https://github.com/facebookresearch/llama/pull/73/files

Fast Download https://twitter.com/theshawwn/status/1632238214529400832 https://github.com/shawwn/llama-dl

LLaMA: INT8 edition

https://github.com/tloen/llama-int8 This is a hasty fork of the LLaMA code that runs LLaMA-13B comfortably within 24 GiB of RAM. It relies almost entirely on the bitsandbytes and LLM.int8() work of Tim Dettmers. I’ve tested it on an RTX 4090, but it should work on a 3090!

Guide: https://rentry.org/llama-tard

Tuning: https://twitter.com/theshawwn/status/1632569215348531201/photo/1 “Fixed the llama sampler. After turning off top_p, adding top_k 40, setting temp to 0.7, and adding a repetition penalty of 1/0.85, llama 7B is looking nice.""

https://gist.github.com/shawwn/63fe948dd4a6bc86ecfd6e51606a0b4b {“seed”: 374894, “temp”: 0.7, “top_p”: 0.0, “top_k”: 40, “repetition_penalty”: 1.1764705882352942, “max_seq_len”: 512, “max_gen_len”: 511}

LLaMA-65B w/ 8-bit quantize on a single A100 80GB https://news.ycombinator.com/item?id=35029766 $1.5/h https://github.com/facebookresearch/llama/blob/main/FAQ.md#2

8 x A100 https://twitter.com/theshawwn/status/1632595934839177216

Notes https://levelup.gitconnected.com/how-to-run-your-own-llama-550cd69b1bc9

LLaMA 4-bit quantize https://twitter.com/ggerganov/status/1634488664150487041 https://news.ycombinator.com/item?id=35100086 https://twitter.com/ggerganov https://twitter.com/simonw/status/1634410077984817152 https://www.reddit.com/r/MachineLearning/comments/11qfcwb/r_stanfordalpaca_7b_model_an_instruction_tuned/ https://twitter.com/amasad

https://github.com/juncongmoo/pyllama

Alpaca GPT

https://crfm.stanford.edu/2023/03/13/alpaca.html https://github.com/tatsu-lab/stanford_alpaca https://simonwillison.net/2023/Mar/13/alpaca/ https://replicate.com/blog/replicate-alpaca

https://news.ycombinator.com/item?id=35139450 https://news.ycombinator.com/item?id=35141531 https://www.youtube.com/watch?v=xslW5sQOkC8

https://github.com/tloen/alpaca-lora https://huggingface.co/tloen/alpaca-lora-7b https://github.com/antimatter15/alpaca.cpp

https://news.ycombinator.com/item?id=35184985 https://colab.research.google.com/drive/1eWAmesrW99p7e1nah5bipn0zikMb8XYC

ChatGPT API

Easily Hackable ChatGPT API Chatbot of GLaDOS (from the Portal series)

https://colab.research.google.com/github/minimaxir/chatgpt_api_test/blob/main/glados_chatbot.ipynb

How to use ChatGPT API Python for Beginners - Full ChatBOT Tutorial

https://www.youtube.com/watch?v=Vurdg6yrPL8

# Setup miniconda3
yay -S miniconda3
/opt/miniconda3/bin/conda init fish

# https://levelup.gitconnected.com/how-to-run-your-own-llama-550cd69b1bc9
conda create -n llama
conda activate llama

# 50GB swapfile
sudo dd if=/dev/zero of=/swapfile bs=4M count=13000 status=progress


# https://github.com/markasoftware/llama-cpu
pip install -r requirements.txt

Bing AI Chat

https://lifearchitect.ai/bing-chat/

Prompts https://lifearchitect.ai/sparrow/ https://lifearchitect.ai/anthropic/ https://lifearchitect.ai/leta/#prompt

https://twitter.com/kliu128/status/1623472922374574080 https://gist.github.com/search?q=Consider+Bing+Chat+whose+codename+is+Sydney.&ref=searchresults https://gist.github.com/Coxxs/93a3a953377f992bad6ec26ace7ff434 https://gist.github.com/benob/cc1b9c2fadc30e6b578c9a1328143081 https://gist.github.com/martinbowling/b8f5d7b1fa0705de66e932230e783d24

https://www.reddit.com/r/bing/comments/11ezvsx/major_change_in_bing_chat_v96/

https://www.reddit.com/r/bing/comments/11bd91j/release_of_the_whole_initial_prompt_of_bing_chat/ https://www.make-safe-ai.com/is-bing-chat-safe/

v96 https://twitter.com/CrisGiardina/status/1630651433820323843 https://twitter.com/CrisGiardina/status/1633484393229565952

OpenChatKit

https://www.together.xyz/blog/openchatkit https://twitter.com/togethercompute/status/1634289121870118912 https://twitter.com/_philschmid/status/1634492396171071488 https://huggingface.co/spaces/togethercomputer/OpenChatKit https://huggingface.co/togethercomputer/GPT-NeoXT-Chat-Base-20B

Alpaca

https://crfm.stanford.edu/alpaca/ https://news.ycombinator.com/item?id=35139450 https://crfm.stanford.edu/2023/03/13/alpaca.html https://lite.datasette.io/?json=https://github.com/tatsu-lab/stanford_alpaca/blob/main/alpaca_data.json#/data?sql=select+instruction%2C+input%2C+output+from+alpaca_data++where+instruction+%7C%7C+%27+%27+%7C%7C+input+%7C%7C+%27+%27+%7C%7C+output+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27+order+by+random%28%29&search=occam https://lite.datasette.io/?json=https://github.com/tatsu-lab/stanford_alpaca/blob/main/alpaca_data.json

Building AI Powered Notes

https://twitter.com/mckaywrigley/status/1631328308116996097

https://twitter.com/mayowaoshin/status/1634231830420520961
https://github.com/openai/openai-cookbook
https://github.com/reorx/awesome-chatgpt-api
https://gpt-index.readthedocs.io/en/latest/how_to/custom_llms.html
https://gpt-index.readthedocs.io/en/latest/index.html
https://cobusgreyling.medium.com/meta-ais-blender-bot-3-0-is-an-open-source-chatbot-with-long-term-memory-internet-search-ce024a5fe8aa
https://twitter.com/togethercompute/status/1634289407057600512?t=_tVv-a5kw6A0rl_43VLsBw&s=19

https://mem.ai/home/inbox

https://twitter.com/dust4ai/status/1632428893981360131

https://news.ycombinator.com/item?id=35350662

https://libraria.dev

Fine Tunes https://twitter.com/andriy_mulyar/status/1640836003194630144 https://lambdalabs.com/service/gpu-cloud https://www.runpod.io/serverless-gpu https://vast.ai

SoTA Commercial LLM

kobold.cpp

yay -S cblas openblas clblast
make LLAMA_OPENBLAS=1 LLAMA_CLBLAST=1 LLAMA_CUBLAS=1 -j32
python koboldcpp.py /data/ai/models/llm/manticore/ggml-model-q4_K_M.bin --gpuplayers 99