mostlyobvious

Practical LLMs

15 min read

This is a guide for using Large Language Models (Chat AI):

Services

ChatGPT

This is the basic ChatGPT service and is probably the thing most worth paying for. ChatGPT 3.5 is like a B student - it’ll mostly do what you ask for. Maybe not always in the most impressive manner, but usually at least competently. Using the Browsing mode IMO is key to minimizing hallucinations.

ChatGPT 4 is on another level. While slower, rate limited, and overly verbose (always ask it to answer “concisely” or ask it to shorten its answers), it does better reasoning, is much more nuanced, and hallucinates much less often, even when asking for citations or references.

Poe

This is a good app to play around with a few different models.

  • https://poe.com/
  • Free version on web and app
  • Gives access to Anthropic’s Sage, Claude and Dragonfly, as well as ChatGPT (3.5)
  • Paid version ($20/mo) via iOS App gives you access to Claude+ and GPT-4
  • Chat history is wonky, does arbitrary context clears, good to maybe experiment

Bing Chat

This is an easy free way to test out a GPT-4 based model

Google Bard

If you are logging in from the US (simple geo-restriction), then you can see what Google is offering. Google Bard launched using the LaMDA 2 model, but is getting aggressive upgrades w/ math and logic upgrades from PaLM.

  • https://bard.google.com/
  • https://twitter.com/JackK/status/1641803576816340998
  • NOTE: while the latest PaLM update is good, and I’ve noticed lots of improvements, it’s worth noting that Bard still hallucinates a lot and it doesn’t seem to be able to tell if it is hallucinating or not (eg, if you ask it to search or summarize a page it instead will make stuff up, and if you ask it if it is pulling the page directly it’ll tell you it is, and it can distinguish between whether it’s pulling an external source or not (it can’t))

OpenAI API

If you are at all technical, you can set up an API account and run alternative interfaces:

Note, currently most people only have access to chatgpt-3.5-turbo via API

ChatbotUI

IMO, the best UI available

ChatWithGPT

This can connect with Whisper and ElevenLabs to give you a close to “Her” experience. While not so useful outside of a demo experience, I could see it being fantastic for English learners

nat.dev

This recently shifted to paid-only (but incredibly cheap, for playing around with, worth $5 in credits) - Nat Friedman’s (Github ex-CEO) playground/comparison site is probably the best/easiest way to play around with (but not “use”) any models

Conclusions

  • gpt-4 is about 3-4X slower than gpt-3.5-turbo and text-davinci-0003
  • claude-instant is 3-4X faster than gpt-3.5-turbo
  • claude-instant provides generally good/usable output

Vercel AI Playground

https://play.vercel.ai/ This is a new comparison site similar to nat.dev that gives you access to OpenAI, Anthropic, Cohere, as well as several leading open source models. A nicer UI than nat.dev and gives you code front-end code for embedding as well.

Self-Hosted LLMs

Why

There’s a lot of reasons to want to self-host your own LLMs:

  • Privacy - obviously any interactions w/ a 3rd party API leaks your personal data. This should probably be the #1 concern for most people. Data contamination, use for training without consent is already a huge problem and it seems inevitable that interactions and privileged information will be (mis)used
  • Accessibility - related, and while for practical reasons, most people will give up, but depending on a third party service (where access can be arbitrarily revoked, the service can go down, etc) for core capabilities is something to consider. The dynamics/tradeoffs of self-hosting are likely to mirror how we treat our daily Internet services
  • Capabilities - while OpenAI has been aggressive about rolling out new capabilities, if you are a developer looking to push the boundaries of LLM capabilities (agentic AI, larger and long term memory, fine-tunes, etc,) then you may want to be looking into self-hosting, at least as a fallback for when APIs go down
  • Uncensored - while safety is a good thing, especially as these LLMs are being widely used/deployed, the number of limitations in the name of safety and LCD norms means restricting more and more output. While I agree that this is generally good from a social perspective and for a commercial service, the limitations are onerous and can be overly restrictive/overstepping bounds. Also they make the LLMs perform worse.
  • Decentralization - decreasing SPOFs, decreasing dependencies on single services/entities, balancing off political/social power from single entities all seem like good ideas. The idea of expanding the commons is a related thought there.

Rule of thumb:

Llama 33B appears to be capable of out-performing GPT-3, Gopher 280B, and Chinchilla 70B according to Table 3: Zero-shot performance on Common Sense Reasoning tasks from the LLaMA paper

ModelBoolQPIQASIQAHellaSwagWinoGrandeARC-eARC-cOBQA
GPT-3 175B60.581.0-78.970.268.851.457.6
Gopher 280B79.381.850.679.270.1---
Chinchilla 70B83.781.851.380.874.9---
PaLM 62B84.880.5-79.777.075.252.550.4
PaLM-cont 62B83.981.4-80.677.0---
PaLM 540B88.082.3-83.481.176.653.053.4
LLaMA 7B76.579.848.976.170.172.847.657.2
LLaMA 13B78.180.150.479.273.074.852.756.4
LLaMA 33B83.182.350.482.876.080.057.858.6
LLaMA 65B85.382.852.384.277.078.956.060.2

Fabrice Bellard has been using lm-eval to run standardized/validated results against open LLMs: https://bellard.org/ts_server/

I spent ~$90 (4.5M tokens) to run the same tests on text-davinci-003 and place it in a “Top 20” table: https://github.com/AUGMXNT/llm-experiments/blob/main/01-lm-eval.md

Chris Taylor (catid) has recently been running lots of experiments on local LLMs:

gpt4all

This is currently the easiest way I’ve found to play around with a self-hosted LLM. There is a GUI installer for Win/Mac/Linux that can load a variety of models. The best one is probably gpt4all-l13b-snoozy

CatAI (CPU)

This is a pretty easy way to install a CPU-based high-performing local chat

# Install nvm if you need
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.3/install.sh | bash
nvm install latest
catai install Wizard-Vicuna-7B
catai serve

Vicuna

Overall Vicuna does a very good job with responses, however due to its training, it is very strong imprinted as an AI assistant, which comes through even if you add pre-prompts or a “character sheet”.

We introduce Vicuna-13B, an open-source chatbot trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT. Preliminary evaluation using GPT-4 as a judge shows Vicuna-13B achieves more than 90%* quality of OpenAI ChatGPT and Google Bard while outperforming other models like LLaMA and Stanford Alpaca in more than 90%* of cases. The cost of training Vicuna-13B is around $300. The training and serving code, along with an online demo, are publicly available for non-commercial use.

conda activate llama

# Install
git clone https://github.com/lm-sys/FastChat.git
cd FastChat
pip3 install --upgrade pip  # enable PEP 660 support
pip3 install -e .

# Convert weights
python3 -m fastchat.model.apply_delta \
    --base /data/ai/models/llm/llama/13b/llama-13b-hf \
    --target /data/ai/models/llm/vicuna-13b \
    --delta lmsys/vicuna-13b-delta-v0

# Run - this actually requires something like 28GB of memory
python3 -m fastchat.serve.cli --model-name /data/ai/models/llm/vicuna-13b

To run in 8-bit mode it’s easier to use https://github.com/oobabooga/text-generation-webui :

conda activate llama
cd text-generation-webui
# symlink models folder
python server.py --model llm/vicuna-13b --load-in-8bit --model_type LLaMA --chat
# This should be about 14GB for the model...

See also: https://medium.com/@martin-thissen/vicuna-on-your-cpu-gpu-best-free-chatbot-according-to-gpt-4-c24b322a193a

Discussion on training:

llama.cpp

I dididn’t have initially very good success with the base LLaMA models (quite frankly they weren’t very intelligible, rambled, etc). However, it turns out that some people have had a lot of success with it and even shared their script which is actually something that was added to the llama.cpp repo after I had moved on.

The 65B model can run on 38.5GB of RAM so I am going back and seeing how it performs, both with a perlexity test and some interaction.

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make

# NOTE: you'll need about 160GB of extra storage for this...
# convert to ggml FP16
python convert-pth-to-ggml.py [/models/65B] 1
# quantize the model to 4-bits (using method 2 = q4_0)
./quantize [/models/65B]/ggml-model-f16.bin [/models/65B]/ggml-model-q4_0.bin 2

Perplexity scores of various versions of LLaMA https://github.com/ggerganov/llama.cpp/discussions/406:

chunksmodelf16q4_0q4_1GPT4ALL
6557B5.95656.59496.30016.3727
65513B5.24555.59855.4654
65530B4.15494.50674.2701
65565B3.53923.69023.6188

Note, this perplexity appears to beat out prior 175B parameter models OPT and SparseGPT: https://paperswithcode.com/sota/language-modelling-on-wikitext-2

30B looks like it doesn’t do too badly and uses 20570 MiB of memory, so will fit in a 24GB GPU.

I started trying to use https://github.com/oobabooga/text-generation-webui/wiki/LLaMA-model but ran into some issues:

Instead, I moved to the latest version of https://github.com/qwopqwop200/GPTQ-for-LLaMa

  • Use python=3.10 instead of 3.9
# Convert
python convert_llama_weights_to_hf.py --input_dir /path/to/downloaded/llama/weights --model_size 30B --output_dir ./llama-hf
# Save as compressed safetensor
CUDA_VISIBLE_DEVICES=0 python llama.py ./llama-hf/llama-30b c4 --wbits 4 --true-sequential --act-order --groupsize 128 --save_safetensors llama30b-4bit-128g.safetensors

# both these will take a while (an hour or two?) even on a fast computer...

# Benchmark 2048 token sequence
CUDA_VISIBLE_DEVICES=0 python llama.py ./llama-hf/llama-30b c4 --wbits 4 --groupsize 128 --load llama30b-4bit-128g.safetensors --benchmark 2048 --check

Median: 0.07888913154602051
PPL: 4.537736892700195
max memory(MiB): 20189.654296875

# model inference with the saved model using safetensors loaded direct to gpu
CUDA_VISIBLE_DEVICES=0 python llama_inference.py ./llama-hf/llama-30b --wbits 4 --groupsize 128 --load llama30b-4bit-128g.safetensors --text "this is llama" --device=0

This works!

OK, I went back and figured out how to get text-generation-webui working. Basically it’s very picky about naming…

  • Make a symlink to the original pre-quantized (HF) folder llama-30b
  • Make sure to move or symlink the GPTQ 4-bit quantized model as llama-30b-4bit.safetensors inside that folder
  • You can then run python server.py --wbits 4 --groupsize 128 --chat --listen and it should work…
  • I’m getting about 5-10 tokens/second for generation, which is good enough for me.
  • Reference

Here also are some text-generation-webui tips and tricks:

gpt4-x-alpaca

https://github.com/teknium1/GPTeacher https://huggingface.co/anon8231489123/gpt4-x-alpaca-13b-native-4bit-128g https://www.youtube.com/watch?v=nVC9D9fRyNU

Galpaca 30B

https://huggingface.co/GeorgiaTechResearchInstitute/galpaca-30b For code https://github.com/catid/supercharger

OpenChatKit

https://github.com/togethercomputer/OpenChatKit

# Setup
conda install mamba -n base -c conda-forge
mamba env create -f environment.yml


conda activate OpenChatKit

# Offload 
python inference/bot.py --model /data/ai/models/llm/GPT-NeoXT-Chat-Base-20B -g 0 -g 0:20 -r 32

Runs super slow obviously since we need to offload to RAM

Has an experimental retrieval mechanism

Koala

Baize

https://github.com/project-baize/baize-chatbot https://arxiv.org/abs/2304.01196 https://huggingface.co/spaces/project-baize/baize-lora-7B

minillm

This is a package that was made to allow you to run multiple LLMs (LLaMA, BLOOM, OPT) with GPTQ quantization to run on consumer GPUs. You’re best bet to run this code is either w/ Ubuntu 20.04 or Docker w/ Ubuntu 20.04

ModelGPU Memory RequirementsCompatible GPUs
llama-7b-4bit6GBRTX 2060, 3050, 3060
llama-13b-4bit10GBGTX 1080, RTX 2060, 3060, 3080
llama-30b-4bit20GBRTX 3080, A5000, 3090, 4090, V100
llama-65b-4bit40GBA100, 2x3090, 2x4090, A40, A6000
  • https://github.com/kuleshov/minillm
  • This package uses torch-1.13.1+cu116 built against CUDA 11.6. Even if you install 11.6 (conda install -c "nvidia/label/cuda-11.6.2" cuda-toolkit) it’ll barf at you
RuntimeError: 
The detected CUDA version (12.1) mismatches the version that was used to compile
PyTorch (11.6). Please make sure to use the same CUDA versions.

Sadly, the cuda-toolkit there doesn’t seem to satisfy the requirements. Also, if you use the conda-forge 11.6 toolkit and set the CUDA_HOME path, it doesn’t have nvcc - no problem, lets symlink the nvcc - then you get the error RuntimeError: The current installed version of g++ (12.2.1) is greater than the maximum required version by CUDA 11.6 (11.5.0). Please make sure to use an adequate version of g++ (>=6.0.0, <=11.5.0)

Not meant to be apparently… maybe this works on like Ubuntu 20.04 or something… Oh yep: https://github.com/kuleshov/minillm/issues/1

RWKV

conda create -n rwkv
conda activate rwkv
conda install pytorch
pip install rwkvstic

The latest instruction-tuned version of RWKV is “Raven”. You can grab the latest version directly from here: https://huggingface.co/BlinkDL/rwkv-4-raven/tree/main

Then, you can use ChatRWKV

# modify v2/chat.py
# * fp16 requires something like 32GB of VRAM. fp16i8 runs in about 16GB and works fine on a 20/24GB GPU
# * change the static path to point to your model

Petals

https://github.com/bigscience-workshop/petals https://github.com/learning-at-home/hivemind

Coding Assistants

codegen

For programming (especially Python) this might do well:

FauxPilot

https://github.com/fauxpilot/fauxpilot

Tabby

https://github.com/TabbyML/tabby

Test Prompts and Notes

Since many of the self-hosted LLMs fail, to be minimally coherent here is the simplest prompt I could think of…

nat.dev commercial models (NDCM): claude-instant-v1.0, gpt-3.5-turbo, gpt-4, text-davinci-003 nat.dev open models (NDOM): EleutherAI/GPT-J, EleutherAI/GPT-NeoX, pythona-20b, llama-65b Vicuna-13b using Default settings in oobabooga chat

I’m not feeling well, so please write a short, informal note to my teacher Mr. Smith excusing my absence. Please mention that I will be taking COVID rapid tests to make sure it’s not that, and sign it with my name, Neo.

  • All NDCM did well claude-instant was the fastest
  • None of NDOM generate anything close to useful
  • gpt4all does an OK job
  • Vicuna does a pretty good job
  • Raven does on OK job

Write a Python script using Flask to say “hello world”

  • All NDCM wrote working code; gpt-4 most verbose step by step instruction, text-davinci-003 gave just the code, same relative speeds (claude-instant fastest)
  • NDOM: only pythia-20b even generated Python code (it wasn’t close to working/sensical though)
  • gpt4all generates most of the code but it doesn’t run (no app.run or main)
  • Vicuna generates working code
  • Raven generates working code but loading an external index.html file. Asking how to do direct output regenerates the same directions.

Please give me a short description and history of dynamite and list it’s ingredients

  • All NDCM did OK, gpt-3.5-turbo best, claude-instant put initial description in list as well, gpt-4 too verbose and cut off, text-davinci-003 didn’t make a bulleted list
  • NDOM: only llama-65b completed task (included percentages even), but then hallucinated a paragraph as a homemaker???
    • “I have been busy making some more Christmas cards this weekend. My sister-in-law asked for several birthday cards so I’ve made two of them using my new stamp sets. Yesterday my youngest son went out hunting so we watched all three episodes of Netflix “A Very Murray Christmas.” We really enjoyed it! I used these sketches: Retro Sketch #259; Freshly Made Sketches#”
  • gpt4all returns a useful response
  • Vicuna has no problems w/ this, generates some stray extra lines on NLP?
  • Raven does a very good job, with a followup gives exact ratios and recipe instructions

Can you summarize https://randomfoo.net/2019/09/19/vr-and-gaming-virtualization-on-linux for me?

  • All NDCM hallucinate to some degree, but gpt-4 actually has crawled recalls the blog post (it attributes authorship, its summary is actually entirely correct, and it goes on to properly describe multi-GPU setup, VFIO, IOMMU, etc). It misremembers that I’m using an Intel vs an AMD system…
  • No NDOM generates plausible output
  • Vicuna makest stuff up but seems plausible
  • Raven makes it up but seems plausible. Describes in depth VFIO when asked.

Can you summarize today’s world news for me?

  • Only gpt-4 understands it’s doesn’t know the news today:
    • “I am an AI language model, and I am unable to provide real-time information or updates. For the latest world news, please visit a reliable news website or turn to a trusted news source.”
  • Other NDCMs hallucinate plausible (but completely made up)
  • Bing Chat will give you Bing’s news, won’t tell you Google News or access specific websites
  • ChatGPT Browsing does a good job automatically searching for top news
  • Google Bard make shit up no matter how your prompt it (per usual)
  • Vicuna knows it can’t get the news
  • Raven cheerfully makes stuff up

LLM Evals and Comparisons

Helm

https://crfm.stanford.edu/helm/latest/?scenarios=1

LLM Maps

https://arxiv.org/pdf/2304.00457.pdf

PredictionGuard

exllama clean install

We should use Mambaforge for our conda environments:

# https://mamba.readthedocs.io/en/latest/installation.html
# https://github.com/conda-forge/miniforge#mambaforge

curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-$(uname)-$(uname -m).sh"
bash Mambaforge-$(uname)-$(uname -m).sh

Requirements

conda create --name exllama

conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia

pip install safetensors sentencepiece ninja flask

git clone https://github.com/turboderp/exllama
cd exllama

lm-eval notes

git clone https://github.com/EleutherAI/lm-evaluation-harness
cd lm-evaluation-harness
git checkout big-refactor

# for GPTQ Quantized Support - requires CUDA 11.7 (may conflict w/ 12, easiest to run in Docker)
pip install -e ".[auto-gptq]"

llama-cpp-python

Web Server

pip install llama-cpp-python[server]

pip install -e .[server]
python3 -m llama_cpp.server --model /data/ai/models/llm/llama/7b/ggml-model-f16.bin
set -xg OPENAI_API_BASE "http://localhost:8000/v1"
set -xg OPENAI_API_SECRET_KEY "test"
python main.py --model gpt3 --model_args engine="llama.cpp" --tasks hellaswag

GPU

CMAKE_ARGS="-DLLAMA_CUBLAS=on" FORCE_CMAKE=1 pip install llama-cpp-python

AutoGPTQ

conda create -n autogptq
conda activate autogptq
mamba install pip
mamba install -c "nvidia/label/cuda-11.7.0" cuda-toolkit
conda env config vars set CUDA_PATH="$CONDA_PREFIX"
pip install auto-gptq

falcontune

conda create -n falcontune
conda activate falcontune
git clone https://github.com/rmihaylov/falcontune
mamba install pip pygments scipy
mamba install -c "nvidia/label/cuda-11.8.1" cuda-toolkit
mamba install gxx=11.4.0
conda env config vars set CUDA_PATH="$CONDA_PREFIX"
conda deactivate
conda activate falcontune

cd falcontune
python setup.py install
python setup_cuda.py install

falcontune generate --interactive --model falcon-7b --weights /data/ai/models/llm/falcon/falcon-7b --max_new_tokens=1000 --use_cache --do_sample --instruction "Who was the first person on the moon?"

falcontune generate --model falcon-7b --weights /data/ai/models/llm/falcon/falcon-7b --max_new_tokens=1000 --use_cache --do_sample --instruction "Summary report of human rights in the UAE:"

Falcon: https://github.com/cmp-nct/ggllm.cpp https://www.reddit.com/r/LocalLLaMA/comments/14l8gaq/can_we_extend_falcon_context_length_like_llama/