Renting
See comparison of costs: https://docs.google.com/spreadsheets/d/1ElwIv0ELTvPW2oEf20TpD9ZunWE6UWP-X-ubZh4KsF0/edit?usp=sharing
- https://cloud.vast.ai/create/
- https://www.runpod.io/gpu-instance/pricing
- https://lambdalabs.com/service/gpu-cloud#pricing
- https://www.fluidstack.io/pricing
- https://coreweave.com/gpu-cloud-pricing
- https://www.paperspace.com/pricing
On-Prem
A100 vs H100 vs MI250
Intel SPR AMX: https://www.nextplatform.com/2023/04/05/why-ai-inference-will-remain-largely-on-the-cpu/
Benchmarking
Tim Dettmer’s GPU for Deep Learning overview is probably the best: https://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/
H100 vs A100 https://www.mosaicml.com/blog/coreweave-nvidia-h100-part-1
Whisper Performance comparison https://www.tomshardware.com/news/whisper-audio-transcription-gpus-benchmarked
- Performance + Perf/Watt
- Windows only - would be better w/ Linux and ROCm, but interesting
- AMD + Nvidia both do great
https://mlcommons.org/en/training-normal-20/ https://mlcommons.org/en/inference-datacenter-30/ https://mlcommons.org/en/inference-edge-30/ https://mlcommons.org/en/inference-mobile-30/
https://lambdalabs.com/blog/nvidia-rtx-4090-vs-rtx-3090-deep-learning-benchmark https://lambdalabs.com/blog/nvidia-rtx-a6000-benchmarks https://lambdalabs.com/blog/nvidia-rtx-a6000-vs-rtx-3090-benchmarks
Compatibility chart https://github.com/oobabooga/text-generation-webui/issues/792
AITemplate-PyTorch Stable Diffusion https://news.ycombinator.com/item?id=33069477 https://news.ycombinator.com/item?id=33073369 https://twitter.com/soumithchintala/status/1592194058126233600?lang=en
AMD
Short answer: don’t bother atm
https://www.nextplatform.com/2021/12/06/stacking-up-amd-mi200-versus-nvidia-a100-compute-engines/ https://ai.facebook.com/blog/gpu-inference-engine-nvidia-amd-open-source/ https://github.com/facebookincubator/AITemplate https://facebookincubator.github.io/AITemplate/tutorial/how_to_infer_pt.html https://www.cerebrium.ai/blog/improve-stable-diffusion-inference-by-50-with-tensorrt-or-aitemplate ROCm https://www.youtube.com/watch?v=UtcaO3zTCKQ
RDNA2 ROCm https://www.reddit.com/r/hardware/comments/qkc6n8/6900xt_tensorflow_mlai_benchmarks_using_rocm/ For RDNA2, you apparently can get LLMs running, but it requires forking/patching both bitsandbytes and GPTQ: https://rentry.org/eq3hg - and this will be true for any library (eg, can you use accelerate? deepspeed? fastgen? who knows, but certainly no one is testing it and AMD doesn’t care if you’re not on CDNA). It’s important to note again, anything that works atm will still only work with last-gen cards, on Linux-only (ROCm does not work through WSL), w/ limited VRAM (no 30Bq4 models), and since RDNA2 tensor support is awful, if the SD benchmarks are anything to go by, performance will still end up worse than an RTX 3050: https://www.tomshardware.com/news/stable-diffusion-gpu-benchmarks https://github.com/broncotc/bitsandbytes-rocm/issues/1#issuecomment-1345828347 https://github.com/nod-ai/SHARK/blob/f898a1d33220256af913a12001b8fe8169a1aa9d/shark/examples/shark_inference/llama/README.md
https://www.reddit.com/r/LocalLLaMA/comments/11zcqj2/llama_optimized_for_amd_gpus/
https://www.reddit.com/r/LocalLLaMA/comments/12rv9xz/i_got_vicuna_running_on_a_rx_7900_xtx_using_a_nix/ https://github.com/lm-sys/FastChat/issues/104#issuecomment-1514838351
Server CUDA setup
sudo tee /etc/apt/preferences.d/cuda-repository-pin-600 > /dev/null <<EOL
Package: nsight-compute
Pin: origin *ubuntu.com*
Pin-Priority: -1
Package: nsight-systems
Pin: origin *ubuntu.com*
Pin-Priority: -1
Package: nvidia-modprobe
Pin: release l=NVIDIA CUDA
Pin-Priority: 600
Package: nvidia-settings
Pin: release l=NVIDIA CUDA
Pin-Priority: 600
Package: *
Pin: release l=NVIDIA CUDA
Pin-Priority: 100
EOL
sudo apt update
sudo apt upgrade -y
sudo apt dist-upgrade -y
sudo apt install software-properties-common
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.0-1_all.deb
sudo dpkg -i cuda-keyring_1.0-1_all.deb
sudo apt install cuda
# Add to your profile...
export CUDA_PATH=/usr/local/cuda
export CUDA_HOME=/usr/local/cuda
export PATH=$PATH:/usr/local/cuda/bin
# nvtop
https://github.com/Syllo/nvtop#ubuntu-ppa
sudo add-apt-repository ppa:flexiondotorg/nvtop
sudo apt install nvtop
- https://cloud.google.com/compute/docs/gpus/install-drivers-gpu
- https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#ubuntu
- https://developer.nvidia.com/blog/updating-the-cuda-linux-gpg-repository-key/
- https://github.com/MicrosoftDocs/azure-docs/blob/main/articles/virtual-machines/linux/n-series-driver-setup.md
Performance
https://www.reddit.com/r/MachineLearning/comments/10zsw62/the_inference_cost_of_search_disruption_large/ https://www.semianalysis.com/p/the-inference-cost-of-search-disruption
2023-06-18 exllama Performance
A100 40GB
65B - 12.18 t/s
python test_benchmark_inference.py -p -d /data/llm/models/robin-65B-v2-GPTQ
-- Tokenizer: /data/llm/models/robin-65B-v2-GPTQ/tokenizer.model
-- Model config: /data/llm/models/robin-65B-v2-GPTQ/config.json
-- Model: /data/llm/models/robin-65B-v2-GPTQ/robin-65b-GPTQ-4bit--1g.no-act.order.safetensors
-- Sequence length: 2048
-- Tuning:
-- --matmul_recons_thd: 8
-- --fused_mlp_thd: 2
-- --sdp_thd: 8
-- Options: ['perf']
** Time, Load model: 154.19 seconds
** Time, Load tokenizer: 0.08 seconds
-- Groupsize (inferred): None
-- Act-order (inferred): no
** VRAM, Model: [cuda:0] 31,830.18 MB
-- Warmup pass 1...
** Time, Warmup: 2.57 seconds
-- Warmup pass 2...
** Time, Warmup: 1.24 seconds
-- Inference, first pass.
** Time, Inference: 1.24 seconds
** Speed: 1554.07 tokens/second
-- Generating 128 tokens, 1920 token prompt...
** Speed: 12.18 tokens/second
-- Generating 128 tokens, 4 token prompt...
** Speed: 13.10 tokens/second
** VRAM, Inference: [cuda:0] 5,386.42 MB
** VRAM, Total: [cuda:0] 37,216.60 MB
33B - 20.72 t/s
python test_benchmark_inference.py -p -d /data/llm/models/robin-33B-v2-GPTQ/
-- Tokenizer: /data/llm/models/robin-33B-v2-GPTQ/tokenizer.model
-- Model config: /data/llm/models/robin-33B-v2-GPTQ/config.json
-- Model: /data/llm/models/robin-33B-v2-GPTQ/robin-33b-GPTQ-4bit--1g.act.order.safetensors
-- Sequence length: 2048
-- Tuning:
-- --matmul_recons_thd: 8
-- --fused_mlp_thd: 2
-- --sdp_thd: 8
-- Options: ['perf']
** Time, Load model: 7.48 seconds
** Time, Load tokenizer: 0.01 seconds
-- Groupsize (inferred): None
-- Act-order (inferred): no
** VRAM, Model: [cuda:0] 16,234.66 MB
-- Warmup pass 1...
** Time, Warmup: 1.92 seconds
-- Warmup pass 2...
** Time, Warmup: 0.64 seconds
-- Inference, first pass.
** Time, Inference: 0.64 seconds
** Speed: 3019.50 tokens/second
-- Generating 128 tokens, 1920 token prompt...
** Speed: 20.72 tokens/second
-- Generating 128 tokens, 4 token prompt...
** Speed: 22.41 tokens/second
** VRAM, Inference: [cuda:0] 3,340.30 MB
** VRAM, Total: [cuda:0] 19,574.96 MB
13B - 47.85 t/s
python test_benchmark_inference.py -p -d /data/llm/models/manticore-13b-chat-pyg-GPTQ/
-- Tokenizer: /data/llm/models/manticore-13b-chat-pyg-GPTQ/tokenizer.model
-- Model config: /data/llm/models/manticore-13b-chat-pyg-GPTQ/config.json
-- Model: /data/llm/models/manticore-13b-chat-pyg-GPTQ/Manticore-13B-Chat-Pyg-GPTQ-4bit-128g.no-act-order.safetensors
-- Sequence length: 2048
-- Tuning:
-- --matmul_recons_thd: 8
-- --fused_mlp_thd: 2
-- --sdp_thd: 8
-- Options: ['perf']
** Time, Load model: 4.02 seconds
** Time, Load tokenizer: 0.01 seconds
-- Groupsize (inferred): 128
-- Act-order (inferred): no
** VRAM, Model: [cuda:0] 6,873.52 MB
-- Warmup pass 1...
** Time, Warmup: 1.32 seconds
-- Warmup pass 2...
** Time, Warmup: 0.27 seconds
-- Inference, first pass.
** Time, Inference: 0.27 seconds
** Speed: 7102.23 tokens/second
-- Generating 128 tokens, 1920 token prompt...
** Speed: 47.85 tokens/second
-- Generating 128 tokens, 4 token prompt...
** Speed: 52.49 tokens/second
** VRAM, Inference: [cuda:0] 1,772.67 MB
** VRAM, Total: [cuda:0] 8,646.19 MB
4090 24GB
30B - 40.04 t/s
CUDA_VISIBLE_DEVICES=0 python -W ignore::UserWarning: test_benchmark_inference.py -d /data/ai/models/llm/robin/robin-33B-v2-GPTQ/ -p
-- Tokenizer: /data/ai/models/llm/robin/robin-33B-v2-GPTQ/tokenizer.model
-- Model config: /data/ai/models/llm/robin/robin-33B-v2-GPTQ/config.json
-- Model: /data/ai/models/llm/robin/robin-33B-v2-GPTQ/robin-33b-GPTQ-4bit--1g.act.order.safetensors
-- Sequence length: 2048
-- Tuning:
-- --matmul_recons_thd: 8
-- --fused_mlp_thd: 2
-- --sdp_thd: 8
-- Options: ['perf']
** Time, Load model: 3.10 seconds
** Time, Load tokenizer: 0.00 seconds
-- Groupsize (inferred): None
-- Act-order (inferred): no
** VRAM, Model: [cuda:0] 16,234.66 MB
-- Warmup pass 1...
** Time, Warmup: 0.87 seconds
-- Warmup pass 2...
** Time, Warmup: 0.63 seconds
-- Inference, first pass.
** Time, Inference: 0.63 seconds
** Speed: 3047.21 tokens/second
-- Generating 128 tokens, 1920 token prompt...
** Speed: 40.04 tokens/second
-- Generating 128 tokens, 4 token prompt...
** Speed: 47.29 tokens/second
** VRAM, Inference: [cuda:0] 3,340.30 MB
** VRAM, Total: [cuda:0] 19,574.96 MB
13B - 83.79 t/s
CUDA_VISIBLE_DEVICES=0 python -W ignore::UserWarning: test_benchmark_inference.py -d /data/ai/models/llm/manticore/manticore-13b-chat-pyg-GPTQ -p
-- Tokenizer: /data/ai/models/llm/manticore/manticore-13b-chat-pyg-GPTQ/tokenizer.model
-- Model config: /data/ai/models/llm/manticore/manticore-13b-chat-pyg-GPTQ/config.json
-- Model: /data/ai/models/llm/manticore/manticore-13b-chat-pyg-GPTQ/Manticore-13B-Chat-Pyg-GPTQ-4bit-128g.no-act-order.safetensors
-- Sequence length: 2048
-- Tuning:
-- --matmul_recons_thd: 8
-- --fused_mlp_thd: 2
-- --sdp_thd: 8
-- Options: ['perf']
** Time, Load model: 1.97 seconds
** Time, Load tokenizer: 0.01 seconds
-- Groupsize (inferred): 128
-- Act-order (inferred): no
** VRAM, Model: [cuda:0] 6,873.52 MB
-- Warmup pass 1...
** Time, Warmup: 0.55 seconds
-- Warmup pass 2...
** Time, Warmup: 0.26 seconds
-- Inference, first pass.
** Time, Inference: 0.26 seconds
** Speed: 7354.54 tokens/second
-- Generating 128 tokens, 1920 token prompt...
** Speed: 83.79 tokens/second
-- Generating 128 tokens, 4 token prompt...
** Speed: 95.46 tokens/second
** VRAM, Inference: [cuda:0] 1,772.67 MB
** VRAM, Total: [cuda:0] 8,646.19 MB
Radeon VII 16GB
13B - 15.08 t/s
python test_benchmark_inference.py -d ~/llm/models/Manticore-13B-GPTQ/ -p (llm)
Successfully preprocessed all matching files.
-- Tokenizer: /home/lhl/llm/models/Manticore-13B-GPTQ/tokenizer.model
-- Model config: /home/lhl/llm/models/Manticore-13B-GPTQ/config.json
-- Model: /home/lhl/llm/models/Manticore-13B-GPTQ/Manticore-13B-GPTQ-4bit-128g.no-act-order.safetensors
-- Sequence length: 2048
-- Tuning:
-- --matmul_recons_thd: 8
-- --fused_mlp_thd: 2
-- --sdp_thd: 8
-- --rmsnorm_no_half2
-- --rope_no_half2
-- --matmul_no_half2
-- --silu_no_half2
-- Options: ['perf']
** Time, Load model: 7.17 seconds
** Time, Load tokenizer: 0.01 seconds
-- Groupsize (inferred): 128
-- Act-order (inferred): no
** VRAM, Model: [cuda:0] 6,873.52 MB - [cuda:1] 0.00 MB
-- Warmup pass 1...
** Time, Warmup: 4.75 seconds
-- Warmup pass 2...
** Time, Warmup: 4.54 seconds
-- Inference, first pass.
** Time, Inference: 4.54 seconds
** Speed: 422.94 tokens/second
-- Generating 128 tokens, 1920 token prompt...
** Speed: 15.08 tokens/second
-- Generating 128 tokens, 4 token prompt...
** Speed: 19.18 tokens/second
** VRAM, Inference: [cuda:0] 2,253.08 MB - [cuda:1] 0.00 MB
** VRAM, Total: [cuda:0] 9,126.60 MB - [cuda:1] 0.00 MB
2023-06-18 llama.cpp
Using master-57cd694
A100 ~20% slower on 65B ~ 10% slower on 33B ~20% slower on 13B
4090: ~25% slower on 30B ~10% slower on 13B
A100 40GB
65B q4_0 - 9.8 t/s
Default uses 40518MB
./main -m /data/llm/models/robin-65b.ggmlv3.q4_0.bin -ngl 99 -t 1 -n 2048 --ignore-eos
main: build = 703 (57cd694)
main: seed = 1687074170
ggml_init_cublas: found 1 CUDA devices:
Device 0: NVIDIA A100-SXM4-40GB
llama.cpp: loading model from /data/llm/models/robin-65b.ggmlv3.q4_0.bin
llama_model_load_internal: format = ggjt v3 (latest)
llama_model_load_internal: n_vocab = 32000
llama_model_load_internal: n_ctx = 512
llama_model_load_internal: n_embd = 8192
llama_model_load_internal: n_mult = 256
llama_model_load_internal: n_head = 64
llama_model_load_internal: n_layer = 80
llama_model_load_internal: n_rot = 128
llama_model_load_internal: ftype = 2 (mostly Q4_0)
llama_model_load_internal: n_ff = 22016
llama_model_load_internal: n_parts = 1
llama_model_load_internal: model size = 65B
llama_model_load_internal: ggml ctx size = 0.18 MB
llama_model_load_internal: using CUDA for GPU acceleration
llama_model_load_internal: mem required = 3724.80 MB (+ 5120.00 MB per state)
llama_model_load_internal: allocating batch_size x 1 MB = 512 MB VRAM for the scratch buffer
llama_model_load_internal: offloading 80 repeating layers to GPU
llama_model_load_internal: offloading non-repeating layers to GPU
llama_model_load_internal: offloading v cache to GPU
llama_model_load_internal: offloading k cache to GPU
llama_model_load_internal: offloaded 83/83 layers to GPU
llama_model_load_internal: total VRAM used: 40518 MB
....................................................................................................
llama_init_from_file: kv self size = 1280.00 MB
system_info: n_threads = 1 / 12 | AVX = 1 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 0 | AVX512_VNNI = 1 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0
| WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | VSX = 0 |
sampling: repeat_last_n = 64, repeat_penalty = 1.100000, presence_penalty = 0.000000, frequency_penalty = 0.000000, top_k = 40, tfs_z = 1.000000, top_p = 0.95000
0, typical_p = 1.000000, temp = 0.800000, mirostat = 0, mirostat_lr = 0.100000, mirostat_ent = 5.000000
generate: n_ctx = 512, n_batch = 512, n_predict = 2048, n_keep = 0
...
llama_print_timings: load time = 385879.37 ms
llama_print_timings: sample time = 1291.27 ms / 2048 runs ( 0.63 ms per token)
llama_print_timings: prompt eval time = 9517.02 ms / 1801 tokens ( 5.28 ms per token)
llama_print_timings: eval time = 208293.79 ms / 2040 runs ( 102.10 ms per token)
llama_print_timings: total time = 220011.47 ms
33B q4_0 - 18.15 t/s
./main -m /data/llm/models/robin-33b.ggmlv3.q4_0.bin -ngl 99 -t 1 -n 2048 --ignore-eos
main: build = 703 (57cd694)
main: seed = 1687075616
ggml_init_cublas: found 1 CUDA devices:
Device 0: NVIDIA A100-SXM4-40GB
llama.cpp: loading model from /data/llm/models/robin-33b.ggmlv3.q4_0.bin
llama_model_load_internal: format = ggjt v3 (latest)
llama_model_load_internal: n_vocab = 32000
llama_model_load_internal: n_ctx = 512
llama_model_load_internal: n_embd = 6656
llama_model_load_internal: n_mult = 256
llama_model_load_internal: n_head = 52
llama_model_load_internal: n_layer = 60
llama_model_load_internal: n_rot = 128
llama_model_load_internal: ftype = 2 (mostly Q4_0)
llama_model_load_internal: n_ff = 17920
llama_model_load_internal: n_parts = 1
llama_model_load_internal: model size = 30B
llama_model_load_internal: ggml ctx size = 0.13 MB
llama_model_load_internal: using CUDA for GPU acceleration
llama_model_load_internal: mem required = 2418.39 MB (+ 3124.00 MB per state)
llama_model_load_internal: allocating batch_size x 1 MB = 512 MB VRAM for the scratch buffer
llama_model_load_internal: offloading 60 repeating layers to GPU
llama_model_load_internal: offloading non-repeating layers to GPU
llama_model_load_internal: offloading v cache to GPU
llama_model_load_internal: offloading k cache to GPU
llama_model_load_internal: offloaded 63/63 layers to GPU
llama_model_load_internal: total VRAM used: 20975 MB
....................................................................................................
llama_init_from_file: kv self size = 780.00 MB
system_info: n_threads = 1 / 12 | AVX = 1 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 0 | AVX512_VNNI = 1 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | VSX = 0 |
sampling: repeat_last_n = 64, repeat_penalty = 1.100000, presence_penalty = 0.000000, frequency_penalty = 0.000000, top_k = 40, tfs_z = 1.000000, top_p = 0.950000, typical_p = 1.000000, temp = 0.800000, mirostat = 0, mirostat_lr = 0.100000, mirostat_ent = 5.000000
generate: n_ctx = 512, n_batch = 512, n_predict = 2048, n_keep = 0
...
llama_print_timings: load time = 6209.23 ms
llama_print_timings: sample time = 1291.20 ms / 2048 runs ( 0.63 ms per token)
llama_print_timings: prompt eval time = 5690.88 ms / 1801 tokens ( 3.16 ms per token)
llama_print_timings: eval time = 112428.57 ms / 2040 runs ( 55.11 ms per token)
llama_print_timings: total time = 120250.73 ms
13B q4_0 - 40.3 t/s
./main -m /data/llm/models/robin-13b.ggmlv3.q4_0.bin -ngl 99 -t 1 -n 2048 --ignore-eos [106/1164]
main: build = 703 (57cd694)
main: seed = 1687075439
ggml_init_cublas: found 1 CUDA devices:
Device 0: NVIDIA A100-SXM4-40GB
llama.cpp: loading model from /data/llm/models/robin-13b.ggmlv3.q4_0.bin
llama_model_load_internal: format = ggjt v3 (latest)
llama_model_load_internal: n_vocab = 32000
llama_model_load_internal: n_ctx = 512
llama_model_load_internal: n_embd = 5120
llama_model_load_internal: n_mult = 256
llama_model_load_internal: n_head = 40
llama_model_load_internal: n_layer = 40
llama_model_load_internal: n_rot = 128
llama_model_load_internal: ftype = 2 (mostly Q4_0)
llama_model_load_internal: n_ff = 13824
llama_model_load_internal: n_parts = 1
llama_model_load_internal: model size = 13B
llama_model_load_internal: ggml ctx size = 0.09 MB
llama_model_load_internal: using CUDA for GPU acceleration
llama_model_load_internal: mem required = 2135.98 MB (+ 1608.00 MB per state)
llama_model_load_internal: allocating batch_size x 1 MB = 512 MB VRAM for the scratch buffer
llama_model_load_internal: offloading 40 repeating layers to GPU
llama_model_load_internal: offloading non-repeating layers to GPU
llama_model_load_internal: offloading v cache to GPU
llama_model_load_internal: offloading k cache to GPU
llama_model_load_internal: offloaded 43/43 layers to GPU
llama_model_load_internal: total VRAM used: 9016 MB
....................................................................................................
llama_init_from_file: kv self size = 400.00 MB
system_info: n_threads = 1 / 12 | AVX = 1 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 0 | AVX512_VNNI = 1 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0
| WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | VSX = 0 |
sampling: repeat_last_n = 64, repeat_penalty = 1.100000, presence_penalty = 0.000000, frequency_penalty = 0.000000, top_k = 40, tfs_z = 1.000000, top_p = 0.95000
0, typical_p = 1.000000, temp = 0.800000, mirostat = 0, mirostat_lr = 0.100000, mirostat_ent = 5.000000
generate: n_ctx = 512, n_batch = 512, n_predict = 2048, n_keep = 0
...
llama_print_timings: load time = 3060.88 ms
llama_print_timings: sample time = 1288.71 ms / 2048 runs ( 0.63 ms per token)
llama_print_timings: prompt eval time = 2913.55 ms / 1801 tokens ( 1.62 ms per token)
llama_print_timings: eval time = 50640.48 ms / 2040 runs ( 24.82 ms per token)
llama_print_timings: total time = 55582.30 ms
4090 24GB
30B - 31.84 t/s
./main -m /data/ai/models/llm/robin/robin-33 -ngl 99 -n 2048 --ignore-eos --threads 1
robin-33b.ggmlv3.q4_0.bin robin-33B-v2-GPTQ/
[lhl@zen2 llama.cpp]$ CUDA_VISIBLE_DEVICES=0 ./main -m /data/ai/models/llm/robin/robin-33b.ggmlv3.q4_0.bin -ngl 99 -n 2048 --ignore-eos --threads 1
main: build = 707 (e1886cf)
main: seed = 1687079938
ggml_init_cublas: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 4090
llama.cpp: loading model from /data/ai/models/llm/robin/robin-33b.ggmlv3.q4_0.bin
llama_model_load_internal: format = ggjt v3 (latest)
llama_model_load_internal: n_vocab = 32000
llama_model_load_internal: n_ctx = 512
llama_model_load_internal: n_embd = 6656
llama_model_load_internal: n_mult = 256
llama_model_load_internal: n_head = 52
llama_model_load_internal: n_layer = 60
llama_model_load_internal: n_rot = 128
llama_model_load_internal: ftype = 2 (mostly Q4_0)
llama_model_load_internal: n_ff = 17920
llama_model_load_internal: n_parts = 1
llama_model_load_internal: model size = 30B
llama_model_load_internal: ggml ctx size = 0.13 MB
llama_model_load_internal: using CUDA for GPU acceleration
llama_model_load_internal: mem required = 2418.39 MB (+ 3124.00 MB per state)
llama_model_load_internal: allocating batch_size x 1 MB = 512 MB VRAM for the scratch buffer
llama_model_load_internal: offloading 60 repeating layers to GPU
llama_model_load_internal: offloading non-repeating layers to GPU
llama_model_load_internal: offloading v cache to GPU
llama_model_load_internal: offloading k cache to GPU
llama_model_load_internal: offloaded 63/63 layers to GPU
llama_model_load_internal: total VRAM used: 20975 MB
....................................................................................................
llama_init_from_file: kv self size = 780.00 MB
system_info: n_threads = 1 / 32 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | VSX = 0 |
sampling: repeat_last_n = 64, repeat_penalty = 1.100000, presence_penalty = 0.000000, frequency_penalty = 0.000000, top_k = 40, tfs_z = 1.000000, top_p = 0.950000, typical_p = 1.000000, temp = 0.800000, mirostat = 0, mirostat_lr = 0.100000, mirostat_ent = 5.000000
generate: n_ctx = 512, n_batch = 512, n_predict = 2048, n_keep = 0
...
llama_print_timings: load time = 10435.28 ms
llama_print_timings: sample time = 835.83 ms / 2048 runs ( 0.41 ms per token)
llama_print_timings: prompt eval time = 5088.28 ms / 1801 tokens ( 2.83 ms per token)
llama_print_timings: eval time = 64073.03 ms / 2040 runs ( 31.41 ms per token)
llama_print_timings: total time = 70471.35 ms
13B - 76.28 t/s
./main -m /data/ai/models/llm/manticore/Manticore-13B-Chat-Pyg.ggmlv3.q4_0.bin -ngl 99 -n 2048 --ignore-eos --threads 1
main: build = 707 (e1886cf)
main: seed = 1687080251
ggml_init_cublas: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 4090
llama.cpp: loading model from /data/ai/models/llm/manticore/Manticore-13B-Chat-Pyg.ggmlv3.q4_0.bin
llama_model_load_internal: format = ggjt v3 (latest)
llama_model_load_internal: n_vocab = 32000
llama_model_load_internal: n_ctx = 512
llama_model_load_internal: n_embd = 5120
llama_model_load_internal: n_mult = 256
llama_model_load_internal: n_head = 40
llama_model_load_internal: n_layer = 40
llama_model_load_internal: n_rot = 128
llama_model_load_internal: ftype = 2 (mostly Q4_0)
llama_model_load_internal: n_ff = 13824
llama_model_load_internal: n_parts = 1
llama_model_load_internal: model size = 13B
llama_model_load_internal: ggml ctx size = 0.09 MB
llama_model_load_internal: using CUDA for GPU acceleration
llama_model_load_internal: mem required = 2135.98 MB (+ 1608.00 MB per state)
llama_model_load_internal: allocating batch_size x 1 MB = 512 MB VRAM for the scratch buffer
llama_model_load_internal: offloading 40 repeating layers to GPU
llama_model_load_internal: offloading non-repeating layers to GPU
llama_model_load_internal: offloading v cache to GPU
llama_model_load_internal: offloading k cache to GPU
llama_model_load_internal: offloaded 43/43 layers to GPU
llama_model_load_internal: total VRAM used: 9016 MB
....................................................................................................
llama_init_from_file: kv self size = 400.00 MB
system_info: n_threads = 1 / 32 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | VSX = 0 |
sampling: repeat_last_n = 64, repeat_penalty = 1.100000, presence_penalty = 0.000000, frequency_penalty = 0.000000, top_k = 40, tfs_z = 1.000000, top_p = 0.950000, typical_p = 1.000000, temp = 0.800000, mirostat = 0, mirostat_lr = 0.100000, mirostat_ent = 5.000000
generate: n_ctx = 512, n_batch = 512, n_predict = 2048, n_keep = 0
...
llama_print_timings: load time = 4195.08 ms
llama_print_timings: sample time = 808.64 ms / 2048 runs ( 0.39 ms per token)
llama_print_timings: prompt eval time = 2404.41 ms / 1801 tokens ( 1.34 ms per token)
llama_print_timings: eval time = 26748.95 ms / 2040 runs ( 13.11 ms per token)
llama_print_timings: total time = 30332.70 ms
M2 MBA
13B - 4.14 t/s
Sideways for extra cooling…
./main -m ~/ai/models/ggml-new/13b-chimera.ggmlv3.q4_0.bin -ngl 1 -n 2048 --ignore-eos (base)
main: build = 704 (ce2c7d7)
main: seed = 1687074647
llama.cpp: loading model from /Users/lhl/ai/models/ggml-new/13b-chimera.ggmlv3.q4_0.bin
llama_model_load_internal: format = ggjt v3 (latest)
llama_model_load_internal: n_vocab = 32000
llama_model_load_internal: n_ctx = 512
llama_model_load_internal: n_embd = 5120
llama_model_load_internal: n_mult = 256
llama_model_load_internal: n_head = 40
llama_model_load_internal: n_layer = 40
llama_model_load_internal: n_rot = 128
llama_model_load_internal: ftype = 2 (mostly Q4_0)
llama_model_load_internal: n_ff = 13824
llama_model_load_internal: n_parts = 1
llama_model_load_internal: model size = 13B
llama_model_load_internal: ggml ctx size = 0.09 MB
llama_model_load_internal: mem required = 9031.70 MB (+ 1608.00 MB per state)
....................................................................................................
llama_init_from_file: kv self size = 400.00 MB
ggml_metal_init: allocating
ggml_metal_init: using MPS
ggml_metal_init: loading '/Users/lhl/ai/llm/llama.cpp/ggml-metal.metal'
ggml_metal_init: loaded kernel_add 0x154f08f80
ggml_metal_init: loaded kernel_mul 0x154f09580
ggml_metal_init: loaded kernel_mul_row 0x154f09aa0
ggml_metal_init: loaded kernel_scale 0x154f09fc0
ggml_metal_init: loaded kernel_silu 0x154f0a4e0
ggml_metal_init: loaded kernel_relu 0x154f0aa00
ggml_metal_init: loaded kernel_gelu 0x154f0af20
ggml_metal_init: loaded kernel_soft_max 0x154f0b5d0
ggml_metal_init: loaded kernel_diag_mask_inf 0x154f0bc30
ggml_metal_init: loaded kernel_get_rows_f16 0x154f0c2b0
ggml_metal_init: loaded kernel_get_rows_q4_0 0x154f0c930
ggml_metal_init: loaded kernel_get_rows_q4_1 0x154f0d120
ggml_metal_init: loaded kernel_get_rows_q2_k 0x154f0d7a0
ggml_metal_init: loaded kernel_get_rows_q3_k 0x154f0de20
ggml_metal_init: loaded kernel_get_rows_q4_k 0x154f0e4a0
ggml_metal_init: loaded kernel_get_rows_q5_k 0x154f0eb20
ggml_metal_init: loaded kernel_get_rows_q6_k 0x154f0f1a0
ggml_metal_init: loaded kernel_rms_norm 0x154f0f850
ggml_metal_init: loaded kernel_norm 0x154f0ff00
ggml_metal_init: loaded kernel_mul_mat_f16_f32 0x154f108d0
ggml_metal_init: loaded kernel_mul_mat_q4_0_f32 0x154f10fb0
ggml_metal_init: loaded kernel_mul_mat_q4_1_f32 0x154f11690
ggml_metal_init: loaded kernel_mul_mat_q2_k_f32 0x154f11d70
ggml_metal_init: loaded kernel_mul_mat_q3_k_f32 0x154f125f0
ggml_metal_init: loaded kernel_mul_mat_q4_k_f32 0x154f12cd0
ggml_metal_init: loaded kernel_mul_mat_q5_k_f32 0x154f133b0
ggml_metal_init: loaded kernel_mul_mat_q6_k_f32 0x154f13a90
ggml_metal_init: loaded kernel_rope 0x154f14580
ggml_metal_init: loaded kernel_alibi_f32 0x154f14e40
ggml_metal_init: loaded kernel_cpy_f32_f16 0x154f156d0
ggml_metal_init: loaded kernel_cpy_f32_f32 0x154f15f60
ggml_metal_init: loaded kernel_cpy_f16_f16 0x154f167f0
ggml_metal_init: recommendedMaxWorkingSetSize = 10922.67 MB
ggml_metal_init: hasUnifiedMemory = true
ggml_metal_init: maxTransferRate = built-in GPU
llama_init_from_file: max tensor size = 87.89 MB
ggml_metal_add_buffer: allocated 'data ' buffer, size = 6984.06 MB, ( 6984.45 / 10922.67)
ggml_metal_add_buffer: allocated 'eval ' buffer, size = 1024.00 MB, ( 8008.45 / 10922.67)
ggml_metal_add_buffer: allocated 'kv ' buffer, size = 402.00 MB, ( 8410.45 / 10922.67)
ggml_metal_add_buffer: allocated 'scr0 ' buffer, size = 512.00 MB, ( 8922.45 / 10922.67)
ggml_metal_add_buffer: allocated 'scr1 ' buffer, size = 512.00 MB, ( 9434.45 / 10922.67)
system_info: n_threads = 4 / 8 | AVX = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | VSX = 0 |
sampling: repeat_last_n = 64, repeat_penalty = 1.100000, presence_penalty = 0.000000, frequency_penalty = 0.000000, top_k = 40, tfs_z = 1.000000, top_p = 0.950000, typical_p = 1.000000, temp = 0.800000, mirostat = 0, mirostat_lr = 0.100000, mirostat_ent = 5.000000
generate: n_ctx = 512, n_batch = 512, n_predict = 2048, n_keep = 0
...
llama_print_timings: load time = 495.29 ms
llama_print_timings: sample time = 1378.12 ms / 2048 runs ( 0.67 ms per token)
llama_print_timings: prompt eval time = 176044.60 ms / 1801 tokens ( 97.75 ms per token)
llama_print_timings: eval time = 492169.10 ms / 2040 runs ( 241.26 ms per token)
llama_print_timings: total time = 669814.74 ms