2023-05-21
llama.cpp breaks q4_0 format again: https://www.reddit.com/r/LocalLLaMA/comments/13md90j/another_new_llamacpp_ggml_breaking_change/
2023-05-14
Merged to master https://www.reddit.com/r/LocalLLaMA/comments/13gok03/llamacpp_now_officially_supports_gpu_acceleration/ https://github.com/ggerganov/llama.cpp/pull/1412 https://github.com/ggerganov/llama.cpp/commit/905d87b70aa189623d500a28602d7a3a755a4769
2023-05-09 llama.cpp GPU
I have a Ryzen 5950X w/ 64GB DDR-3800 and an RTX 4090 (24GB).
For those interested in testing the code (I didn’t see any docs):
git clone https://github.com/JohannesGaessler/llama.cpp llama.cpp-johannegaessler
cd llama.cpp-johannegaessler
git fetch
git branch -v -a
git switch dequantize-matmul-2
make LLAMA_CUBLAS=1
Here’s the CPU results:
$ ./main -m /data/ai/models/llm/llama/30b/llama-30b-ggml-q4_0.bin -t 16 -n 128 (exllama)
main: build = 527 (3ed4588)
main: seed = 1683680334
llama.cpp: loading model from /data/ai/models/llm/llama/30b/llama-30b-ggml-q4_0.bin
llama_model_load_internal: format = ggjt v1 (latest)
llama_model_load_internal: n_vocab = 32000
llama_model_load_internal: n_ctx = 512
llama_model_load_internal: n_embd = 6656
llama_model_load_internal: n_mult = 256
llama_model_load_internal: n_head = 52
llama_model_load_internal: n_layer = 60
llama_model_load_internal: n_rot = 128
llama_model_load_internal: ftype = 2 (mostly Q4_0)
llama_model_load_internal: n_ff = 17920
llama_model_load_internal: n_parts = 1
llama_model_load_internal: model size = 30B
llama_model_load_internal: ggml ctx size = 135.75 KB
llama_model_load_internal: mem required = 21695.49 MB (+ 3124.00 MB per state)
llama_init_from_file: kv self size = 780.00 MB
system_info: n_threads = 16 / 32 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | VSX = 0 |
sampling: repeat_last_n = 64, repeat_penalty = 1.100000, presence_penalty = 0.000000, frequency_penalty = 0.000000, top_k = 40, tfs_z = 1.000000, top_p = 0.950000, typical_p = 1.000000, temp = 0.800000, mirostat = 0, mirostat_lr = 0.100000, mirostat_ent = 5.000000
generate: n_ctx = 512, n_batch = 512, n_predict = 128, n_keep = 0
2001-2005 - Master of Arts (M.A.) in International Relations, University of St Andrews
2006-2009 - PhD Student, University of St Andrews
2007-2008 - Research Assistant, University of St Andrews
2008-2010 - Lecturer in International Relations and Security Studies, University of St Andrews
June 2010 to present – Senior Lecturer, University of St Andrews
Teaching and PhD supervision
“A
llama_print_timings: load time = 2092.53 ms
llama_print_timings: sample time = 50.16 ms / 128 runs ( 0.39 ms per run)
llama_print_timings: prompt eval time = 461.30 ms / 2 tokens ( 230.65 ms per token)
llama_print_timings: eval time = 57703.89 ms / 127 runs ( 454.36 ms per run)
llama_print_timings: total time = 59866.55 ms
And with 50/60 layers (eg if you have a WM running or need context), about 3X faster:
$ ./main -m /data/ai/models/llm/llama/30b/llama-30b-ggml-q4_0.bin -t 16 -n 128 --gpu_layers 50 (exllama)
main: build = 527 (3ed4588)
main: seed = 1683678995
llama.cpp: loading model from /data/ai/models/llm/llama/30b/llama-30b-ggml-q4_0.bin
llama_model_load_internal: format = ggjt v1 (latest)
llama_model_load_internal: n_vocab = 32000
llama_model_load_internal: n_ctx = 512
llama_model_load_internal: n_embd = 6656
llama_model_load_internal: n_mult = 256
llama_model_load_internal: n_head = 52
llama_model_load_internal: n_layer = 60
llama_model_load_internal: n_rot = 128
llama_model_load_internal: ftype = 2 (mostly Q4_0)
llama_model_load_internal: n_ff = 17920
llama_model_load_internal: n_parts = 1
llama_model_load_internal: model size = 30B
llama_model_load_internal: ggml ctx size = 135.75 KB
llama_model_load_internal: mem required = 21695.49 MB (+ 3124.00 MB per state)
llama_init_from_file: kv self size = 780.00 MB
system_info: n_threads = 16 / 32 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | VSX = 0 |
sampling: repeat_last_n = 64, repeat_penalty = 1.100000, presence_penalty = 0.000000, frequency_penalty = 0.000000, top_k = 40, tfs_z = 1.000000, top_p = 0.950000, typical_p = 1.000000, temp = 0.800000, mirostat = 0, mirostat_lr = 0.100000, mirostat_ent = 5.000000
generate: n_ctx = 512, n_batch = 512, n_predict = 128, n_keep = 0
50 years ago today, one of the most iconic speeches in American history took place. Martin Luther King, Jr gave his "I Have a Dream" speech on this day at the Lincoln Memorial in Washington D.C. He delivered his powerful message from the steps of the Lincoln Memorial to an audience of some 250,000 civil rights supporters who had converged on our nation's capital for the March on Washington for Jobs and Freedom on August 28, 1963. The speech is widely regarded as one of the greatest oratorical performances in American history.
llama_print_timings: load time = 3326.08 ms
llama_print_timings: sample time = 54.11 ms / 128 runs ( 0.42 ms per run)
llama_print_timings: prompt eval time = 624.29 ms / 2 tokens ( 312.14 ms per token)
llama_print_timings: eval time = 15736.76 ms / 127 runs ( 123.91 ms per run)
llama_print_timings: total time = 19137.01 ms
And w/ all 60 layers on GPU, another 2X:
$ ./main -m /data/ai/models/llm/llama/30b/llama-30b-ggml-q4_0.bin -t 16 -n 128 --gpu_layers 60 (exllama)
main: build = 527 (3ed4588)
main: seed = 1683679354
llama.cpp: loading model from /data/ai/models/llm/llama/30b/llama-30b-ggml-q4_0.bin
llama_model_load_internal: format = ggjt v1 (latest)
llama_model_load_internal: n_vocab = 32000
llama_model_load_internal: n_ctx = 512
llama_model_load_internal: n_embd = 6656
llama_model_load_internal: n_mult = 256
llama_model_load_internal: n_head = 52
llama_model_load_internal: n_layer = 60
llama_model_load_internal: n_rot = 128
llama_model_load_internal: ftype = 2 (mostly Q4_0)
llama_model_load_internal: n_ff = 17920
llama_model_load_internal: n_parts = 1
llama_model_load_internal: model size = 30B
llama_model_load_internal: ggml ctx size = 135.75 KB
llama_model_load_internal: mem required = 21695.49 MB (+ 3124.00 MB per state)
llama_init_from_file: kv self size = 780.00 MB
system_info: n_threads = 16 / 32 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | VSX = 0 |
sampling: repeat_last_n = 64, repeat_penalty = 1.100000, presence_penalty = 0.000000, frequency_penalty = 0.000000, top_k = 40, tfs_z = 1.000000, top_p = 0.950000, typical_p = 1.000000, temp = 0.800000, mirostat = 0, mirostat_lr = 0.100000, mirostat_ent = 5.000000
generate: n_ctx = 512, n_batch = 512, n_predict = 128, n_keep = 0
2018 SACRAMENTO KINGS ALL-STAR COURT DESIGN CONTEST (THE “CONTEST”)
This Contest is sponsored by Sacramento Basketball Holdings, LLC d/b/a the Sacramento Kings (“Sponsor”). The Contest begins on or about November 19, 2018 and ends on or about December 23, 2018 (the “Entry Period”) for electronic entries. Entry forms postmarked prior to November 19, 2018 will not be accepted
llama_print_timings: load time = 3578.37 ms
llama_print_timings: sample time = 51.73 ms / 128 runs ( 0.40 ms per run)
llama_print_timings: prompt eval time = 651.60 ms / 2 tokens ( 325.80 ms per token)
llama_print_timings: eval time = 7627.65 ms / 127 runs ( 60.06 ms per run)
You may also want to talk to u/ReturningTarzan and check out his repo https://github.com/turboderp/exllama as he’s been making some memory optimizations in particular there…
2023-05-04 30/65B
Posted and chatted w: https://www.reddit.com/user/ReturningTarzan https://www.reddit.com/r/LocalLLaMA/comments/13710u8/comment/jitarbe/?utm_source=reddit&utm_medium=web2x&context=3 see: https://github.com/turboderp/exllama
Memory Usage