- Using Ubuntu 24.04 LTS, it dies on suspend for some reason
- Running Arch Linux it seems to be OK.
Nvidia BS
https://www.techpowerup.com/gpu-specs/geforce-rtx-3080-ti-mobile.c3840 https://www.techpowerup.com/vgabios/253063/253063 https://www.techpowerup.com/download/nvidia-nvflash/
- nvflash has a Linux version but can’t seem to read the memory from the GPU
Power Limits
The Power Limit is 80W (and can not be modified by nvidia-smi) but there is supposed to be a dynamic boost to 100W. However, in Linux, it doesn’t seem to work. However, if you run nvidia-powerd it will boost up to 90W
systemctl enable nvidia-powerd.service
systemctl start nvidia-powerd.service
- https://www.reddit.com/r/linux_gaming/comments/1bbdig6/psa_nvidia_may_be_artificially_reducing_your/
Also you should set to performance:
# https://wiki.archlinux.org/title/CPU_frequency_scaling#tuned
paru -S tuned
sudo systemctl enable tuned
sudo systemctl start tuned
sudo tuned-adm profile accelerator-performance
# Power
watch -n 1 "nvidia-smi -q | grep 'Power Draw'"
# Clocks
watch "nvidia-smi -q | grep 'Clocks' -A4"
This gives a significant speed difference (+21% PP, +26.8% TG) w/ nvidia-powerd and performance set:
# 80W
❯ ./llama-bench -m /models/gguf/Meta-Llama-3-8B-Instruct-Q4_K_M.gguf (base)
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 3080 Ti Laptop GPU, compute capability 8.6, VMM: yes
| model | size | params | backend | ngl | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------------: | ---------------: |
| llama 8B Q4_K - Medium | 4.58 GiB | 8.03 B | CUDA | 99 | pp512 | 1514.41 ± 59.17 |
| llama 8B Q4_K - Medium | 4.58 GiB | 8.03 B | CUDA | 99 | tg128 | 42.27 ± 0.28 |
build: 828d6ff7 (3580)
# 90W
❯ ./llama-bench -m /models/gguf/Meta-Llama-3-8B-Instruct-Q4_K_M.gguf (base)
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 3080 Ti Laptop GPU, compute capability 8.6, VMM: yes
| model | size | params | backend | ngl | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------------: | ---------------: |
| llama 8B Q4_K - Medium | 4.58 GiB | 8.03 B | CUDA | 99 | pp512 | 1679.35 ± 159.17 |
| llama 8B Q4_K - Medium | 4.58 GiB | 8.03 B | CUDA | 99 | tg128 | 53.61 ± 0.24 |
build: 828d6ff7 (3580)
# Also set performance
❯ ./llama-bench -m /models/gguf/Meta-Llama-3-8B-Instruct-Q4_K_M.gguf (base)
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 3080 Ti Laptop GPU, compute capability 8.6, VMM: yes
| model | size | params | backend | ngl | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------------: | ---------------: |
| llama 8B Q4_K - Medium | 4.58 GiB | 8.03 B | CUDA | 99 | pp512 | 1831.97 ± 65.50 |
| llama 8B Q4_K - Medium | 4.58 GiB | 8.03 B | CUDA | 99 | tg128 | 53.15 ± 0.52 |
build: 828d6ff7 (3580)
# A bit faster PP w/ FA
❯ ./llama-bench -m /models/gguf/Meta-Llama-3-8B-Instruct-Q4_K_M.gguf -fa 1 (base)
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 3080 Ti Laptop GPU, compute capability 8.6, VMM: yes
| model | size | params | backend | ngl | fa | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -: | ------------: | ---------------: |
| llama 8B Q4_K - Medium | 4.58 GiB | 8.03 B | CUDA | 99 | 1 | pp512 | 1953.21 ± 92.83 |
| llama 8B Q4_K - Medium | 4.58 GiB | 8.03 B | CUDA | 99 | 1 | tg128 | 51.57 ± 0.82 |
build: 828d6ff7 (3580)
Other Settings
# GPU 0: NVIDIA GeForce RTX 3080 Ti Laptop GPU
# Sadly, doesn't have much effect
sudo nvidia-smi -pm 1
sudo nvidia-smi -pl 100
sudo nvidia-smi --lock-gpu-clocks=0,2100 --mode=1
# No difference
# sudo modprobe nvidia NVreg_EnableMSI=1
# sudo modprobe nvidia NVreg_EnablePCIeGen3=1
# sudo modprobe nvidia NVreg_EnableStreamMemOPs=1
# sudo modprobe nvidia NVreg_DynamicPowerManagement=0
# sudo modprobe nvidia NVreg_RegistryDwords="PowerMizerEnable=0x1; PowerMizerDefault=0x1; PowerMizerDefaultAC=0x1"
llama.cpp compile
- CUDA requires gxx 13.2.0 so use mamba
- Requires LD_LIBRARY_PATH for CONDA_PREFIX/lib and /opt/cuda/lib to compile
mamba install gcc=13.2.0 gxx=13.2.
mamba env config vars set LD_LIBRARY_PATH=$CONDA_PREFIX/lib:/opt/cuda/lib
Nvidia Lockups
https://chatgpt.com/c/fbe2f508-4b42-4646-8c0e-c3772c16c8d8 https://forum.manjaro.org/t/downgrade-nvidia-drivers-to-545/159589 https://bbs.archlinux.org/viewtopic.php?id=293400 https://forums.developer.nvidia.com/t/series-550-freezes-laptop/284772/4 https://forums.developer.nvidia.com/t/555-release-feedback-discussion/293652/103?page=6 https://bbs.archlinux.org/viewtopic.php?id=293400 https://forums.developer.nvidia.com/t/555-release-feedback-discussion/293652/22?page=2 https://www.reddit.com/r/linux_gaming/comments/1dnl42h/unstable_experience_after_installing_nvidia_555/ https://forums.developer.nvidia.com/t/device-driver-crash-unable-to-handle-page-fault-after-suspend-resume-with-version-555-58-02-on-linux-kernel-v6-9-9/300380 https://forum.endeavouros.com/t/nvidia-550-drivers-causing-hard-crashing-during-updates/53493/21?page=2
We install 525 - if we’re going back, might as well get to 100W PL?
paru -S nvidia-525xx-dkms nvidia-525xx-utils nvidia-525xx-settings libxnvctrl-525xx opencl-nvidia-525xx
525 requires CUDA 12.0:
mamba install -c "nvidia/label/cuda-12.0.1" cuda-toolkit
mamba env config vars set CUDA_PATH="$CONDA_PREFIX"
mamba env config vars set CUDA_HOME="$CONDA_PREFIX"
mamba install gcc=12.0 gxx=12
mamba activate
https://wiki.archlinux.org/title/Arch_Linux_Archive
sudo systemctl enable --now nvidia-resume.service
https://forums.developer.nvidia.com/t/series-550-freezes-laptop/284772/107 nvidia-open-dkms zswap.enabled=0” and “numa=off
pacman -R nvidia-dkms nvidia-settings nvidia-utils
yay -S nvidia-535xx-dkms nvidia-535xx-utils nvidia-535xx-settings