SYCL https://www.khronos.org/sycl/ XLA https://github.com/openxla/xla
zwaps 11 hours ago | prev | next [–]
If you want to go an run inference/train on one of these fresh and sweet LLMs using bitsandbytes/PEFT, which is really where the excitement is at, you gotta use CUDA pretty much. This is the story now. And the story for the innovation before was the same. Use CUDA, or wait for AMD to catch up a year late and with a worse version of everything.
I mean, sure, you could compile your stuff to XLA or just, I dunno, set up 800 of these cards and train the whole thing on ROCM. But then would you really, really use AMD instead of some TPUs?
AMD made Machine Learning either impossible, unsupported or a chore on most of their hardware for years. Their stack sucks and no one wanted to implement it, and in fact they rarely supported most of their own GPUs.
Yes you have GPUs. But we need also drivers. And software. People have been yelling at AMD about this for years and years.
Instead, AMD has made it clear that Machine Learning is not a priority for the company. Hence, they deserve the lack of traction. Investing in AMD hardware for ML has literally been a mistake at every point in recent history. Imagine if you bought a bunch of (insert last Gen card which is no longer supported by their stack) how dumb you’d look.
Releasing a GPGPU card now? Honestly, why bother? No one is gonna buy it.
https://llvm.org/docs/AMDGPUUsage.html
https://news.ycombinator.com/item?id=26150682
10 points by lhl 8 hours ago | parent | prev | next [–] … lThat being said, it’s important to note that ROCm is Linux only. Not only that, but ROCm’s GPU support has actually been decreasing over the past few years. The current list: https://rocm.docs.amd.com/en/latest/release/gpu_os_support.h… Previously (2022): https://docs.amd.com/bundle/Hardware_and_Software_Reference_…
The ELI5 is that a few years back, AMD split their graphics (RDNA) and compute (CDNA) architectures, which Nvidia does too, but notably (what Nvidia definitely doesn’t do, and a key to their success IMO) AMD also decided they would simply not support any CUDA-parity compute features on Windows or their non “compute” cards. In practice, this means that community/open-source developers will never have, tinker, port, or develop on AMD hardware, while on Nvidia you can start with a GTX/RTX card on your laptop, and use the same code up to an H100 or DGX.
llama.cpp is a super-high profile project, has almost 200 contributiors now, but AFAIK, no contributors from AMD. If AMD doesn’t have the manpower, IMO they should simply be sending nsa free hardware to top open source project/library developers (and on the software side, their #1 priority should be making sure every single current GPU they sell is at least “enabled” if not “supported” in ROCm, on Linux and Windows).
[1] https://github.com/SlyEcho/llama.cpp/tree/hipblas
slavik81 42 minutes ago | unvote | root | parent | next [–] All RDNA2 GPUs and many Vega GPUs could use the same ISA (modulo bugs). Long ago, there was an assumption made that it was safer to treat each minor revision of the GPU hardware as having its own distinct ISA, so that if a hardware bug were found it could be addressed by the compiler without affecting the code generation for other hardware. In practice, this resulted in all but the flagship GPUs being ignored as libraries only ended up getting built for those GPUs in the official binary releases. And in source releases of the libraries, needlessly specific ifdefs frequently broke compilation for all but the flagship ISAs.
There was an implicit assumption that just building for more ISAs was no big deal. That assumption was wrong, but the good news is that big improvements to compatibility can be made even for existing hardware just by more thoughtful handling of the GFX ISAs.
If you know what you’re doing, it’s possible to run ROCm on nearly all AMD GPUs. As I’ve been packaging the ROCm libraries for Debian, I’ve been enabling support for more hardware. Most GFX9 and GFX10 AMD GPUs should be supported in packages on Debian Experimental in the upcoming days. That said, it will need extensive testing on a wide variety of hardware before it’s ready for general use. And we still have lots more libraries to package before all the apps that people care about will run on Debian.
A dive into the AMD driver workflow https://geohot.github.io/blog/jekyll/update/2023/06/07/a-dive-into-amds-drivers.html I ended up getting a response from high level people at AMD. It was still very light on any real technical information, but it did include some great phrases like “I am able to replicate the issues you are facing” and some mockable phrases like “We are hoping that this will improve your perception of AMD products and this will be reflected in your public messaging.”
Though they did end up sending me a ROCm 5.6 driver tarball that seems to be able to run rocm_bandwidth_test and gpu-burn in loops on 2x 7900XTX. So it fixed the main reported issue! Sadly, they asked me not to distribute it, and gave no more details on what the issue is.
A note culturally, I do sadly feel like what they responded to was george is upset and saying bad things about us which is bad for our brand and not holy shit we have a broken driver released panicing thousands of people’s kernels and crashing their GPUs. But it’s a start.
Followup to: https://news.ycombinator.com/item?id=36193625 https://news.ycombinator.com/item?id=36189705
AMD ROCm: a wasted opportunity
https://threedots.ovh/blog/2022/05/amd-rocm-a-wasted-opportunity/ https://news.ycombinator.com/item?id=34832660
danieldk 3 months ago | parent | next [–]
It really is effectively an advertisement for CUDA
100%. At a previous employer, we had some budget left after buying a new NVIDIA GPU machine, but not enough to buy another bunch of workstation/server-class cards. So we decided to buy an additional machine with two Radeon VII, which were quite affordable for the compute it provided. So, we wanted to try if that is a viable path for the future.
Unfortunately, ROCm was death by a thousand paper cuts. You could barely train any model without ROCm crashing with some memory order, weird performance regressions, or garbage output. The only thing we ever really got working was transformer training, with various ROCm libraries and PyTorch pinned to specific revisions and a bunch of patches. Luckily I had the foresight to encode all the dependencies with Nix and put all the Nix files in git, so we could reproduce this exact working version. We attempted to update the stack at various occasions, but it would always lead to more breakage.
Two colleagues meticulously reported bugs for issues that were encountered. But AMD did not really seem interested to try to fix these things (or were just severely understaffed).
After this experience I vowed to never buy AMD again for ML for at least a decade.|
| | | | | |---|---|---| |  ||paulmd 3 months ago \ |
| | | | | |---|---|---| | ||dotnet00 3 months ago \ |
| | | | | |---|---|---| | ||JonChesterfield 3 months ago \ |