this generate a different result

[wwbnjsace]

When using the exact same parameters, the results generated by this method are inconsistent compared to those generated by the official method. The gender and music style differ, and sometimes the quality is worse than that of the official method. Why is this happening? i use FP16

[DocShotgun]

The most likely cause is that there's just a lot of randomness involved in the generation process. With temp of 1, top-p of 0.93, and rep pen of 1.1, generations will vary significantly between multiple runs with the same parameters, simply from different tokens being randomly sampled. As far as turbo found during implementation, the pre-sampling logits aren't significantly different.

As for the results sometimes being subjectively "worse" in quality than the official method, that could just be due to randomness as above. Not sure if there's any objective way to measure something like that, but if you don't like the result, then you're free to just use the original implementation.

[sujitvasanth]

I've been looking a lot into Yue pipelines and deterministic outputs (Meaning the same prompt produces indiscernible output differences between runs on the same machine).

1, the Yue original repo is deterministic but very slow
2. Yue-exllamav2 is deterministic for stage 2 at all quantisations and fst if batched
3. Yue-exllamax2 is deterministic for stage 1 if the unquantized model is used with exl2_config.MAX_Q_GEMM_ROWS = 0
4. huggingface stage 1 bits and bytes 4 bit quantisation is deterministic too

The non-determinism in exllamav2 seems to be coming from the compiled CUDA atomic additions during race conditions from the C compiled module in exllama. Im trying at present to rewrite and recompile that part of the kernel...
exllamav2/exllamav2_ext/cuda/comp_units/../q_gemm_kernel.cuh

chatgot will easily generate deterministic but slower code for this function
Does anyone have experience at building exllama CUDA kernal from source?

[sgsdxzy]

As far as I know, exl2 is non-deterministic in exchange for better performance.

[turboderp]

Yes, the nondeterminism is a tradeoff for speed.

I maintain that a+b = b+a. The difference in practice amounts to rounding noise, and any application that relies on a repeatable noise pattern is misguided, at least outside of cryptography or hashing or whatever. If you do build anything around the idea of LLM inference being deterministic, you'll be disappointed to learn that almost all "deterministic" implementations will change the order of operations (and with it the rounding noise) depending on factors like batch size, runtime version and compute capability.

I do get that some superficial form of determinism is nice to have, at least for testing purposes, so I am taking it into account in the new kernels for ExLlamaV3. They may be somewhat back-portable to V2, or V3 may support EXL2 models, I'm not sure yet.

Just keep in mind you can not actually validate a function with chaotic-dynamic characteristics in a deterministic context, so arguably the nondeterminism is a feature, not a bug. Without it, at least make sure to introduce some deliberate noise while testing.

As for compiling ExLlama, on Linux you generally just need the CUDA Toolkit installed, and then you can install it with pip install . in the repo directory. You can also do EXLLAMA_NOCOMPILE=1 pip install . to install it in JIT mode. In this mode it will build and cache the C++ extension on first use, and whenever any of the source files is updated.

[sujitvasanth]

@turboderp Really appreciate your comments, and your library produces outstanding speed increases, but at a time when text/image LLMs were/are the main us,e but now this is being broadened, and I think audio brings new challenges. I appreciate your comments about batch size and pytorch versions.

1. the new scenario is when an end user wishes to iterate on a song... the song needs to be the same if adding to it or if changing a lyric or chorus for verse, style, instrument etc...
   if the song itself changes markedly on each run its like walking on randomly shifting sand, if that makes sense? Also you cant easily get back or work on a particularly good output
   as I said earlier the level of "determinism" of using exllamav2 with the default non-quantised cot model is acceptable (indistinguishable between runs) but we need the low VRAM usage of the quantised models. admittedly, speed will be slower but probably faster than inference with the non-quantised version.

2. I managed to compile from the newest version of your repo with a custom CUDA kernel after a bit of trial and error... but your newest version of exllama github repo breaks the yeu-exllama code due to some function call changes and would be less ideal for other users needing to make many changes to existing yue-exllamav2 infer_stage1.py and infer_stage2.py
   I failed to compile with the older version of the library that yue-exllama's wheels point to...i presume a problem with python library versioning...even that isn't pluig and play - I needed to change some of the exllama Cache imports - so presumably I'm still not using the correct exllama library version that normally gets loaded.

3. can I confirm that the atomic addition part that needs changing is only in
   exllamav2/exllamav2_ext/cuda/q_gemm_kernel.cuh
   or if any other parts of the kernel would need changing to eliminate the atomic addition issues with quantised exl2 models?

4. I'm using a thread lock strategy, meaning that each addition has to be done in sequence,
   although I cant see why we can't have a cache for the threads and add them in a predetermined order. but presumably this is essentially the same thing.
   is this overly simplistic in terms of a fix for the atomic addition?

I would appreciate any help with the above...should I raise an issue on exxlamav2 repo or is here better?

[sujitvasanth]

update in my deterministic Odysee...
found that the exllamav2 version 0.2.3 allows running the yeu-exllamav2 inference code without alteration.. would be helpful to state this in yeu-exllama requirements

The simplest way to make CUDA kernel adjustments is you can edit the python library directly in place and alter the CUDA code
JIT can be forced to recompile by using the command line
rm -rf ~/.cache/torch_extensions/*

regarding CUDA kernels
I made the changes to atomic additions and dot products in exllamav2/exllamav2_ext/cuda/q_gemm_kernel.cuh but output is still not deterministic
I reworking the dequantisation algorithms in cuda/quant directory as that also is not deterministic - but just ran now and still not deterministic!

[sujitvasanth]

I've given up for now... got it working deterministically using bitsandbytes 4 bit for stage 1 and exllama for stage 2.. VRAM 7Gb for a 3 minute song (6 segments at 3000 tokens per segment) takes 10 minutes to generate on rtx3090. Was able to use huggingface streaming to reduce VRAM.