Stable Beluga/llama2?

[daveaitel]

This is absolutely awesome work. I'd love to see how to use this with Stable Beluga or LLAMA2? It looks like it should be a quick port, unless I'm mistaken?

[rlouf]

Thank you! Could you point to the repos where I can find these models? I may be able to answer your question then

[coltonflowers1]

Here is a popular C++ implementation of LLAMA and a couple other related models, though it looks like they have their own grammar-based guided generation.

[rlouf]

They do, but our method has better scaling properties. It looks like it is going to be a bit of work to integrate, but could do it if there's a lot of demand for it.

[daveaitel]

Here is a chunk of code from one of my projects. :)

-dave

 if use_beluga:
        tokenizer = AutoTokenizer.from_pretrained("stabilityai/StableBeluga-7B", use_fast=False)
        model = AutoModelForCausalLM.from_pretrained("stabilityai/StableBeluga-7B", torch_dtype=torch.float16, low_cpu_mem_usage=True, device_map="auto")
        pipelines["beluga-tokenizer"] = tokenizer
        pipelines["beluga-model"] = model

def process_chunk(chunk, prelude, system_prompt, tokenizer, model):
    """
    Process a single chunk of text.
    """
    if not system_prompt:
        system_prompt = "### System:\nYou are an AI that follows instructions extremely well. Help as much as you can.\n\n"
    prompt = f"{system_prompt}### User: {prelude}{chunk}\n\n### Assistant:\n"

    with cudamutex:
        inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
        output = model.generate(**inputs, do_sample=True, top_p=0.95, top_k=0, max_new_tokens=1000)

        ret = tokenizer.decode(output[0], skip_special_tokens=True)
        ret = ret.replace(prompt, "")

    return ret

def stable_beluga_llm(text, prelude="", system_prompt=None):
    """
    A generic way to run text through our LLM. Handles chunking natively for long values of text.
    """
    global pipelines
    model = pipelines["beluga-model"]
    tokenizer = pipelines["beluga-tokenizer"]

    # Tokenize the entire text
    tokens = tokenizer.tokenize(text)

    processed_texts = []

    # Define a maximum token threshold, keeping some margin for system_prompt and Assistant's tokens
    MAX_TOKENS = 1500  # Adjust this as per model's limit

    # Split the tokens into chunks that respect the MAX_TOKENS limit
    for i in range(0, len(tokens), MAX_TOKENS):
        chunk_tokens = tokens[i:i+MAX_TOKENS]

        # Convert chunk tokens back to text
        chunk_text = tokenizer.convert_tokens_to_string(chunk_tokens)
        processed_texts.append(process_chunk(prelude, chunk_text, system_prompt, tokenizer, model))

    # Combine all the chunks
    ret = ' '.join(processed_texts)

[rlouf]

What does cudamutex do?

[daveaitel]

It's just a simple mutex that avoids more than one thread going in there.

…

On Fri, Aug 18, 2023, 10:02 PM Rémi Louf ***@***.***> wrote: What does cudamutex do? — Reply to this email directly, view it on GitHub <#230 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE25MYXQFQKRVMSPICC57UTXWANCTANCNFSM6AAAAAA3RRPF5Y> . You are receiving this because you authored the thread.Message ID: ***@***.*** com>