Why does llama.cpp generate different outputs with the same seed on different OS?

[tdurieux]

Hello,

I am trying to generate the same output using llama.cpp but the outputs vary depending on the operating system (different binaries). As far as I know, I am using the same parameters.

Do you have an idea what could cause this? What can I do to ensure that the same output is generated?

I am doing my experiments on docker.

alpine

llama.cpp-alpinelatest  | main: build = 607 (ffb06a3)
llama.cpp-alpinelatest  | main: seed  = 12345678
llama.cpp-alpinelatest  | llama.cpp: loading model from /models/7B/ggml-model-q4_0.bin
llama.cpp-alpinelatest  | llama_model_load_internal: format     = ggjt v3 (latest)
llama.cpp-alpinelatest  | llama_model_load_internal: n_vocab    = 32000
llama.cpp-alpinelatest  | llama_model_load_internal: n_ctx      = 512
llama.cpp-alpinelatest  | llama_model_load_internal: n_embd     = 4096
llama.cpp-alpinelatest  | llama_model_load_internal: n_mult     = 256
llama.cpp-alpinelatest  | llama_model_load_internal: n_head     = 32
llama.cpp-alpinelatest  | llama_model_load_internal: n_layer    = 32
llama.cpp-alpinelatest  | llama_model_load_internal: n_rot      = 128
llama.cpp-alpinelatest  | llama_model_load_internal: ftype      = 2 (mostly Q4_0)
llama.cpp-alpinelatest  | llama_model_load_internal: n_ff       = 11008
llama.cpp-alpinelatest  | llama_model_load_internal: n_parts    = 1
llama.cpp-alpinelatest  | llama_model_load_internal: model size = 7B
llama.cpp-alpinelatest  | llama_model_load_internal: ggml ctx size =    0.07 MB
llama.cpp-alpinelatest  | llama_model_load_internal: mem required  = 5407.71 MB (+ 1026.00 MB per state)
llama.cpp-alpinelatest  | .
llama.cpp-alpinelatest  | llama_init_from_file: kv self size  =  256.00 MB
llama.cpp-alpinelatest  | 
llama.cpp-alpinelatest  | system_info: n_threads = 2 / 2 | AVX = 1 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 | 
llama.cpp-alpinelatest  | sampling: repeat_last_n = 64, repeat_penalty = 1.100000, presence_penalty = 0.000000, frequency_penalty = 0.000000, top_k = 40, tfs_z = 1.000000, top_p = 0.950000, typical_p = 1.000000, temp = 0.800000, mirostat = 0, mirostat_lr = 0.100000, mirostat_ent = 5.000000
llama.cpp-alpinelatest  | generate: n_ctx = 512, n_batch = 512, n_predict = 1024, n_keep = 0
llama.cpp-alpinelatest  | 
llama.cpp-alpinelatest  | 
llama.cpp-alpinelatest  |  Building a website can be done in 10 simple steps:
llama.cpp-alpinelatest  | 1. Determine your needs and how you want to position yourself.
llama.cpp-alpinelatest  | 2. Decide on a domain name.
llama.cpp-alpinelatest  | 3. Choose a hosting service.
llama.cpp-alpinelatest  | 4. Determine the look of your site.
llama.cpp-alpinelatest  | 5. Purchase a template, or hire someone to create one for you.
llama.cpp-alpinelatest  | 6. Customize the template.
llama.cpp-alpinelatest  | 7. Determine how you want to manage the content of your site. You can use a free application like WordPress, which is easy to set up and maintain. Or, you can purchase an application like Joomla or Drupal. There are many others available. These will give you more control over your content, but they can be much more complex to operate.
llama.cpp-alpinelatest  | 8. Create the structure of your site, including creating pages that correspond with your needs. For example, if you’re a photographer and want to sell prints on line, there would be a page for each product you are offering—such as “Prints” or “Photo Albums.” If you want to have an events calendar, create the structure of the site so that when someone clicks on the link they will go directly to your calendar.
llama.cpp-alpinelatest  | 9. Create unique content; this can be text, video and/or images. This is where the work begins.
llama.cpp-alpinelatest  |  [end of text]
llama.cpp-alpinelatest  | 
llama.cpp-alpinelatest  | llama_print_timings:        load time =  1550.53 ms
llama.cpp-alpinelatest  | llama_print_timings:      sample time =   108.92 ms /   302 runs   (    0.36 ms per token)
llama.cpp-alpinelatest  | llama_print_timings: prompt eval time =  1490.81 ms /    14 tokens (  106.49 ms per token)
llama.cpp-alpinelatest  | llama_print_timings:        eval time = 44495.60 ms /   301 runs   (  147.83 ms per token)
llama.cpp-alpinelatest  | llama_print_timings:       total time = 46220.36 ms

debian

llama.cpp-debianlatest  | main: build = 607 (ffb06a3)
llama.cpp-debianlatest  | main: seed  = 12345678
llama.cpp-debianlatest  | llama.cpp: loading model from /models/7B/ggml-model-q4_0.bin
llama.cpp-debianlatest  | llama_model_load_internal: format     = ggjt v3 (latest)
llama.cpp-debianlatest  | llama_model_load_internal: n_vocab    = 32000
llama.cpp-debianlatest  | llama_model_load_internal: n_ctx      = 512
llama.cpp-debianlatest  | llama_model_load_internal: n_embd     = 4096
llama.cpp-debianlatest  | llama_model_load_internal: n_mult     = 256
llama.cpp-debianlatest  | llama_model_load_internal: n_head     = 32
llama.cpp-debianlatest  | llama_model_load_internal: n_layer    = 32
llama.cpp-debianlatest  | llama_model_load_internal: n_rot      = 128
llama.cpp-debianlatest  | llama_model_load_internal: ftype      = 2 (mostly Q4_0)
llama.cpp-debianlatest  | llama_model_load_internal: n_ff       = 11008
llama.cpp-debianlatest  | llama_model_load_internal: n_parts    = 1
llama.cpp-debianlatest  | llama_model_load_internal: model size = 7B
llama.cpp-debianlatest  | llama_model_load_internal: ggml ctx size =    0.07 MB
llama.cpp-debianlatest  | llama_model_load_internal: mem required  = 5407.71 MB (+ 1026.00 MB per state)
llama.cpp-debianlatest  | .
llama.cpp-debianlatest  | llama_init_from_file: kv self size  =  256.00 MB
llama.cpp-debianlatest  | 
llama.cpp-debianlatest  | system_info: n_threads = 2 / 24 | AVX = 1 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 | 
llama.cpp-debianlatest  | sampling: repeat_last_n = 64, repeat_penalty = 1.100000, presence_penalty = 0.000000, frequency_penalty = 0.000000, top_k = 40, tfs_z = 1.000000, top_p = 0.950000, typical_p = 1.000000, temp = 0.800000, mirostat = 0, mirostat_lr = 0.100000, mirostat_ent = 5.000000
llama.cpp-debianlatest  | generate: n_ctx = 512, n_batch = 512, n_predict = 1024, n_keep = 0
llama.cpp-debianlatest  | 
llama.cpp-debianlatest  | 
llama.cpp-debianlatest  |  Building a website can be done in 10 simple steps:
llama.cpp-debianlatest  | 1. Determine your needs and how you want to position yourself.
llama.cpp-debianlatest  | 2. Decide on a domain name.
llama.cpp-debianlatest  | 3. Choose a hosting service.
llama.cpp-debianlatest  | 4. Determine the look of your site.
llama.cpp-debianlatest  | 5. Purchase a template, or hire someone to create one for you.
llama.cpp-debianlatest  | 6. Customize the template.
llama.cpp-debianlatest  | 7. Insert content into the template.
llama.cpp-debianlatest  | 8. Publish the site on your hosting server.
llama.cpp-debianlatest  | 9. Promote it.
llama.cpp-debianlatest  | 10. Maintain and update it.
llama.cpp-debianlatest  | It's important that you decide what kind of website you want before you begin because each stage requires a different set of skills and techniques, which means you can either spend a lot more time trying to figure things out yourself (and wasting your valuable time), or you can hire someone who specializes in building websites.
llama.cpp-debianlatest  | Before you build the site, you'll need to decide on your goal, including how you want to position yourself. Your website should represent an accurate picture of what you do and how you do it. This means that if you're a writer, for example, you shouldn't have a graphic design service listed as one of the things you offer (unless you also happen to be a graphic designer). And don't give people reasons not to hire you--make sure your site reflects who you are and what you do.
llama.cpp-debianlatest  | The next step is to determine what kind of domain name you want to use for your website. If you already have the perfect domain name, then that's great! You can skip right on to point #3 below. If not, read on…
llama.cpp-debianlatest  | How To Choose The Best Domain Name For Your Site
llama.cpp-debianlatest  | There are a number of ways to find a suitable domain name, but most experts recommend that you do your best to find something that is related to your business or industry. This way, people will have an easier time remembering it and typing it into their browsers.
llama.cpp-debianlatest  | The best way to find good domain names is by using a tool called a “domain name generator”. These tools search the web for all of the websites in your chosen field/industry and show you what domains are available (i.e., not already taken). The next step will be to figure out how much you can afford to spend on hosting, because this will also determine which hosting services you’ll use for your site.
llama.cpp-debianlatest  | Choose a Domain Name Hosting Service
llama.cpp-debianlatest  | Website hosting is the service that gives you space on their servers so that you can create and maintain your website. They provide the necessary technical means for you to host your own website (with all of its content) over the internet, from anywhere in the world. It essentially acts as the “home” of your site.
llama.cpp-debianlatest  | There are literally thousands of hosting services out there today, but we recommend that you go with one of the most reputable ones. If they’re not well-known and established, it could be a problem for you if something goes wrong with their service or support--which is why we suggest you stick to one like GoDaddy or Site5 (we use them for our own websites).
llama.cpp-debianlatest  | It really depends on how much money you have. The best hosting services will charge you $10 or more per month, but they’re the most established and reliable. Cheaper ones may only cost you a few dollars per month. The cheapest hosting services are often referred to as shared web hosting--which is okay for beginners or those who simply don't want to spend much money on their website.
llama.cpp-debianlatest  |  [end of text]
llama.cpp-debianlatest  | 
llama.cpp-debianlatest  | llama_print_timings:        load time =  1452.43 ms
llama.cpp-debianlatest  | llama_print_timings:      sample time =   282.24 ms /   838 runs   (    0.34 ms per token)
llama.cpp-debianlatest  | llama_print_timings: prompt eval time = 50855.15 ms /   528 tokens (   96.32 ms per token)
llama.cpp-debianlatest  | llama_print_timings:        eval time = 117419.70 ms /   835 runs   (  140.62 ms per token)
llama.cpp-debianlatest  | llama_print_timings:       total time = 168724.75 ms

centos

llama.cpp-centoslatest  | main: build = 607 (ffb06a3)
llama.cpp-centoslatest  | main: seed  = 12345678
llama.cpp-centoslatest  | llama.cpp: loading model from /models/7B/ggml-model-q4_0.bin
llama.cpp-centoslatest  | llama_model_load_internal: format     = ggjt v3 (latest)
llama.cpp-centoslatest  | llama_model_load_internal: n_vocab    = 32000
llama.cpp-centoslatest  | llama_model_load_internal: n_ctx      = 512
llama.cpp-centoslatest  | llama_model_load_internal: n_embd     = 4096
llama.cpp-centoslatest  | llama_model_load_internal: n_mult     = 256
llama.cpp-centoslatest  | llama_model_load_internal: n_head     = 32
llama.cpp-centoslatest  | llama_model_load_internal: n_layer    = 32
llama.cpp-centoslatest  | llama_model_load_internal: n_rot      = 128
llama.cpp-centoslatest  | llama_model_load_internal: ftype      = 2 (mostly Q4_0)
llama.cpp-centoslatest  | llama_model_load_internal: n_ff       = 11008
llama.cpp-centoslatest  | llama_model_load_internal: n_parts    = 1
llama.cpp-centoslatest  | llama_model_load_internal: model size = 7B
llama.cpp-centoslatest  | llama_model_load_internal: ggml ctx size =    0.07 MB
llama.cpp-centoslatest  | llama_model_load_internal: mem required  = 5407.71 MB (+ 1026.00 MB per state)
llama.cpp-centoslatest  | .
llama.cpp-centoslatest  | llama_init_from_file: kv self size  =  256.00 MB
llama.cpp-centoslatest  | 
llama.cpp-centoslatest  | system_info: n_threads = 2 / 24 | AVX = 1 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 | 
llama.cpp-centoslatest  | sampling: repeat_last_n = 64, repeat_penalty = 1.100000, presence_penalty = 0.000000, frequency_penalty = 0.000000, top_k = 40, tfs_z = 1.000000, top_p = 0.950000, typical_p = 1.000000, temp = 0.800000, mirostat = 0, mirostat_lr = 0.100000, mirostat_ent = 5.000000
llama.cpp-centoslatest  | generate: n_ctx = 512, n_batch = 512, n_predict = 1024, n_keep = 0
llama.cpp-centoslatest  | 
llama.cpp-centoslatest  | 
llama.cpp-centoslatest  |  Building a website can be done in 10 simple steps:
llama.cpp-centoslatest  | 1. Determine your needs and how you want to position yourself.
llama.cpp-centoslatest  | 2. Decide on a domain name.
llama.cpp-centoslatest  | 3. Choose a hosting service.
llama.cpp-centoslatest  | 4. Determine the look of your site.
llama.cpp-centoslatest  | 5. Purchase a template, or hire someone to create one for you.
llama.cpp-centoslatest  | 6. Customize the template.
llama.cpp-centoslatest  | 7. Determine how you want to manage the content of your site. This will either be through static pages (a fixed number of pages), or dynamic pages (content that changes).
llama.cpp-centoslatest  | 8. Create a sitemap, if necessary.
llama.cpp-centoslatest  | 9. Register with search engines and link exchange partners.
llama.cpp-centoslatest  | 10. Promote your website.
llama.cpp-centoslatest  | The main goal for any new web page is to attract visitors, which will ultimately lead to sales/inquiries/donations, etc. Once you have decided on the purpose of your site (do not start a site without knowing why), then you can focus in on what you want to accomplish with it. If your purpose is to get more traffic to your business, then you should create a marketing campaign that will generate leads. If your purpose is to sell products online, and your competition is eBay or Amazon, etc., then you need to do everything possible to outperform them in search engine rankings.
llama.cpp-centoslatest  | Step 2: Determine your needs and how you want to position yourself
llama.cpp-centoslatest  | The next step after deciding on the purpose of your site is to make sure that your site meets your specific needs (do not overextend yourself). There are three main purposes for a new web page: to build a brand, to sell products or services online, or to provide useful information to others. These three areas should be used as the cornerstones when designing and building your website, because they will determine how you position yourself on the Internet.
llama.cpp-centoslatest  | For example, if you want to build a brand for your business, then you need to focus on creating great content (articles) that can be found via search engines or by people who may visit your site directly. A blog is the perfect way to do this because it allows you to create a steady flow of new quality content which can be ranked well in search engine results. If your purpose for building a website is to sell products/services online, then make sure that every page on your site has a form for someone to contact you. You do not want people to have to click off your site to complete their purchase (eBay).
llama.cpp-centoslatest  | Step 3: Choose the right host and domain name
llama.cpp-centoslatest  | Finding the best hosting company is crucial for success because this will be one of the biggest determiners of how well your page ranks on search engines. There are hundreds of web hosts out there, but I recommend that you use a trusted host such as Bluehost or Hostgator (which are used by 60% of all websites). Avoid cheap hosting companies because these will often put your site in the same data center as millions of other sites which will make it very difficult to rank. Additionally, they rarely provide good support and you want to be sure that you can rely on a good host when problems arise during setup or maintenance.
llama.cpp-centoslatest  | I also recommend using GoDaddy since this is one of the largest domain registration companies with over 50 million registered domains (out of 168 million total). I prefer them because they use Google Adwords as their top level domain for all new websites, and this can help your page rank more quickly. They are also very reliable and provide quality support if you ever run into problems or need to make adjustments in the future.
llama.cpp-centoslatest  | Step 4: Get your site up! (you’re almost done)
llama.cpp-centoslatest  | You’ve set everything up, but now it needs to go online so that people can actually find you on Google when they search for relevant keywords and phrases. The first thing you will need to do is buy a domain name since this is the main address of your page. You can then install WordPress onto the host as your website’s operating system (or any other CMS).
llama.cpp-centoslatest  | Step 5: Set up an auto-response email for people who contact you directly
llama.cpp-centoslatest  | You should also consider setting up auto-responses to emails that are sent directly to your site since it will be easy for potential buyers to send you questions and express interest. This is because many of them will likely have no idea how to find you on Google or what to write in a search engine when they want to see who else is selling the product, so they might just send an email directly from their computer. This makes it easy for them to accidentally stumble upon your site.
llama.cpp-centoslatest  | Step 6: Put up relevant and useful content
llama.cpp-centoslatest  | 
llama.cpp-centoslatest  | llama_print_timings:        load time =  1471.57 ms
llama.cpp-centoslatest  | llama_print_timings:      sample time =   329.25 ms /  1024 runs   (    0.32 ms per token)
llama.cpp-centoslatest  | llama_print_timings: prompt eval time = 78066.40 ms /   785 tokens (   99.45 ms per token)
llama.cpp-centoslatest  | llama_print_timings:        eval time = 141436.78 ms /  1020 runs   (  138.66 ms per token)
llama.cpp-centoslatest  | llama_print_timings:       total time = 220024.14 ms

Ubuntu

llama.cpp-ubuntulatest  | main: build = 607 (ffb06a3)
llama.cpp-ubuntulatest  | main: seed  = 12345678
llama.cpp-ubuntulatest  | llama.cpp: loading model from /models/7B/ggml-model-q4_0.bin
llama.cpp-ubuntulatest  | llama_model_load_internal: format     = ggjt v3 (latest)
llama.cpp-ubuntulatest  | llama_model_load_internal: n_vocab    = 32000
llama.cpp-ubuntulatest  | llama_model_load_internal: n_ctx      = 512
llama.cpp-ubuntulatest  | llama_model_load_internal: n_embd     = 4096
llama.cpp-ubuntulatest  | llama_model_load_internal: n_mult     = 256
llama.cpp-ubuntulatest  | llama_model_load_internal: n_head     = 32
llama.cpp-ubuntulatest  | llama_model_load_internal: n_layer    = 32
llama.cpp-ubuntulatest  | llama_model_load_internal: n_rot      = 128
llama.cpp-ubuntulatest  | llama_model_load_internal: ftype      = 2 (mostly Q4_0)
llama.cpp-ubuntulatest  | llama_model_load_internal: n_ff       = 11008
llama.cpp-ubuntulatest  | llama_model_load_internal: n_parts    = 1
llama.cpp-ubuntulatest  | llama_model_load_internal: model size = 7B
llama.cpp-ubuntulatest  | llama_model_load_internal: ggml ctx size =    0.07 MB
llama.cpp-ubuntulatest  | llama_model_load_internal: mem required  = 5407.71 MB (+ 1026.00 MB per state)
llama.cpp-ubuntulatest  | .
llama.cpp-ubuntulatest  | llama_init_from_file: kv self size  =  256.00 MB
llama.cpp-ubuntulatest  | 
llama.cpp-ubuntulatest  | system_info: n_threads = 2 / 24 | AVX = 1 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 | 
llama.cpp-ubuntulatest  | sampling: repeat_last_n = 64, repeat_penalty = 1.100000, presence_penalty = 0.000000, frequency_penalty = 0.000000, top_k = 40, tfs_z = 1.000000, top_p = 0.950000, typical_p = 1.000000, temp = 0.800000, mirostat = 0, mirostat_lr = 0.100000, mirostat_ent = 5.000000
llama.cpp-ubuntulatest  | generate: n_ctx = 512, n_batch = 512, n_predict = 1024, n_keep = 0
llama.cpp-ubuntulatest  | 
llama.cpp-ubuntulatest  | 
llama.cpp-ubuntulatest  |  Building a website can be done in 10 simple steps:
llama.cpp-ubuntulatest  | 1. Determine your needs and how you want to position yourself.
llama.cpp-ubuntulatest  | 2. Decide on a domain name.
llama.cpp-ubuntulatest  | 3. Choose a hosting service.
llama.cpp-ubuntulatest  | 4. Determine the look of your site.
llama.cpp-ubuntulatest  | 5. Purchase a template, or hire someone to create one for you.
llama.cpp-ubuntulatest  | 6. Customize the template.
llama.cpp-ubuntulatest  | 7. Insert content into the template.
llama.cpp-ubuntulatest  | 8. Publish the site on your hosting server.
llama.cpp-ubuntulatest  | 9. Promote it.
llama.cpp-ubuntulatest  | 10. Maintain and update it.
llama.cpp-ubuntulatest  | It's important that you decide what kind of website you want before you begin because each stage requires a different set of skills and techniques, which means you can either spend a lot more time trying to figure things out yourself (and wasting your valuable time), or you can hire someone who specializes in building websites.
llama.cpp-ubuntulatest  | Before you build the site, you'll need to decide on your goal, including how you want to position yourself. Your website should represent an accurate picture of what you do and how you do it. This means that if you're a writer, for example, you shouldn't have a graphic design service listed as one of the things you offer (unless you also happen to be a graphic designer). And don't give people reasons not to hire you--make sure your site reflects who you are and what you do.
llama.cpp-ubuntulatest  | The next step is to determine what kind of domain name you want to use for your website. If you already have the perfect domain name, then that's great! You can skip right on to point #3 below. If not, read on…
llama.cpp-ubuntulatest  | How To Choose The Best Domain Name For Your Site
llama.cpp-ubuntulatest  | There are a number of ways to find a suitable domain name, but most experts recommend that you do your best to find something that is related to your business or industry. This way, people will have an easier time remembering it and typing it into their browsers.
llama.cpp-ubuntulatest  | The best way to find good domain names is by using a tool called a “domain name generator”. These tools search the web for all of the websites in your chosen field/industry and show you what domains are available (i.e., not already taken). The next step will be to figure out how much you can afford to spend on hosting, because this will also determine which hosting services you’ll use for your site.
llama.cpp-ubuntulatest  | Choose a Domain Name Hosting Service
llama.cpp-ubuntulatest  | Website hosting is the service that gives you space on their servers so that you can create and maintain your website. They provide the necessary technical means for you to host your own website (with all of its content) over the internet, from anywhere in the world. It essentially acts as the “home” of your site.
llama.cpp-ubuntulatest  | There are literally thousands of hosting services out there today, but we recommend that you go with one of the most reputable ones. If they’re not well-known and established, it could be a problem for you if something goes wrong with their service or support--which is why we suggest you stick to one like GoDaddy or Site5 (we use them for our own websites).
llama.cpp-ubuntulatest  | It really depends on how much money you have. The best hosting services will charge you $10 or more per month, but they’re the most established and reliable. Cheaper ones may only cost you a few dollars per month. The cheapest hosting services are often referred to as shared web hosting--which is okay for beginners or those who simply don't want to spend much money on their website.
llama.cpp-ubuntulatest  |  [end of text]
llama.cpp-ubuntulatest  | 
llama.cpp-ubuntulatest  | llama_print_timings:        load time =  1451.36 ms
llama.cpp-ubuntulatest  | llama_print_timings:      sample time =   319.17 ms /   838 runs   (    0.38 ms per token)
llama.cpp-ubuntulatest  | llama_print_timings: prompt eval time = 50923.23 ms /   528 tokens (   96.45 ms per token)
llama.cpp-ubuntulatest  | llama_print_timings:        eval time = 118251.09 ms /   835 runs   (  141.62 ms per token)
llama.cpp-ubuntulatest  | llama_print_timings:       total time = 169660.99 ms

[Green-Sky]

can you try and use the same binary on all platforms, so we can see if it is actually the operating system/hardware.

this comes down to floating point imprecisions on various hardware and compilers and compilation flags.

[tdurieux]

Everything is running on the same hardware, therefore, it comes to the os/libraries. I don't succeed to create a binary that is working on all OSs, I am not sure if it is possible.

[Green-Sky]

try setting LLAMA_STATIC when building. and build it on the os with the oldest glibc. that should result in a pretty portable binary.

[Green-Sky]

we also have this here https://github.com/ggerganov/llama.cpp/blob/acc111caf93fc6681450924df9f99679c384c59e/CMakeLists.txt#L372-L374
not sure if that can be applied too

[tdurieux]

I created a portable binary and it is now generating the exact same output. Thank you for your help!

[d-kleine]

try setting LLAMA_STATIC when building. and build it on the os with the oldest glibc. that should result in a pretty portable binary.

Ehm, how can you do that?