Possible FR: Server endpoint for metadata in loaded model

[BradHutchings]

When the server starts up, it prints out metadata from the model it loads. For example:

llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Meta Llama 8B Instruct v3.1
llama_model_loader: - kv   3:                            general.version str              = v3.1
llama_model_loader: - kv   4:                           general.finetune str              = Instruct
llama_model_loader: - kv   5:                           general.basename str              = Meta-Llama
llama_model_loader: - kv   6:                         general.size_label str              = 8B
llama_model_loader: - kv   7:                            general.license str              = llama3.1
llama_model_loader: - kv   8:                   general.base_model.count u32              = 1
llama_model_loader: - kv   9:                  general.base_model.0.name str              = Meta Llama 3.1 8B
llama_model_loader: - kv  10:          general.base_model.0.organization str              = Meta Llama
llama_model_loader: - kv  11:              general.base_model.0.repo_url str              = https://huggingface.co/meta-
llama/Met...

I would like an endpoint that puts all the key/value pairs into a JSON block. This would be especially helpful for getting the correct English name the model developer intends into a client UI. Such attribution is requested by many of the popular models on Huggingface.

I am going to try to make this work. I need to create a new endpoint in server.cpp. I need to find the code that spits out the k/v pairs and repurpose it to spit out JSON. I'll update my implementation in comments.

I also want to make this work for the older version of llama.cpp that is in llamafile.

I'd appreciate any help on this in comments! When we get it working, I'll make an FR Issue.

-Brad
Brad Hutchings
brad@DemoMachine.net

[BradHutchings]

I think I just found the right way to do this... /v1/models endpoint. I can expand what is in the "meta" value on server.cpp line 2441:

    json model_meta() const {
        return json {
            {"vocab_type",  llama_vocab_type    (model)},
            {"n_vocab",     llama_n_vocab       (model)},
            {"n_ctx_train", llama_n_ctx_train   (model)},
            {"n_embd",      llama_n_embd        (model)},
            {"n_params",    llama_model_n_params(model)},
            {"size",        llama_model_size    (model)},
        };
    }

[BradHutchings]

This works as expected. server.cpp replace the model_meta() function (around line 2441) with:

    json model_meta() const {
		char general_architecture[64];
		char general_type[64];
		char general_name[64];
		char general_version[64];
		char general_finetune[64];
		char general_basename[64];
		char general_size_label[64];
		char general_license[64];
		
		llama_model_meta_val_str(model, "general.architecture",	general_architecture, 64);
		llama_model_meta_val_str(model, "general.type",	general_type, 64);
		llama_model_meta_val_str(model, "general.name",	general_name, 64);
		llama_model_meta_val_str(model, "general.version",	general_version, 64);
		llama_model_meta_val_str(model, "general.finetune",	general_finetune, 64);
		llama_model_meta_val_str(model, "general.basename",	general_basename, 64);
		llama_model_meta_val_str(model, "general.size_label",	general_size_label, 64);
		llama_model_meta_val_str(model, "general.license",	general_license, 64);

        return json {
            {"vocab_type",  llama_vocab_type    (model)},
            {"n_vocab",     llama_n_vocab       (model)},
            {"n_ctx_train", llama_n_ctx_train   (model)},
            {"n_embd",      llama_n_embd        (model)},
            {"n_params",    llama_model_n_params(model)},
            {"size",        llama_model_size    (model)},
			{"general.architecture", general_architecture },
			{"general.type", general_type },
			{"general.name", general_name },
			{"general.version", general_version },
			{"general.finetune", general_finetune },
			{"general.basename", general_basename },
			{"general.size_label", general_size_label },
			{"general.license", general_license },
        };
    }

/v1/models endpoint returns something like this now:

{
  "object": "list",
  "data": [
    {
      "id": "models/Meta-Llama-8B-Instruct-v3.1-q8_0.gguf",
      "object": "model",
      "created": 1725657552,
      "owned_by": "llamacpp",
      "meta": {
        "vocab_type": 2,
        "n_vocab": 128256,
        "n_ctx_train": 131072,
        "n_embd": 4096,
        "n_params": 8030263296,
        "size": 8532942848,
        "general.architecture": "llama",
        "general.type": "model",
        "general.name": "Meta Llama 8B Instruct v3.1",
        "general.version": "v3.1",
        "general.finetune": "Instruct",
        "general.basename": "Meta-Llama",
        "general.size_label": "8B",
        "general.license": "llama3.1"
      }
    }
  ]
}

[slaren]

You can use llama_model_meta_count and llama_model_meta_key_by_index/llama_model_meta_val_str_by_index to enumerate all the meta data.

[BradHutchings]

Thanks for this. Looks like I'd need to be careful with types of each meta data item if I iterate everything that's there. Apps are going to expect some common fields, so I don't feel too bad about picking them out in the server.

[BradHutchings]

Here's some code for to patch the server.cpp that is currently in llamafile:

    svr.Get("/v1/models", [&llama, &params](const httplib::Request& req, httplib::Response& res)
            {
                res.set_header("Access-Control-Allow-Origin", req.get_header_value("Origin"));
                std::time_t t = std::time(0);

                char general_architecture[64];
                char general_type[64];
                char general_name[64];
                char general_version[64];
                char general_finetune[64];
                char general_basename[64];
                char general_size_label[64];
                char general_license[64];
                
                llama_model_meta_val_str(llama.model, "general.architecture",	general_architecture, 64);
                llama_model_meta_val_str(llama.model, "general.type",	general_type, 64);
                llama_model_meta_val_str(llama.model, "general.name",	general_name, 64);
                llama_model_meta_val_str(llama.model, "general.version",	general_version, 64);
                llama_model_meta_val_str(llama.model, "general.finetune",	general_finetune, 64);
                llama_model_meta_val_str(llama.model, "general.basename",	general_basename, 64);
                llama_model_meta_val_str(llama.model, "general.size_label",	general_size_label, 64);
                llama_model_meta_val_str(llama.model, "general.license",	general_license, 64);

                json models = {
                    {"object", "list"},
                    {"data", {
                        {
                            {"id", params.model_alias},
                            {"object", "model"},
                            {"created", t},
                            {"owned_by", "llamacpp"},
                            {"meta", {
                                {"general.architecture", general_architecture},
                                {"general.type", general_type},
                                {"general.name", general_name},
                                {"general.version", general_version},
                                {"general.finetune", general_finetune},
                                {"general.basename", general_basename},
                                {"general.size_label", general_size_label},
                                {"general.license", general_license},
                            }},
                        },
                    }}
                };