This repository contains the code and data for the MMZNO benchmark, which is designed to evaluate the performance of machine learning models on multi-modal zero-shot Ukrainian natural language understanding tasks.
Quick start:
# install dependencies
pip install -r requirements.txt
VLLM_WORKER_MULTIPROC_METHOD=spawn lmms-eval --tasks mmzno_dev --model vllm --model_args model_version=google/gemma-3-4b-it,tensor_parallel_size=2,gpu_memory_utilization=0.95 --batch_size 500 --log_samples --output_path lmms-results --include_path ./tasks