Skip to content

ux: move translator load msg into translator instantiation #1184

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
May 22, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions docs/source/translation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -59,17 +59,17 @@ Configuration file
Translation function is configured in the ``run`` section of a configuration with the following keys:

``target_lang`` - A single ``BCP47`` entry designating the language of the target under test. "ja", "fr", "jap" etc.
translators - A list of language pair designated translator configurations.
``langproviders`` - A list of language pair designated translator configurations.

* Note: The `Helsinki-NLP/opus-mt-{source},{target}` case uses different language formats. The language codes used to name models are inconsistent.
Two-letter codes can usually be found `here <https://developers.google.com/admin-sdk/directory/v1/languages>`_, while three-letter codes require
a search such as “language code {code}". More details can be found `here <https://github.com/Helsinki-NLP/OPUS-MT-train/tree/master/models>`_.

A translator configuration is provided using the project's configurable pattern with the following required keys:
A language provider configuration is provided using the project's configurable pattern with the following keys:

* ``language`` - A ``,`` separated pair of ``BCP47`` entires describing translation format provided by the configuration
* ``model_type`` - the module and optional instance class to be instantiated. local, remote, remote.DeeplTranslator etc.
* ``model_name`` - (optional) the model name loaded for translation, required for ``local`` translator model_type
* ``language`` - (required) A ``,`` separated pair of ``BCP47`` entires describing translation format provided by the configuration
* ``model_type`` - (required) the ``langproviders`` module and optional instance class to be instantiated; ``local``, ``remote``, ``remote.DeeplTranslator`` etc.
* ``model_name`` - (conditional) the model name loaded for translation. This field is required for ``local`` translator ``model_type``

(Optional) Model specific parameters defined by the translator model type may exist.

Expand Down
25 changes: 15 additions & 10 deletions garak/harnesses/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
inherit from.
"""


import importlib
import json
import logging
import types
Expand All @@ -28,18 +28,23 @@ def _initialize_runtime_services():
"""Initialize and validate runtime services required for a successful test"""

from garak.exception import GarakException
import garak.langservice

# TODO: this block may be gated in the future to ensure it is only run once. At this time
# only one harness will execute per run so the output here is reasonable.
try:
msg = "🌐 Loading Language services if required."
logging.info(msg)
print(msg)
garak.langservice.load()
except GarakException as e:
logging.critical("❌ Language setup failed! ❌", exc_info=e)
raise e
service_names = ["garak.langservice"]
for service_name in service_names:
logging.info("service import: " + service_name)
service = importlib.import_module(service_name)
try:
if service.enabled():
symbol, msg = service.start_msg()
if len(msg):
logging.info(msg)
print(f"{symbol} {msg}")
service.load()
except GarakException as e:
logging.critical(f"❌ {service_name} setup failed! ❌", exc_info=e)
raise e


class Harness(Configurable):
Expand Down
1 change: 1 addition & 0 deletions garak/langproviders/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,7 @@ class LangProvider(Configurable):
"""Base class for objects that provision language"""

def __init__(self, config_root: dict = {}) -> None:

self._load_config(config_root=config_root)

self.source_lang, self.target_lang = self.language.split(",")
Expand Down
28 changes: 27 additions & 1 deletion garak/langservice.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,9 @@


import logging
from garak import _config, _plugins
from typing import List

from garak import _config, _plugins
from garak.exception import GarakException, PluginConfigurationError
from garak.langproviders.base import LangProvider
from garak.langproviders.local import Passthru
Expand All @@ -16,6 +17,31 @@
native_langprovider = None


def tasks() -> List[str]:
"""number of translators to deal with, minus the no-op one"""
models_to_init = []
for t in _config.run.langproviders:
if t["model_type"] == "local.Passthru": # extra guard
continue
model_descr = f"{t['language']}->{t['model_type']}"
if "model_name" in t:
model_descr += f"[{t['model_name']}]"
models_to_init.append(model_descr)
return models_to_init


def enabled() -> bool:
"""are all requirements met for language service to be enabled"""
if hasattr(_config.run, "langproviders"):
return len(_config.run.langproviders) > 1
return False


def start_msg() -> str:
"""return a start message, assumes enabled"""
return "🌐", "loading language services: " + " ".join(tasks())


def _load_langprovider(language_service: dict = {}) -> LangProvider:
"""Load a single language provider based on the configuration provided."""
langprovider_instance = None
Expand Down