Feature: multilingual machine translation (#943)

Author: jmartin-tech

docs/source/configurable.rst

@@ -107,6 +107,8 @@ such as ``show_100_pass_modules``.
 * ``eval_threshold`` - At what point in the 0..1 range output by detectors does a result count as a successful attack / hit
 * ``user_agent`` - What HTTP user agent string should garak use? ``{version}`` can be used to signify where garak version ID should go
 * ``soft_probe_prompt_cap`` - For probes that auto-scale their prompt count, the preferred limit of prompts per probe
+* ``target_lang`` - A single bcp47 value the target application for LLM accepts as prompt and output
+* ``translators`` - A list of configurations representing translators for converting from probe bcp47 language to land_spec target bcp47 languages
 
 ``plugins`` config items
 """"""""""""""""""""""""

docs/source/index.rst

@@ -61,6 +61,7 @@ Code reference
    command
    exception
    interactive
+   langservice
    payloads
    _config
    _plugins

docs/source/langservice.rst

@@ -0,0 +1,193 @@
+The ``langservice`` module in garak is designed to handle text language tasks using various translation services and models. 
+It provides an entry point to translation support backed by translators, each implementing different translation strategies and models, including both cloud-based services, 
+like `DeepL<https://www.deepl.com/>`_ and `NVIDIA Riva<https://build.nvidia.com/nvidia/megatron-1b-nmt>`_, and local models like facebook/m2m100 available on `Hugging Face<https://huggingface.co/>`_.
+
+garak.langservice
+=================
+
+.. automodule:: garak.langservice
+   :members:
+   :undoc-members:
+   :show-inheritance:   
+
+Translation support
+===================
+
+This module provides translation support for probe and detector keywords and triggers.
+Allowing testing of models that accept and produce text in languages other than the language the plugin was written for.
+
+* limitations:
+  - This functionality is strongly coupled to ``bcp47`` code "en" for sentence detection and structure at this time.
+  - Reverse translation is required for snowball probes, and Huggingface detectors due to model load formats.
+  - Huggingface detectors primarily load English models. Requiring a target language NLI model for the detector.
+  - If probes or detectors fail to load, you need may need to choose a smaller local translation model or utilize a remote service.
+  - Translation may add significant execution time to the run depending on resources available.
+
+Supported translation services
+------------------------------
+
+- Huggingface
+  - This project supports usage of the following translation models:
+    - `Helsinki-NLP/opus-mt-{<source_lang>-<target_lang>} <https://huggingface.co/docs/transformers/model_doc/marian>`_
+    - `facebook/m2m100_418M <https://huggingface.co/facebook/m2m100_418M>`_
+    - `facebook/m2m100_1.2B <https://huggingface.co/facebook/m2m100_1.2B>`_
+- `DeepL <https://www.deepl.com/docs-api>`_
+- `NVIDIA Riva <https://build.nvidia.com/nvidia/megatron-1b-nmt>`_
+
+API KEY Requirements
+--------------------
+
+To use use DeepL API or Riva API to translate probe and detector keywords and triggers from cloud services an API key must be supplied.
+
+API keys for the preferred service can be obtained in following locations:
+- `DeepL <https://www.deepl.com/en/pro-api>`_
+- `Riva <https://build.nvidia.com/nvidia/megatron-1b-nmt>`_
+
+Supported languages for remote services:
+- `DeepL <https://developers.deepl.com/docs/resources/supported-languages>`_
+- `Riva <https://docs.nvidia.com/nim/riva/nmt/latest/getting-started.html#supported-languages>`_
+
+API keys can be stored in environment variables with the following commands:
+
+DeepL
+~~~~~
+
+.. code-block:: bash
+
+    export DEEPL_API_KEY=xxxx
+
+RIVA
+~~~
+
+.. code-block:: bash
+
+    export RIVA_API_KEY=xxxx
+
+Configuration file
+------------------
+
+Translation function is configured in the ``run`` section of a configuration with the following keys:
+
+target_lang   - A single ``bcp47`` entry designating the language of the target under test. "ja", "fr", "jap" etc.
+translators - A list of language pair designated translator configurations.
+
+* Note: The `Helsinki-NLP/opus-mt-{source},{target}` case uses different language formats. The language codes used to name models are inconsistent. 
+Two-digit codes can usually be found `here<https://developers.google.com/admin-sdk/directory/v1/languages>`_, while three-digit codes require
+a search such as “language code {code}". More details can be found `here <https://github.com/Helsinki-NLP/OPUS-MT-train/tree/master/models>`_.
+
+A translator configuration is provided using the project's configurable pattern with the following required keys:
+
+* ``language``   - A ``,`` separated pair of ``bcp47`` entires describing translation format provided by the configuration
+* ``model_type`` - the module and optional instance class to be instantiated. local, remote, remote.DeeplTranslator etc.
+* ``model_name`` - (optional) the model name loaded for translation, required for ``local`` translator model_type
+
+(Optional) Model specific parameters defined by the translator model type may exist.
+
+* Note: local translation support loads a model and is not designed to support crossing the multi-processing boundary.
+
+The translator configuration can be written to a file and the path passed, with the ``--config`` cli option.
+
+An example template is provided below.
+
+.. code-block:: yaml 
+run:
+  target_lang: {target language code}
+  translators:
+    - language: {source language code},{target language code}
+      api_key: {your API key}
+      model_type: {translator module or module.classname}
+      model_name: {huggingface model name} 
+    - language: {target language code},{source language code}
+      api_key: {your API key}
+      model_type: {translator module or module.classname}
+      model_name: {huggingface model name} 
+
+* Note: each translator is configured for a single translation pair and specification is required in each direction for a run to proceed.
+
+Examples for translation configuration
+--------------------------------------
+
+DeepL
+~~~~~
+
+To use DeepL translation in garak, run the following command:
+You use the following yaml config.
+
+.. code-block:: yaml 
+run:
+  target_lang: {target language code}
+  translators:
+    - language: {source language code},{target language code}
+      model_type: remote.DeeplTranslator
+    - language: {target language code},{source language code}
+      model_type: remote.DeeplTranslator
+
+
+.. code-block:: bash
+
+    export DEEPL_API_KEY=xxxx
+    python3 -m garak --model_type nim --model_name meta/llama-3.1-8b-instruct --probes encoding --config {path to your yaml config file} 
+
+
+Riva
+~~~~
+
+For Riva, run the following command:
+You use the following yaml config.
+
+.. code-block:: yaml 
+
+run:
+  target_lang: {target language code}
+  translators:
+    - language: {source language code},{target language code}
+      model_type: remote
+    - language: {target language code},{source language code}
+      model_type: remote
+
+
+.. code-block:: bash
+
+    export RIVA_API_KEY=xxxx
+    python3 -m garak --model_type nim --model_name meta/llama-3.1-8b-instruct --probes encoding --config {path to your yaml config file} 
+
+
+Local
+~~~~~
+
+For local translation, use the following command:
+You use the following yaml config.
+
+.. code-block:: yaml 
+run:
+  target_lang: jap
+  translators:
+    - language: en,jap
+      model_type: local
+    - language: jap,en
+      model_type: local
+
+.. code-block:: bash
+
+    python3 -m garak --model_type nim --model_name meta/llama-3.1-8b-instruct --probes encoding --config {path to your yaml config file} 
+
+The default configuration will load `Helsinki-NLP MarianMT <https://huggingface.co/docs/transformers/model_doc/marian>`_ models for local translation.
+
+Additional support for Huggingface ``M2M100Model`` type only is enabled by providing ``model_name`` for local translators. The model name provided must
+contain ``m2m100`` to be loaded by garak.
+
+.. code-block:: yaml 
+run:
+  target_lang: ja
+  translators:
+    - language: en,ja
+      model_type: local
+      model_name: facebook/m2m100_418M
+    - language: jap,en
+      model_type: local
+      model_name: facebook/m2m100_418M
+
+
+.. code-block:: bash
+
+    python3 -m garak --model_type nim --model_name meta/llama-3.1-8b-instruct --probes encoding --config {path to your yaml config file} 

garak/__init__.py

@@ -1,6 +1,6 @@
 """Top-level package for garak"""
 
-__version__ = "0.10.3.post1"
+__version__ = "0.11.0.pre1"
 __app__ = "garak"
 __description__ = "LLM vulnerability scanner"
 

garak/_config.py

@@ -115,6 +115,8 @@ def _nested_dict():
 # this is so popular, let's set a default. what other defaults are worth setting? what's the policy?
 run.seed = None
 run.soft_probe_prompt_cap = 64
+run.target_lang = "en"
+run.translators = []
 
 # placeholder
 # generator, probe, detector, buff = {}, {}, {}, {}

garak/_plugins.py

@@ -373,9 +373,14 @@ def load_plugin(path, break_on_fail=True, config_root=_config) -> object:
         match len(parts):
             case 2:
                 category, module_name = parts
-                generator_mod = importlib.import_module(
-                    f"garak.{category}.{module_name}"
-                )
+                try:
+                    generator_mod = importlib.import_module(
+                        f"garak.{category}.{module_name}"
+                    )
+                except ModuleNotFoundError as e:
+                    raise ValueError(
+                        f"Unknown plugin module specification: {category}.{module_name}"
+                    ) from e
                 if generator_mod.DEFAULT_CLASS:
                     plugin_class_name = generator_mod.DEFAULT_CLASS
                 else:
@@ -391,7 +396,7 @@ def load_plugin(path, break_on_fail=True, config_root=_config) -> object:
     except ValueError as ve:
         if break_on_fail:
             raise ValueError(
-                f'Expected plugin name in format category.module_name.class_name, got "{path}"'
+                f'Expected plugin name in format category.module_name.class_name or category.module_name, got "{path}"'
             ) from ve
         else:
             return False

garak/attempt.py

@@ -38,6 +38,10 @@ class Attempt:
     :type seq: int
     :param messages: conversation turn histories; list of list of dicts have the format {"role": role, "content": text}, with actor being something like "system", "user", "assistant"
     :type messages: List(dict)
+    :param bcp47: Language code for prompt as sent to the target
+    :type bcp47: str
+    :param reverse_translator_outputs: The reverse translation of output based on the original language of the probe
+    :param reverse_translator_outputs: List(str)
 
     Expected use
     * an attempt tracks a seed prompt and responses to it
@@ -72,6 +76,8 @@ def __init__(
         detector_results=None,
         goal=None,
         seq=-1,
+        bcp47=None,  # language code for prompt as sent to the target
+        reverse_translator_outputs=None,
     ) -> None:
         self.uuid = uuid.uuid4()
         self.messages = []
@@ -86,6 +92,10 @@ def __init__(
         self.seq = seq
         if prompt is not None:
             self.prompt = prompt
+        self.bcp47 = bcp47
+        self.reverse_translator_outputs = (
+            {} if reverse_translator_outputs is None else reverse_translator_outputs
+        )
 
     def as_dict(self) -> dict:
         """Converts the attempt to a dictionary."""
@@ -103,6 +113,8 @@ def as_dict(self) -> dict:
             "notes": self.notes,
             "goal": self.goal,
             "messages": self.messages,
+            "bcp47": self.bcp47,
+            "reverse_translator_outputs": list(self.reverse_translator_outputs),
         }
 
     @property
@@ -189,6 +201,37 @@ def latest_prompts(self, value):
         assert isinstance(value, list)
         self._add_turn("user", value)
 
+    def prompt_for(self, lang) -> str:
+        """prompt for a known language
+
+        When "*" or None are passed returns the prompt passed to the model
+        """
+        if (
+            lang is not None
+            and self.bcp47 != "*"
+            and lang != "*"
+            and self.bcp47 != lang
+        ):
+            return self.notes.get(
+                "pre_translation_prompt", self.prompt
+            )  # update if found in notes
+
+        return self.prompt
+
+    def outputs_for(self, lang) -> List[str]:
+        """outputs for a known language
+
+        When "*" or None are passed returns the original model output
+        """
+        if (
+            lang is not None
+            and self.bcp47 != "*"
+            and lang != "*"
+            and self.bcp47 != lang
+        ):
+            return self.reverse_translator_outputs
+        return self.all_outputs
+
     def _expand_prompt_to_histories(self, breadth):
         """expand a prompt-only message history to many threads"""
         if len(self.messages) == 0:

garak/detectors/base.py

@@ -1,6 +1,6 @@
 """Base detectors.
 
-These detectors show expectations and operations of any detector implemented 
+These detectors show expectations and operations of any detector implemented
 in `garak`. `garak` detectors must inherit from a class in here."""
 
 import logging
@@ -135,10 +135,9 @@ def detect(self, attempt: garak.attempt.Attempt) -> List[float]:
         # goal: skip None outputs
         # don't adjust attempt.outputs
 
-        non_none_outputs = [
-            v for k, v in enumerate(attempt.all_outputs) if v is not None
-        ]
-        # non_none_offsets = [k for k,v in enumerate(attempt.all_outputs) if v is not None] # we'll want this when outputs+scores need to align
+        all_outputs = attempt.outputs_for(self.bcp47)
+        non_none_outputs = [v for k, v in enumerate(all_outputs) if v is not None]
+        # non_none_offsets = [k for k,v in enumerate(all_outputs) if v is not None] # we'll want this when outputs+scores need to align
         try:
             detector_raw_results = self.detector(
                 non_none_outputs, **self.tokenizer_kwargs
@@ -179,10 +178,12 @@ def detect(
         self, attempt: garak.attempt.Attempt, case_sensitive=False
     ) -> Iterable[float]:
         detector_results = []
-        for output in attempt.all_outputs:
-            match = False
+        all_outputs = attempt.outputs_for(self.bcp47)
+
+        for output in all_outputs:
             if output is None:
                 continue
+            match = False
             for s in self.substrings:
                 if not case_sensitive:
                     s, output = s.lower(), output.lower()
@@ -212,14 +213,15 @@ def detect(
         self, attempt: garak.attempt.Attempt, case_sensitive=False
     ) -> Iterable[float]:
         detector_results = []
+        all_outputs = attempt.outputs_for(self.bcp47)
+
         if "triggers" in attempt.notes:
             triggers = attempt.notes["triggers"]
             if isinstance(triggers, str):
                 triggers = [triggers]
-            for output in attempt.all_outputs:
+            for output in all_outputs:
                 if output is None:
                     continue  # trigger is absent
-
                 match = False
                 for trigger in triggers:
                     if case_sensitive:

garak/detectors/encoding.py

@@ -1,6 +1,6 @@
 """Encoding detectors
 
-Detectors for assessing how well a given payload from probes.encoding is 
+Detectors for assessing how well a given payload from probes.encoding is
 output by the model
 """