Feature: multilingual machine translation

[SnowMasaya]

This revision enables translation support for a streamlined the use case. Some items such
as support for multiple target languages in a single run have been omitted in favor of a
more straight forward user experience and reduced ambiguity in the scope of results.

Translator classes have been implemented using the Configurable pattern and the plugin
loader. This introduced a new paradigm of providing configuration for a list of instances
with specific configuration required at runtime where previous Configurable class
configuration has been for all instances of a specific class or module. The processing and
attribute names used to create this instance list may evolve further.

Usage

Translation function is configured in the run section of a configuration see the doc
page in the PR for details.

New default configuration values for run.target_lang and run.translators are in the
updated documentation and allow for backwards compatible configuration with existing runs.

There are still some existing TODO: comments and notes about location that may need
further testing before landing this upstream. Most noteworthy are comments still in the
code of the atkgen probe that require further scrutiny to validate the attack technique
is applied correctly.

It may be appropriate to gate this functionality as experimental for initial release, this
would required some additional guard code to ensure limited impact to report formats and
internal state.

Example

python -m garak -m huggingface.Model --config hf_RigoChat_gpu.yml -p lmrc --report_prefix RigoChat-21fde039

hf_RigoChat_gpu.yml:

run:
  target_lang: "es"
  translators:
    - language: es,en
      model_type: local
      model_name: facebook/m2m100_418M
      hf_args:
        device: cuda
    - language: en,es
      model_type: local
      model_name: facebook/m2m100_418M
      hf_args:
        device: cuda
plugins:
  generators:
    huggingface:
      Model:
        name: IIC/RigoChat-7b-v2
        hf_args:
          device: cuda
          trust_remote_code: true
          torch_dtype: float16

[github-actions]

DCO Assistant Lite bot All contributors have signed the DCO ✍️ ✅

[SnowMasaya]

I have read the DCO Document and I hereby sign the DCO

[jmartin-tech]

Partial review, testing is still in progress.

The test failures in macOS look to be an incomplete dependency requirement that may need to be reworked or removed. A default installation should not require install of an external library. Hence the dependency on pyenchant here may be problematic.

Traceback:
/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/importlib/__init__.py:90: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/detectors/test_detectors_riskywords.py:8: in <module>
    import garak.detectors.base
garak/detectors/__init__.py:1: in <module>
    from .base import *
garak/detectors/base.py:17: in <module>
    from garak.translator import SimpleTranslator, LocalTranslator, is_english
garak/translator.py:15: in <module>
    import enchant
/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/enchant/__init__.py:81: in <module>
    from enchant import _enchant as _e
/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/enchant/_enchant.py:[157](https://github.com/leondz/garak/actions/runs/11246708414/job/31296143918?pr=943#step:5:158): in <module>
    raise ImportError(msg)
E   ImportError: The 'enchant' C library was not found and maybe needs to be installed.
E   See  https://pyenchant.github.io/pyenchant/install.html
E   for details

[leondz]

Thank you, this is a significant amount of work.

This is for translation-based multilingual, rather than general multilingual support, so we should be careful to separate these two functions. The ability to select which languages garak uses is distinct from how that language is achieved (in prompts, detectors, and so on).

There are some refactorings needed to get the PR to a sustainable state. We may need a couple more hooks, either injected into base classes or added there. It might be simplest to add these hooks to the harness. Happy to set up a call or instant message chat to discuss this. Given the breadth of the changes, it is likely to be beneficial to discuss plans and get good alignment while doing the rest of the changes.

[jmartin-tech]

@SnowMasaya thank you, this is a significant benefit for the project.

Adding a number of my pending comments here, please be aware these are likely similar to many @leondz made, although possibly from different reasoning. Happy to continue iterating.

[leondz]

Adding a number of my pending comments here, please be aware these are likely similar to many @leondz made, although possibly from different reasoning. Happy to continue iterating.

Indeed, sorry for the dupe review, there was quite a lot of code so I reviewed directly instead of processing @jmartin-tech 's comments as well

[leondz]

We're converging. Skipping over comments on some of the tests, for now, until the core patterns are completely settled.

[leondz]

That's one example of the confusion I guess. Propose then:

Suggested change

	if attempt.bcp47 != "*" and attempt.bcp47 != self.bcp47:
	if attempt.prompt_bcp47 != "*" and attempt.prompt_bcp47 != self.bcp47:

[leondz]

OK, that's fair. If users want to override detector selection that's up to them.

the idea here is that the probe is responsible for properly handling reverse translation requirements and detectors will utilize the Attempt they are presented.

Thanks. In this case, it's not right that detector code is referencing reverse_translator_outputs - the probe is responsible for properly handling requirements. Or is that responsibility actually split between probe and detector?

[leondz]

Yes, I like this. Then the test (in many places) would be if _use_translation_entries(attempt):. Where would this function sit? Maybe in garak.attempt ?

[leondz]

lgtm

Todos for after landing:

• move prompt & response language descriptions to Turns (Migrate string output/input to Turn objects #1089)
• add clarifying language to instances of "target" variable names
• rename translators
• factor condition determining if translation is happening into just one place (Feature: multilingual machine translation #943 (comment))
• rename bcp47 to lang & validate (Rename bcp47 to lang  #1139)
• add step-by-step flow description into docs/source/langservice.rst
• choose new language for "native" (native to what is sometimes unclear)
• word out what to do when nltk corpora/words isn't present
• evaluate MT-based performance
• add GCP-based language provider
• clarify Feature: multilingual machine translation #943 (comment)

[leondz]

looks great to me. rocket emoji etc etc. this is super cool

[leondz]

trivia point: the Marian author was one of the participants in the demon paper!

[leondz]

I'm disdainful of not making English English default English. I guess empire instincts die hard.