[meta] Internationalization and localization (`i18n`, `l10n`) and internal working vocabulary

[fititnt]

Quick links:

• https://en.wikipedia.org/wiki/Internationalization_and_localization
• https://www.w3.org/International/questions/qa-i18n

---

This issue may be used to make references to the internal working vocabulary and how to deal with Internationalization and localization in special for the [meta issue] hxlm #11.

A lot of work already was done, but in addition to be used internally, for tools like the https://json-schema.org/ that can be used to generate helpers for who uses code editors like VSCode when editing YAML by hand, for allow multiple languages (even for the key, not just he content) eventually we may need to generate the JSON schemas (there is no native way to make JSON Schemas multilanguage).

TODO: add more context

[fititnt]

Maybe will be possible to conditionally load JSON Schemas (and, with JSON Schemas, means auto complete) based not just on file extension (think like ola-mundo.por.hdp.yml vs hello-world.eng.hdp.yml but something like salve-mundi.mul.hdp.yml salve-mundi.hdp.yml.

• https://json-schema.org/understanding-json-schema/reference/conditionals.html
• https://json-schema.org/understanding-json-schema/reference/combining.html
• https://json-schema.org/understanding-json-schema/structuring.html#structuring

---

Edit: from salve-mundi.mul.hdp.yml to salve-mundi.hdp.yml

[fititnt]

Almost there...

hdpcli tests/hrecipe/hello-world.hrecipe.hdp.yml --objectivum-linguam RUS

[fititnt]

We will need to use recursion.

And this need to not try to translate even the inline example data (or, at least for now, the country/territory ISO 2 code names). But I think this still not as bad as the need to be well done, in special when parsing a unknown language to avoid some sort of recursive DDoS.

[fititnt]

We're already able to export the internal representation (heavily based on Latin) on the 6 UN languages plus Portuguese!!!

(It still not checking for input syntax beyond what JSON Schema warn the user, but ok, it's something!)

1. Make any know vocabulary equally valid

The things really shine if any of the 7 languages are able to be equally valid as a full working project. That's the idea. This feature alone make it an huge appeal to use.

Note: the core_vocab, while always will try to export to an unique ID per language, tolerates aliases.

1.1 aliases are good... but the idea is not overuse for macro languages

In other words: core_vocab (plus user ad-hoc customization for unknown languages) tolerate some variation on input. But still an good idea at some point don't force entire macro languages (like Arabic and Chinese) on same ISO 639-3 ISO codes (ok that this could be an hot fix, but is not an ideal)

If necessary, I think we can implement some way to just override part of a vocabulary. So for example if his 20% of an individual language share acceptable conventions with the macro language, we make the HDP itself allow this

2. What would be the "official" version of a file?

Even if, in practice, most teams that already use English as working language would use thing.eng.hdp.yml I like the idea that for resources that are created from someone else still with the very exact file and the HDP tools could still tolerate on the fly more than one file on disk.

This may not be as relevant when everyone speaks the same language, but at least can work as benchmark for when it is with HDP files from others.

2.1 What about if two files on disk are out of sync (like someone edit a version)

I think either by default (should be allow to enable/disable with configuration) or by extra command line, some way to detect if two resources in different languages would deliver different results

2.2 What if two same resources NEED to be different? (Like an file from someone else had an error; or need an update before pass for next person)

Again, I think that this case may need to implicitly allow some way to know that two resources are almost the same... But changes are allowed (maybe just allow override a few parameters).

At a more basic level, I think that just having a small name chance, (like thing-v2.eng.hdp.yml) could do the trick. This may sound lazy, but would be sufficient to not raise errors.

But the idea would be something that (not this week, maybe not this month because I need to do other stuff out of this project) eventually allow digitally signing an HDP file.

And the process of digitally signing, when necessary, needs to me allow humans who could do this a lot of time, but without actually automate too much: do exist a reason why smart cards like Yubikeys have an physical button, and is scary.

---

Edit: added example

fititnt@bravo:/workspace/git/EticaAI/HXL-Data-Science-file-formats$ hdpcli tests/hrecipe/salve-mundi.hrecipe.mul.hdp.yml --objectivum-linguam RUS

urn:hdp:OO:HS:local:salve-mundi.hrecipe.mul.hdp.yml:
  силосная:
    группа:
      - salve-mundi
    описание:
      ENG: Hello World!
      POR: Olá Mundo!
    страна:
      - AO
      - BR
      - CV
      - GQ
      - GW
      - MO
      - MZ
      - PT
      - ST
      - TL
    тег:
      - CPLP
    язык: MUL
  трансформация-данных:
    - _recipe:
        - aggregators:
            - sum(population) as Population#population
          filter: count
          patterns: adm1+name,adm1+code
        - filter: clean_data
          number: population
          number_format: .0f
      идентификатор: example-processing-with-a-JSON-spec
      пример:
        - источник:
            _sheet_index: 1
            iri: https://data.humdata.org/dataset/yemen-humanitarian-needs-overview
        - источник:
            данные:
              - - header 1
                - header 2
                - header 3
              - - '#item +id'
                - '#item +name'
                - '#item +value'
              - - ACME1
                - ACME Inc.
                - '123'
              - - XPTO1
                - XPTO org
                - '456'
          цель:
            данные:
              - - header 1
                - header 2
                - header 3
              - - '#item +id'
                - '#item +name'
                - '#item +value'
              - - ACME1
                - ACME Inc.
                - '123'
              - - XPTO1
                - XPTO org
                - '456'