[meta] `HXLm.lisp` and/or related strategies for portable 'Turing complete' HDP custom functions

[fititnt]

Related topics:

• [meta] HDP Declarative Programming (working draft) [meta] HDP Declarative Programming (working draft) #16
  • The HDP YAML (+JSON) syntax have potential to, at minimum, be an way to humans document (without leak access keys and passwords, yet with minimum viability to allow be parsed by programs) how to access datasets
    • The urnresolver: Uniform Resource Names - URN Resolver urnresolver: Uniform Resource Names - URN Resolver #13 could aid the part of how to point to the resources themselves
    • The way to express Acceptable Use Policies (ones that could be enforced by tools) still not defined, because this need be good enough to allow localization in natural languages supported by the drafted HDP
• hxl-yml-spec-to-hxl-json-spec: HXL Data processing specs exporter hxl-yml-spec-to-hxl-json-spec: HXL Data processing specs exporter #14
  • The HXL Data processing specs is known to work in production for years. The official wiki says the underlining usage is focused for coders, but the HXL-Proxy already provides a front-end for end users.
    • At bare minimum any syntax should be able to also be transpiled to an to JSON processing specs for HXL data so it could run on existing HXL-proxy (and command line tools) with the same stability of tools already working in the past.
      • Note: Lisp-like syntax is even less user-friendly than YAML syntactic sugar to JSON processing specs for HXL data, so this approach is not expected to replace the possibility of users doing without Lisp-like syntax.
  • Note: as this comment hxl-yml-spec-to-hxl-json-spec: HXL Data processing specs exporter #14 (comment) an early version, not integrated on HXLm.HDP, but accessible via hdpcli, already transpile YAML structure to HXL Data processing specs, supported by libhxl cli tools and the HXL-proxy. The notable exception is the inline to use as input (so instead of giving an external URL, the processing spec would already have the input data) and the inline output (that could be used to test if the filter was validated).
    • But if we manage to let users, before pass data to libhxl-python cli tool or the HXL-proxy to save the input, like on a local file or implement the HXLm.core.io to allow even write google spreadsheets, as hard as this extra step may be, it would allow validation of entire HXL-like systems, like if it was with more close to real data and meet requirements of an 'design-by-contract programming' testing the full chain of components, not just the internals of the HXL data processing implementation
• [meta] Internationalization and localization (i18n, l10n) and internal working vocabulary [meta] Internationalization and localization (i18n, l10n) and internal working vocabulary #15
  • An proof of concept the HDP already support 6 UN working languages plus Latin and Portuguese.
    • The early prototypes actually were faster to make than core refactoring to an more OOP approach with strong focus on automated testing for internals. But the point here is that the HDP yaml files already support several natural languages and to add new ones , do not need to know internals, just change some files on hxlm/ontologia
  • While the same supported languages do not need to be both for the high level HDP yaml keywords (ideally, since is less terms, that should be done first!) the pertinence here is, whatever would be an language syntax for create HDP macro/extensions/plugins, this must be done already optimizing to to provide an Source-to-source compiler
    • In other words: even if we decide to implement keywords using English/Latin, as soon as do exist help to add new natural languages to knowledge graph, this should allow source to source compiling. This means that the syntax should be "L10N friendly" from start

Related concepts:

• Concept: Spreadsheets functions (both the part that is viable localize the key terms and also the need to be able to have some sort of Turing completeness to make them, even if direct access to file system or network we do not allow)
  • Google Sheets, several languages options: https://support.google.com/docs/answer/58515#zippy=%2Cchange-the-language-for-functions
  • Microsoft Excel, several language options: https://support.microsoft.com/en-us/office/supported-languages-79be5ebe-9130-455e-b019-e3ce99367bb5
• Concept: design-by-contract programming / Design by contract (DbC) https://en.wikipedia.org/wiki/Design_by_contract
• Concept: International programming languages
  • https://en.wikipedia.org/wiki/Non-English-based_programming_languages#International_programming_languages

---

About this topic

This topic is a draft. It will be referenced on specific commits and other discussions.

But to say upfront that as much as possible, the idea here is keep as much as possible documents that could be used by decision makers to authorize usage and/or for people who document that datasets do exist (even if they do not say on the document how to find them) and, for what is not feasible already have via the underlining python implementation, allow customization.

Note that these customization, while not explicitly sandboxes (but could be) do not need to be allowed to have direct disk or network access. This approach is not just more safe, also open room for they be more reusable and (this is very important!) simplify documentation on how to use, even by individuals who would not speak the same language.

[fititnt]

I believe that, at least at this moment for the JavaScript part of HDP, we use prefix HDPb instead of HDP

Maybe this never get implemented, but Lisp dialects have the tradition to be the easiest ones to allow Self-hosting (compilers). I'm not saying that the entire HDP implementaton in python should be written in an lisp dialect that is not even formalized, but at least the javascript part could have this b as a hint for Bootstrapping (compilers)