Integrate Unicode Inflection into Unicode Message Format

[grhoten]

This is a GSoC Project idea.

Difficulty/Size: Higher

The goal is to create a way for the MessageFormatter in ICU4C to leverage the Unicode Inflection project for creating grammatically correct sentences. These are currently 2 independent projects, and Unicode Inflection is only available with C and C++ API at this time. Unicode Inflection supports all of the use cases in the expected outcomes, but MessageFormatter has no way to use such functionality at this time.

Below is background material for the Unicode Inflection concepts with 3 different syntaxes that are not the MessageFormat 2 syntax. Unicode Inflection supports all of the use cases in the expected outcomes.

• UTW 2023 Automatic Grammar Agreement in Message Formatting
• S12T1 Authoring Grammatically Correct Conversational Templates for Siri
• Let's Come To An Agreement About Our Words :: IMUG 2017.02.16

Here is some additional background information between these projects.

• UTW 2024 Solving Inflection
• UTW 2024 MessageFormat 2 Technical Preview: Where Are We Now?

Expected Outcomes

1. A minimum viable integration can generate these messages depending on the grammatical properties of the object being inserted into the sentence.
   1. English
      1. The {object} is on
      2. The {object} are on
   2. French
      1. La {object} est allumée
      2. Les {object} sont allumées
      3. L'{object} est allumé
      4. Le {object} est allumé
      5. Les {object} sont allumés
2. Bonus integrations include the following topics
   1. Support quantities. E.g. 1 foot/2 feet
   2. Support lists. E.g. An object, and a table
   3. Support pronouns, especially in Spanish and Arabic. E.g. Here is {pronoun} location.
   4. Support spoken text. E.g. one foot/two feet.
   5. Support inflecting words. E.g. mouse + plural → mice
   6. Support SemanticConcept to allow custom inflections.
   7. Support inflection alternative when the word being inserted into the sentence is not in the lexical dictionary. E.g. 😀 isn’t in the lexical dictionary. So you may want to use “L(a|e) 😀 est allumé(e)” in French.

Skills

• Required: C/C++
• Required: Ability to use cmake for building the code
• Preferred: Love of languages.

[nilaygit-10721]

I am ready to work on this; please confirm that I can proceed.

[grhoten]

Do you want to take this task as a part of the GSoC? If so, please make sure that you register and apply for this project through the GSoC website.

[nilaygit-10721]

@grhoten can you guide me for register and apply?

[grhoten]

I believe the application process opens on March 24 according to the GSoC timeline.

[evelez7]

Hi! My understanding is MessageFormat 2.0 makes it easier for localized messages to be delivered. Thus in an English example a translator would have to provide the following two strings from the provided examples:

1. The {object} is on.
2. The {object} are on.

If Inflection support was included, would a translator only have to provide The {$object | inflect} on. and let Inflection handle the inference based on $object quantity? Is the goal that Inflection should be a standard function for MessageFormat 2? Or am I misunderstanding how Inflection works?

Is there a standard API for Inflection yet? From #23 it seems like there isn't, so I'm assuming creating one would be involved here, at least as a prototype for MessageFormat?

[grhoten]

Hi! My understanding is MessageFormat 2.0 makes it easier for localized messages to be delivered. Thus in an English example a translator would have to provide the following two strings from the provided examples:

Lots of implementations tend to just provide 1 string, and they leave it grammatically broken or incomplete. The goal is to not be broken. MF 2 doesn't have the ability to detect if a word is going to be singular or plural, nor masculine or feminine. It doesn't provide a way to modify the case of the object or to make the referenced object definite.

1. The {object} is on.
2. The {object} are on.

If Inflection support was included, would a translator only have to provide The {$object | inflect} on. and let Inflection handle the inference based on $object quantity? Is the goal that Inflection should be a standard function for MessageFormat 2? Or am I misunderstanding how Inflection works?

You're close. The is/are would need to be included in some way to inflect. Here are 2 ways to integrate such functionality.

1. Provide functionality like the second and third videos above. In that case, the translator explicitly provides the dependencies and constraints. I think MF 2 is geared towards this scenario a little better, but I'm unsure.
2. The framework tries to guess the dependencies and specifies the constraints to Unicode Inflection. That's the first video. That's nicer for the translator, and it's close to what you were suggesting. That's harder for the implementor. Even if implementing this option, you still need the first option for edge cases.

Is there a standard API for Inflection yet? From #23 it seems like there isn't, so I'm assuming creating one would be involved here, at least as a prototype for MessageFormat?

The code already works, and the API likely won't change much further before 1.0. There are already a lot of tests. The current work is transitioning from stub test data to data derived from Wikidata.

The issue that you reference was submitted before getting confirmation of the large initial code contribution. It's also talking about integration with ICU, which is a different topic. I'd ignore that issue for now, since it doesn't involve MessageFormat 2.0.

[grhoten]

Some people have asked what is needed to do this. Here are some files that highlight how to use Unicode Inflection.

I recommend taking a look at these test files on how this stuff works.
https://github.com/unicode-org/inflection/tree/main/inflection/test/resources/inflection/dialog/inflection
https://github.com/unicode-org/inflection/tree/main/inflection/test/resources/inflection/dialog/pronoun
https://github.com/unicode-org/inflection/blob/main/inflection/test/src/inflection/dialog/InflectionTest.cpp
https://github.com/unicode-org/inflection/blob/main/inflection/test/src/inflection/dialog/PronounConceptTest.cpp
https://github.com/unicode-org/inflection/blob/main/inflection/test/src/inflection/grammar/synthesis/QuantifyTest.cpp

The InflectionTest.cpp provides a simple example, and half of that code is devoted to XML parsing, which you shouldn't need to do with MF2 hopefully.

If you need another example, take a look at FormatTest.cpp and the files at inflection/dialog/format. In the C++ code, most of that is also devoted to XML parsing, but it provides an example way to map such syntax.

[AbigaleD]

Hi @grhoten I am Abigale I’ve reviewed the files you mentioned, the test files in the dialog/inflection folder and the instructional YouTube videos. It’s clear that integrating Unicode Inflection into MessageFormat 2 will require a dynamic mechanism for grammatical transformations.

From my understanding, integrating Inflection into MF2 involves creating a system where the MessageFormatter handles inflections based on object attributes. One approach I’m considering is extending the template format to support a custom inflect function for processing properties at runtime. I haven’t reviewed all the code yet, as the project seems large, so any guidance on how these functions will be integrated would be helpful. Should I prioritize handling edge cases (e.g., emojis or slang) or ensuring compatibility with the current MessageFormat syntax?

Additionally, I’m preparing my GSoC application. I have experience with C/C++, Makefile, and CMake.I speak English and Chinese(Native) and I’m learning German and Dutch as I plan to further my studies in the Netherlands. These make me feel kinda fit for the unicode projects.

I’m working on my proposal and CV for GSoC. Is there anything specific the mentors like to see, or anything else I can do for now?

[grhoten]

@AbigaleD I would say that MF2 has the syntax, and Unicode Inflection is the inflection engine. I believe that MF2 has a way to register formatting functions, but I haven't investigated those details. If you look at https://messageformat.unicode.org/, you will see that it says the following:

MessageFormat 2 allows developers to define custom functions to handle complex formatting and matcher needs.

That is precisely what this code needs to use to integrate into MF2. Handling things like slang are probably best handled with item 2.vi.

MF2 is probably the fourth syntax being proposed to be put on top of this type of code. 3 of them was prior to Unicode Inflection becoming open source.

Thanks for your interest, and I eagerly await your proposal.

[BHK4321]

Hi @grhoten,
I hope you're doing well. I’ve submitted my GSoC proposal for this idea and wanted to thank you for your earlier guidance. I’ve also made a few contributions recently #105 and really enjoyed working with the community so far.
I'm excited about the possibility of contributing more deeply to this project and would really appreciate any feedback you might have on the proposal when you get a chance.
Looking forward to hearing from you!

Best regards,
Bhaskar