Make it possible to distinguish hard and soft hyphens

[urieli]

Currently there is no way of distinguishing hard and soft HYP elements.

Example of a hard hyphen:

I separated the words by a non-
breaking space.

Example of a soft hyphen:

I separated the words by a non-break-
ing space.

However, since the OCR system can often distinguish the two (e.g. by checking a lexicon of known words), it should be able to pass this information to downstream systems in the Alto file, since this information could affect OCR-to-text and OCR layer indexing strategies.

I suggest changing the HYP element to include a new HARD_HYPHEN attribute, as follows:

<xsd:element name="HYP" minOccurs="0">
  <xsd:annotation>
    <xsd:documentation>A hyphenation char. Can appear only at the end of a line.</xsd:documentation>
  </xsd:annotation>
  <xsd:complexType>
    <xsd:attribute name="HEIGHT" type="xsd:float" use="optional"/>
    <xsd:attribute name="WIDTH" type="xsd:float" use="optional"/>
    <xsd:attribute name="HPOS" type="xsd:float" use="optional"/>
    <xsd:attribute name="VPOS" type="xsd:float" use="optional"/>
    <xsd:attribute name="CONTENT" type="xsd:string" use="required"/>
    <xsd:attribute name="HARD_HYPHEN" type="xsd:boolean" use="optional">
      <xsd:annotation>
        <xsd:documentation>True if this is a hard-hyphen (would appear in the word regardless of print location), false if this is a soft hyphen (only appears in the word if it is split at the end of a line).</xsd:documentation>
      </xsd:annotation>
    </xsd:attribute>
  </xsd:complexType>
</xsd:element>

[cipriandinu]

Thank you for your proposal, we will discuss it and take it into account for the next release (5.0)

[cipriandinu]

Maybe this should be discussed in a larger context, see #43

[cipriandinu]

follow up on this topic: on the last boad meeting we have discussed three options, as possible solutions (see also https://github.com/altoxml/board/blob/gh-pages/minutes/2024/2024-10-17%20ALTO%20Board%20Meeting%20Minutes.md):

1. Add an attribute as initially proposed (HARD_HYPHEN)
2. Add an attribute like HYPHEN_TYPE with a set of possible values to cover different variations (for example hyphens described in Allow variants for hyphen elements (HYP) #43 )
3. No changes, but get the info from CONTENT attribute (where proper unicode character should be used), then a processor could make a distinction based on character code; Another alternative to distinguish between HYP types would be by SUBS_CONTENT attribute of previous/next String that should remove the hyphen only in case this is soft hyphen.

@urieli - additional feedback from your side would help for a final decision on our next meeting

[urieli]

Regarding the suggestion to use HYPHEN_TYPE, I'm not sure what the possible values would be. If these include alternative hyphen glyphs mentioned in issue #43, I don't feel it has anything to do with the current issue.
Any glyph representing a hyphen could potentially be either soft or hard.
In a historical Yiddish corpus I annotated recently, three different glyphs were used to represent hyphens: the standard minus character U+002D, the Hebrew alphabet maqaf U+05BE, and the double oblique hyphen U+2E17.
These need to go in the CONTENT attribute.
Any one of them could be soft or hard.

More generally, the question of hyphen variants discussed in #43 doesn't seem to have anything to do with whether the hyphen is hard or soft.

Regarding the suggestion of using the CONTENT attribute:
This is clearly wrong, as the CONTENT should give the glyph, whereas the Unicode U+00AD soft hyphen is "not a normal printable character but essentially an invisible hyphenation hint" (see discussion on https://jkorpela.fi/shy.html ).
Not only would this be a misuse of the soft hyphen character, but it would also prevent us from indicating that other glyphs, such as the double-oblique hyphen mentioned in Issue #43, are actually soft.

Regarding the suggestion to use SUBS_CONTENT: this is the workaround I currently use. However, it is not satisfactory, because it involves fairly complex logic: concatenating the CONTENT for the last word on the current line and the first word on the next line, and comparing them with SUBS_CONTENT (which could still contain a hard hyphen elsewhere in the word).
I currently ask people designing downstream systems to use this logic, but given its complexity, I'm never sure they do it right - it's a source of potential errors.

So, I still feel the best option is to add a HARD_HYPHEN attribute.
Or alternatively a HYPHEN_TYPE attribute, but with only two values: "Hard" and "Soft".