Missing conditional mandatory statements for BT-773-Tender (and more)

[JulianMa]

Hello,

i want to discuss a class of missing conditional mandatory statements, which right now requires alot of xsd parsing to workaround and make our meta-driven eForms approach work.

The Field BT-773-Tender is not conditional mandatory right now (1.11.0-rc.3, also previous versions),
but the XSD type efbc:TermCode is:

<xsd:complexType name="SubcontractingTermType">
		<xsd:sequence>
			<xsd:element ref="efac:FieldsPrivacy" minOccurs="0" maxOccurs="unbounded"/>
			<xsd:element ref="efbc:TermAmount" minOccurs="0" maxOccurs="1"/>
			<xsd:element ref="efbc:TermDescription" minOccurs="0" maxOccurs="unbounded"/>
			<xsd:element ref="efbc:TermPercent" minOccurs="0" maxOccurs="1"/>
			<xsd:element ref="efbc:TermCode" minOccurs="1" maxOccurs="1"/>
			<xsd:element ref="efbc:PercentageKnownIndicator" minOccurs="0" maxOccurs="1"/>
			<xsd:element ref="efbc:ValueKnownIndicator" minOccurs="0" maxOccurs="1"/>
		</xsd:sequence>
	</xsd:complexType>

The Group inside the Visual Model is found for example in subtype 29, in Group GR-LotTender-Subcontracting.
Right now, every Field inside that Group is Optional. So when a user is not entering any data, the XML Node will not be generated, because it has no value. This is fine, because the SubcontractingTermType is only used one time with optional cardinallity:

<xsd:element ref="efac:SubcontractingTerm" minOccurs="0" maxOccurs="unbounded"/>

But as soons as there is data entered in one of the optional fields, the XML will be invalid, because a node of the SubcontractingTermType will be created, with no value for efbc:TermCode:

<efac:SubcontractingTerm>
    <efbc:PercentageKnownIndicator>true</efbc:PercentageKnownIndicator>
    <efbc:ValueKnownIndicator>true</efbc:ValueKnownIndicator>
</efac:SubcontractingTerm>

This is obviously not valid, because it is not satisfying the xsd from above, but it is valid from the point of view from the metadata.

<xsd:element ref="efbc:TermCode" minOccurs="1" maxOccurs="1"/>

Just making BT-773-Tender always Mandatory is also not a great option, because you don't have to specify it, when there is no SubcontractingTermType at all.

So either we need another property in the fields.json, which marks a fields as required-when-parent-exists, or we need alot of EFX to express this mandatory conditionaly, when any neighboring nodes exist.

We are using the second way right now, which is really complicated, prone to errors and not readable in the fields.json:

• Parse all xsd types and find mandatory fields (minOccurs="1")
• Filter by fields which have no mandatory inside their field.json
• Find all other nodes, which can appear inside the same parrent as the found node.
• emit EFX, which makes the field mandatory, when any of the other children are set.

which in the end looks like this:

"mandatory" : {
      "value" : false,
      "constraints" : [ {
        "value" : "{ND-SubcontractedActivity} ${(BT-553-Tender is present) or (BT-554-Tender is present) or (BT-555-Tender is present) or (BT-730-Tender is present) or (BT-731-Tender is present) or ( ... alot more ...)}// CUSTOM RULE ADDED IN PREPROCESSING",
        "severity" : "ERROR",
        "valueRef" : "fn_508",
        "valueContext" : "ND-SubcontractedActivity"
      } ]

But this can't be the best solution for this Problem. This problem exists for alot of Fields (over 200 in eForms Version 1.7.3).

[YvesJo]

Hi,
Part of the Business Logic is expressed in the form of conditional rules which have been removed with SDK 1.3 and progressively reintroduced starting with SDK 1.8.
For performance and usability reasons, rules may not be expressed to consider all possible orders the UI fields are filled in. The way rules are expressed is from the more generic to the more specific information and follows a business logic.
In the case of the Subcontracted Activity, the initial question to answer is whether the contract will be subcontracted or not.
When no subcontracting takes place, it is not possible to specify further subcontracting information.
When subcontracting takes place, then further information about subcontracting may be specified.
BT-773-Tender is the "entry point" for the Subcontracting group of information.
Except for some forms (29 to 31), the Business Group Subcontracting BG-180 is optional and it is therefore not possible to require the presence of BT-773-Tender as this would make this BG mandatory.
Having an XML element mandatory may not be the best option and when it is about the extensions, we have some room and may change this. Outside the extensions, and therefore in the UBL standard, changing an element cardinality to optional is not possible.
When the XML instance reaches the CVS and is missing a mandatory XML element, the first error is about XSD non-conformance (lexical validation) even before schematron rules are applied. Not all cases may be fixed by updating the XDS and preventive options on the UI side have to be considered.
There are various options that may help the user fill the form properly:

• highlight fields that must be filled in first,
• have a dynamic UI which hides/displays the fields based on the Stage 4 forbidden/mandatory rules

If there is a need of additional information in the field.json, please share your thought about this.
KR

[JulianMa]

Hi,
thank you for your response! I think you missunderstood me here. Let me try to explain the exact Issue and where i'm coming from.
First of all, we are very familliar with the SDK and the order of filling fields does not matter for us at all.

Our Form-UI is fully dynamic and reactive to user input. We are transpiling EFX to JavaScript for our frontend.
Dynamic forbidden and mandatory fields will have their properties updated, as soon as a dependency has changed. You might know our application from a prior meeting.
Given the SDK is containing the same information as xsd + schematron, our frontend will always produce a fully valid eForms Notice. But as soon as informations are missing, we are running into problems, which will result in either xsd or schematron violations.

Lets take the Subcontracting Group again: 25.json -> GR-LotTender-Subcontracting
When ignoring the unpublished groups/fields, there are 6 fields inside that group:
BT-553-Tender, BT-554-Tender, BT-555-Tender, BT-773-Tender, BT-730-Tender, BT-731-Tender

First of all, the group is optional in subtype 25. But lets assume the user wants to specify a subcontract.

We can render the group and display all of the given BT's. After property evaluation and efx execution, all of those fields are optional.
So lets asume a user enters a value inside BT-553-Tender, but leaves BT-773-Tender empty. This is fully valid, when just using data from the fields.json. No constraints are violated and no forbidden/mandatory rules have changed.
So in theory, this should produce a valid notice.

After converting our concept-model to the actual notice xml, the Subcontracting group will look like this:

<efac:SubcontractingTerm>
    <efbc:TermAmount currencyID="EUR">12.99</efbc:TermAmount>
</efac:SubcontractingTerm>

Which is not valid, because BT-773-Tender (<efbc:TermCode>) is missing, but mandated by xsd.

So the final question is: How can we tell the user that BT-773-Tender is required in this situation, when there is no available metadata to extract that information?

EFX does a really good job in modelling all business rules and this is working great with our approach. But we often run into xsd problems, because they are not fully reflected in the metadata.

BT-773-Tender is the "entry point" for the Subcontracting group of information.

Yes! Exactly! It is the Entry Point! But this Information is not available anywhere (xsd of course, not in metadata), but crucial for generating the notice and guiding the users inside the UI. Getting this tiny piece of information requires alot of processing on our end and for sure on everyones end, who wants to implement a dynamic form editor.

This is everything i'm asking for here: A flag which is called entryPointForGroup or similar, which specifies that this fields needs to be filled, when the user whishes to use the group.

[YvesJo]

Hi,
Thanks for this clarification.
Be aware that conditional rules were deactivated with SDK 1.3 and progressively reintroduced starting with SDK 1.8.
The rules for the Subcontracting fields are reactivated with SDK 1.11 and, in their absence, the fields appear as optional.
This may be the information you are expecting.
Drawing attention of the user to some specific fields on the UI would reduce the probability of error facing.