s1kd-brexcheck ERROR: Invalid object path in BREX

[MihailCosmin]

Hi,

I have 2 huge BREX files, more than 8000 lines of code combined.
And the s1kd-brexcheck fails with ERROR: Invalid object path in BREX
Step by step I have commented the rules that were responsible for these errors. And found that there were only 4 rules that were causing such error.

Is there any option to run s1kd-brexcheck and ignore such errors?
More exactly, just skip the BREX rules that are "broken" (although they are not really broken, maybe xmlXPathEvalExpression is not 100% correct).

The rules that do not work:

<structureObjectRule id="RULE-153"> -->
	<objectPath allowedObjectFlag="1">/dmodule/pdf:style</objectPath>
	<objectUse>Custom PDF styling rules. Not S1000D compliant.</objectUse>
</structureObjectRule>

<structureObjectRule id="RULE-321">
      <objectPath allowedObjectFlag="0">(//dmStatus/applic/displayText/simplePara[lower-case(.)[contains(.,'FFF')]])</objectPath>
</structureObjectRule>

<structureObjectRule id="RULE-555">
   <objectPath allowedObjectFlag="1">//dmCode[attribute::infoCode != "003" and matches(@assyCode, '.{2}')]</objectPath>
</structureObjectRule>

<structureObjectRule id="RULE-677">
    <objectPath allowedObjectFlag="1">/dmodule/identAndStatusSection/pmStatus/reasonForUpdate[not(@id= /dmodule/content//@reasonForUpdateRefIds/tokenize(normalize-space(), '\t'))]</objectPath>
</structureObjectRule>

[kibook]

Part of the issue is that libxml2 only supports XPath 1.0, so these functions from XPath 2.0 won't work:

• lower-case
• matches
• tokenize

I believe the issue with the first example is the use of a namespace. I think I need to register all namespaces declared in ancestor nodes to the xmlXPathContext used to resolve the XPath expression. Does your BREX DM have a declaration like this somewhere?

xmlns:pdf="http://someurl.com"

As long as it's on the <structureObjectRule> or any ancestor node (like the root <dmodule> node), that should be valid XPath 1.0.

There's no option currently to skip invalid rules, but I will definitely change it so that invalid rules are skipped (with a WARNING) instead of causing the whole check to be aborted. Thanks for the suggestion, @MihailCosmin !

[kibook]

Rules with "invalid" XPath will be skipped as of 37583f2

Now I just need to fix namespaces so that XPath using them aren't also considered invalid.

[kibook]

44ffdf5 should fix the use of XPath prefixes, as long as they have been properly declared in the BREX XML.

I mentioned declaring them on the <structureObjectRule> element before, but you can also declare them on the <objectPath> itself, or any ancestor node:

<structureObjectRule id="RULE-153">
    <objectPath xmlns:pdf="http://example.com" allowedObjectFlag="1">/dmodule/pdf:style</objectPath>
    <objectUse>Custom PDF styling rules. Not S1000D compliant.</objectUse>
</structureObjectRule>

Thanks again for pointing these out, the XPath namespace prefixes in particular is something I hadn't really considered.

[kibook]

Since S1000D (Issue 4.0+) does recommend XPath 2.0 in the specification, I'm looking in to extending libxml2's XPath support.

I'm not sure it can be made fully compliant to XPath 2.0 without modifying libxml2 itself, since XPath 2.0 isn't completely backwards compatible with XPath 1.0 (https://www.w3.org/TR/xpath20/#id-backwards-compatibility), but it's easy to add new functions, like matches. That may be "good enough".

I created a new project for "libxpath2": https://github.com/kibook/libxpath2. The idea is that once I get this somewhat stable, s1kd-brexcheck can use it to support XPath 2.0-like functions at the very least.

[MihailCosmin]

Hi, That would be great. Because now I see there were many more BREX rules that used Xpath 2.0. The reason they were not picked up before, was because I was checking only one DM. As soon as I started checking all DMs, more BREX rules were found to be wrong.

I see that Saxon offers support for Xpath 2.0

Saxon-HE (Home Edition) is an open source product available under the Mozilla Public License version 2.0. It provides implementations of XSLT (3.0), XQuery (3.1), and XPath (2.0 and 3.1) at the basic level of conformance defined by W3C, plus from Saxon 10 the optional features higher-order functions and dynamic evaluation. It is available for Java and .NET.

They have also some files for C: https://www.saxonica.com/download/c.xml

Not sure if you can use these, as I am a complete zero when talking about C.

[kibook]

Do you know if you use XPath 2.0-specific features besides functions, such as the if or for keywords? I'm not sure if those can be implemented as easily as adding extra functions.

Would you be able to provide a list of the XPath 2.0 functions used in your BREX? Eventually, I would like to implement all XPath 2.0 functions, but that's going to take some time. If you can give me a list of the ones you use, I could focus on those first.

I'll also look at Saxon to see if it would be possible to have an option to use its XPath engine to validate rules instead of libxml2 if the user chooses, in order to get complete XPath 2.0 support.

[MihailCosmin]

Do you know if you use XPath 2.0-specific features besides functions, such as the if or for keywords?

No cases where "if" or "for" are used.
In fact all other rules that were wrong used only the three functions from below.

lower-case
matches
tokenize

But full support for Xpath 2.0 could still be needed.

[kibook]

Thanks for the list! However, after looking more into XPath 2.0, I don't think just adding XPath 2.0-like functions will be enough. For example, the way the tokenize function is used in your example is specific to XPath 2.0 syntax.

I'll probably focus more on getting the Saxon XPath engine working within s1kd-brexcheck.

[kibook]

I have added an experimental option to use Saxon to evaluate the objectPath instead of libxml2. In my tests, it worked successfully on all the example XPath 2.0-based rules you listed.

At least for now, it is a compile-time option, so that Saxon is not a requirement to build and run s1kd-brexcheck. To enable it:

1. Download and install Saxon/C from http://www.saxonica.com/saxon-c/index.xml

2. Build with:

   $ make BREXCHECK_XPATH_ENGINE=saxon
   

I've only tested it on Linux so far, so the Makefile might still need some adjustments to work out-of-the-box on Windows with MinGW/Cygwin.

I have included a slightly modified version of the Saxon/C API along with the s1kd-brexcheck source that addresses a bug in the latest released version (1.2.1): https://saxonica.plan.io/issues/4513. The XPathProcessor::declareNamespace function for declaring namespaces for use in XPath expressions (like pdf:style) was broken.

There are still some major issues left to work out, like what appear to be big memory leaks within the Saxon/C library itself, though I could also just be misunderstanding how to properly free everything. Either way, it doesn't seem suitable to use on large sets of DMs in one call right now, but a separate call for each DM works okay, though it is significantly slower.

[MihailCosmin]

Do I need to add an argument to the brexcheck command to do the check using saxon?
I'm trying on Linux and I'm getting these errors:

xmlXPathCompOpEval: function **lower-case** not found
XPath error : Unregistered function
s1kd-brexcheck: WARNING: Ignoring invalid object path in ...

xmlXPathCompOpEval: function **matches** not found
XPath error : Unregistered function
s1kd-brexcheck: WARNING: Ignoring invalid object path in ...

XPath error : Invalid expression
s1kd-brexcheck: WARNING: Ignoring invalid object path in ...

[kibook]

Did you build with:

$ make BREXCHECK_XPATH_ENGINE=saxon

From your errors, it is still using libxml2's XPath engine. You can check which engine it uses with:

$ s1kd-brexcheck --version

There is a new XPath engine: line.

I'm still working out what is needed for a binary package that would have Saxon enabled, so you will have to build from source to test.

[MihailCosmin]

Yup, that did it.

Now it seems to work. I also made an error in a DM according to a BREX rule that used matches, and it was detected.

Just for info. I tried to build on Windows and I got the below errors:

$ make BREXCHECK_XPATH_ENGINE=saxon
g++ -O3 -c -Isaxon/Saxon.C.API -o saxon/saxon.o `pkg-config --cflags libxml-2.0` saxon/saxon.cpp
cc -O3 -c -o saxon/SaxonCGlue.o saxon/Saxon.C.API/SaxonCGlue.c
cc -O3 -c -o saxon/SaxonCXPath.o saxon/Saxon.C.API/SaxonCXPath.c
g++ -O3 -c -o saxon/SaxonProcessor.o saxon/Saxon.C.API/SaxonProcessor.cpp
saxon/Saxon.C.API/SaxonProcessor.cpp: In member function ‘void SaxonProcessor::setResourcesDirectory(const char*)’:
saxon/Saxon.C.API/SaxonProcessor.cpp:516:2: error: ‘strncat_s’ was not declared in this scope; did you mean ‘strncat’?
  516 |  strncat_s(_getResourceDirectory(), destSize,dir, strlen(dir));
      |  ^~~~~~~~~
      |  strncat
make: *** [Makefile:55: saxon/SaxonProcessor.o] Error 1