Peppol TestBed issues

[wartab]

Hello,

We're currently in the process of migrating away from Oxalis for performance reasons. We want to merge the efforts of swapping the implementation at the same time as we do our certificate renewal that is happening soon for us.

For that purpose, we based our work on phase4-peppol-standalone.

Just for sanity's sake before running further tests, we tried to re-run the Peppol TestBed suite, but besides the TLS test, we fail the sending and receiving tests.
However, when sending (and receiving) documents to our own Test access point, we get no errors and everything works as expected. We also tested the new implementation (with phase4-peppol-standalone) against our old Oxalis access point in prod (please forgive me for testing in prod) and everything worked fine as well.

Are there any known issues regarding TestBed currently? Am I missing something that I perhaps forgot about?

This is the error I'm getting when receiving a document from the TestBed:

[2025-09-09 11:52:44,710] [phase4] [http-nio-8080-exec-6] ERROR o.apache.wss4j.common.crypto.Merlin -- Cannot find key for certificate in keystore of type [PKCS12] from provider [SUN version 24] with size [1] and aliases: {private_key_fo
r_pkcs12_certificate}
[2025-09-09 11:52:44,716] [phase4] [http-nio-8080-exec-6] ERROR c.h.p.i.s.SoapHeaderElementProcessorWSS4J -- Error processing the WSSecurity Header (WS Security error: FAILED_CHECK)
org.apache.wss4j.common.ext.WSSecurityException: Cannot find key for certificate
        at org.apache.wss4j.dom.processor.EncryptedKeyProcessor.getPrivateKey(EncryptedKeyProcessor.java:323)
        at org.apache.wss4j.dom.processor.EncryptedKeyProcessor.handleToken(EncryptedKeyProcessor.java:216)
        at org.apache.wss4j.dom.processor.EncryptedKeyProcessor.handleToken(EncryptedKeyProcessor.java:91)
        at org.apache.wss4j.dom.engine.WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:340)
        at org.apache.wss4j.dom.engine.WSSecurityEngine.processSecurityHeader(WSSecurityEngine.java:251)
        at com.helger.phase4.incoming.soap.SoapHeaderElementProcessorWSS4J._verifyAndDecrypt(SoapHeaderElementProcessorWSS4J.java:229)
        at com.helger.phase4.incoming.soap.SoapHeaderElementProcessorWSS4J.processHeaderElement(SoapHeaderElementProcessorWSS4J.java:569)
        at com.helger.phase4.incoming.AS4IncomingHandler._processSoapHeaderElements(AS4IncomingHandler.java:493)
        at com.helger.phase4.incoming.AS4IncomingHandler.processEbmsMessage(AS4IncomingHandler.java:661)
        at com.helger.phase4.incoming.AS4RequestHandler._handleSoapMessage(AS4RequestHandler.java:1644)
        at com.helger.phase4.incoming.AS4RequestHandler.lambda$handleRequest$5(AS4RequestHandler.java:2090)
        at com.helger.phase4.incoming.AS4IncomingHandler.parseAS4Message(AS4IncomingHandler.java:391)
        at com.helger.phase4.incoming.AS4RequestHandler.handleRequest(AS4RequestHandler.java:2109)
        at com.helger.phase4.servlet.AS4XServletHandler.handleRequest(AS4XServletHandler.java:239)
        at com.helger.phase4.servlet.AS4XServletHandler.handleRequest(AS4XServletHandler.java:268)

As stated before, this works if we send to ourselves and the program shows error messages when any of the keystore parameters are wrong (file, password, alias, private.password), so we're sure they're correct.

This is the error I'm getting when sending a document to the TestBed:

{
      "description":"An undefined error occurred.",
      "errorDetails":"The incoming Peppol message could not be processed.: Technical details: com.helger.phase4.peppol.Phase4PeppolException - TestBed Error Response: [Code=-4204, Description=Failed to correlate inbound request to a valid running test case. Make sure that your test case is in a running state and that you have followed the directions., Trace=6d6f3ab7-ca6c-4eb0-a15a-ffbbcc9108dc]",
      "category":"Content",
      "refToMessageInError":"bb561cb4-fc3f-488b-bee7-d1629ddd29c7@phase4",
      "errorCode":"EBMS:0004",
      "severity":"failure",
      "shortDescription":"Other"
    }

[phax]

Hmm - hard to say from the outside.
Most likely your AP truststore is only configured for the production CA, with the Testbed however you need to use the Test PKI and the Test truststore. Does that sound feasible?

Edit: in general, there are no known issues with the Testbed (as the Testbed also uses phase4)

[wartab]

I figured it wouldn't be easy to figure out just from what I said, sorry it's a bit vague.

As far as I can tell, I have done everything that's required to be considered in the test environment.

Extract of application.properties

global.production=false

peppol.stage=test

#org.apache.wss4j.crypto.merlin.truststore.file=truststore/2018/pilot-truststore.jks
org.apache.wss4j.crypto.merlin.truststore.file=truststore/2025/ap-test-truststore.p12
org.apache.wss4j.crypto.merlin.truststore.password=peppol

smpclient.truststore.path=truststore/2025/smp-test-truststore.p12
smpclient.truststore.password=peppol

Tried to switch between the old and the new trust store (jks vs p12), but it doesn't seem to be related. The parameters are doing something, because I get warnings when I change them to files that don't exist.

I don't think it is related to test vs. prod, because the document I successfully sent to myself over the test network has "smlDnsZone":"acc.edelivery.tech.ec.europa.eu." in it's metadata and the C3 certificate is indeed the one contained in my test .p12 file.

Edit: just understood why you might be thinking that I was in prod. When I tested the prod certificate, I had changed all those parameters before testing, in fact it wasn't even on the same server.

[phax]

Okay, thanks for the clarifications. The truststore configuration is fine (if the truststore types are set to PKCS12).

The problem is about finding the correct private key for an inbound message.
Obviously a keystore is configured, and that PKCS12 keystore contains one entry private_key_for_pkcs12_certificate - so it sounds okay.
So if this is not the problem, then eventually the sender made a mistake and encrypted with the wrong certificate. Does that sound like it could be a viable option for the source of the proble?

[wartab]

Yes, the truststore type are set to PKCS12 (for all truststores, be it for our own certificate, for ap, and for smp).

The sender that fails to send the document is the TestBed, in this case. It's hard to imagine that they might be doing things wrong, but perhaps there's some issue due to the recent PKI migration? We got the new certificates very recently and I know something changed compared to our old test-certificate from a couple years ago. Before, the entry name would just be "cert" instead of "private_key_for_pkcs12_certificate". Obviously, that shouldn't make a difference, but that's the only thing that stuck out to me.

One other thing I might be thinking of is that the TestBed might be caching certificates for Peppol identifiers after fetching them once from an SMP? Before doing my first test, we still had the SMP entry with the outdated certificate. Do you happen to know if that's a likely cause? I didn't think that could be an issue, since you don't need to be registered on an SMP to pass the TestBed (at least not in the past).

[phax]

Ah okay, I think we're getting closer :) The name of the key doesn't matter - as long as it is configured correctly.

Are you eventually trying to use the Testbed with G3 certificates already? The current testbed can only deal with G2 certificates. For the transmission tests, the Testbed also doesn't do SMP lookups - it is using the client certificate you are providing on "onboarding" (you know what I mean). So if you used a G3 AP certificate to access the Testbed, it will encrypt the messages with this G3 certificate. Please make sure that the certificate you use in the Testbed and the certificate you configured as key in your application are the same.

[wartab]

haha, I think we're onto something. I remember we were stuck on the form where we had to choose between G2 and G3, because there was no explanation and we picked 3 just because it was a higher number.

And now that you mention all of this, the Peppol Testbed website never actually authenticated me with the test certificate. It would only let me access the testbeds if I loaded the prod certificate into Firefox. I just tested it and it still does not work with the test certificate. So I assume that the certificate that is being used by the TestBed is the prod one and not the test one.

If you ask me, this is all way too complicated and sophisticated, I don't envy people who have to onboard now. Is there someone I can contact about this?

[phax]

Well, generally the Peppol ServiceDesk is the place to go - https://openpeppol.atlassian.net/servicedesk/customer/portal/1

As a workaround, try opening the Testbed in a private browser window - this should ask you for a certificate every time.

What browser are you using? The selection of the client certificate for each URL can be configured differently.

[wartab]

I think I had already tried the private navigation, it didn't ask me anything. I remember when we first used the Testbed things seemed to actually work.

The browser is Firefox.

I'll reach out tomorrow, thank you for all the help!

[phax]

Okay. See https://support.mozilla.org/en-US/questions/1416729 for some general purpose guideline - hope it helps

[yumtwinkle]

Following along .... looks like the error only happens with the TestBed, not in prod or local setups.

[phax]

The current version of the Peppol Testbed is designed to work only with G2 test certificates for the majority of cases