Logstash SSL Error

[GeoCoastie]

Having issues with Logstash crashing and not running correctly. Ive attempted to re-run the config and auth scripts, remove volumes and have yet to find a solution. Malcolm is running fine with one Hedgehog VM.

running docker logs malcolm-logstash-1

gives this output :

[2025-05-20T16:06:59,991][INFO ][org.logstash.beats.BeatsHandler] [local: 172.29.1.17:5044, remote: 172.29.1.1:48412] Handling exception: io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors (caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors)
[2025-05-20T16:06:59,991][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:500) ~[netty-codec-4.1.109.Final.jar:4.1.109.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-4.1.109.Final.jar:4.1.109.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.109.Final.jar:4.1.109.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.109.Final.jar:4.1.109.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.109.Final.jar:4.1.109.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.109.Final.jar:4.1.109.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.109.Final.jar:4.1.109.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.109.Final.jar:4.1.109.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.109.Final.jar:4.1.109.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.109.Final.jar:4.1.109.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.109.Final.jar:4.1.109.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.109.Final.jar:4.1.109.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.109.Final.jar:4.1.109.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.109.Final.jar:4.1.109.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.109.Final.jar:4.1.109.Final]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]
        

any assistance, suggestions is greatly appreciated.

[mmguero]

Hmmm, I haven't seen that exact error before. Let's double check a few things. Run these commands in the Malcolm installation directory:

• grep BEATS_SSL config/beats-common.env
• ls -l ./logstash/certs/ ./filebeat/certs/
• file ./logstash/certs/* ./filebeat/certs/*
• md5sum ./filebeat/certs/*

Then run md5sum /opt/sensor/sensor_ctl/logstash-client-certificates/* on the hedgehog and compare the values of those three files with the md5sums from the last command, and see if they match? Also, on the hedgehog, run grep BEAT_LS /opt/sensor/sensor_ctl/control_vars.conf and make sure those values are the three filenames within /opt/sensor/sensor_ctl/logstash-client-certificates/.

If you just want to "get things working" here's what I'd suggest to just regenerate the certs for logstash/filebeat and re-sync them up:

1. stop malcolm (./scripts/stop) and hedgehog (/opt/sensor/sensor_ctl/shutdown)
2. ./scripts/auth_setup
3. choose fwcerts (if it's the dialog-based menu, arrow down to it, hit space to populate the "radio button" and hit enter; or select the number and hit enter if it's the plaintext menu)
4. run ls -l ./logstash/certs/ ./filebeat/certs/ again and verify the timestamps on those files are new (ie., they just got regenerated)
5. resync the certs (see this guide, this is the malcolm side (go down to the "transfer self-signed" section) and this is the hedgehog side.
6. start Malcolm and wait a few minutes for it to come up (./scripts/start)
7. start Hedgehog (/opt/sensor/sensor_ctl/supervisor.sh)

Let me know how it goes.

[GeoCoastie]

Logstash is now healthy, I am going to let it run for a while to verify any additional errors.

The certs within /logstash-client-certficates/ were not the same hash values as within /filebeat/certs/
after backing up those certs and copying the correct hashed certs to the folder, restarting services I now see OpenSearch dashboards populating with data. More specifically i can see "Connections" data flowing through that dashboard.

on Malcolm I still see this error...

Next step is to just re-run the auth-setup scripts and resync the certs.

Ill follow back up here in a day or so. Thank you for the assistance!

[GeoCoastie]

Here is the output from Malcolm and Hedgehog from the terminal.