Azure Monitor Logs Sink openssl issue

[wiperpaul]

Hi,

I am having an issue with the Azure Monitor Logs sink, when I enable the sink instead of blackhole I receive an openssl error
2025-02-07T12:28:28.318108Z ERROR vector::topology::builder: Configuration error. error=Sink "azure_monitor_logs_sink": OpenSSL error

I noticed in prior issue that this can be related to a connection issue such as malformed connection I, but I am struggling to debug as my resourceid, customer_id, shared_key are all set correctly. I have also the following Envs set and CA certs present.

Is there any more debugging information available or suggestions. I am not sure if there is a tracing option for OpenSSL itself that is available.

VECTOR_LOG:                  Trace
SSL_CERT_DIR:                /etc/ssl/certs
RUST_BACKTRACE:              full

sinks:
    azure_monitor_logs_sink:
      type: azure_monitor_logs
      inputs:
        - azure_events_parse
      azure_resource_id: "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/example-rg/providers/Microsoft.OperationalInsights/workspaces/example-la"
      customer_id: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
      host: "ods.opinsights.azure.com"
      log_type: "mytablesourcename"
      shared_key: "redacted_for_issue"
      time_generated_key: "timestamp"

[wiperpaul]

root@vector-0:/# openssl s_client -connect mycustomerid.ods.opinsights.azure.com:443
CONNECTED(00000003)
depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root G2
verify return:1
depth=1 C = US, O = Microsoft Corporation, CN = Microsoft Azure RSA TLS Issuing CA 03
verify return:1
depth=0 C = US, ST = WA, L = Redmond, O = Microsoft Corporation, CN = ODS-SSL201712WEU.azure.com
verify return:1

I can connect via openssl when I turn off the sink and exec into the container

[pront]

Hmm, this is tricky. A few ideas here:

• I assume you already read this: https://vector.dev/docs/reference/configuration/tls/
• vector validate --config /path/to/config.toml
• export SSLKEYLOGFILE=/tmp/ssl_key_log (you can refer to openssl docs)
• Compare openssl version -a with ldd $(which vector) | grep ssl (use strings for macOS)

[wiperpaul]

Thanks for the quick response the suggestions were useful. I can confirm the probe finds my certificates properly and that I am not mounting the certs in a way that overwrites or has bad perms.

• The vector validate command also gives the same OpenSSL error.
• openssl s_client command with same env for cert dir works
• tcpdump on my debug container targeted at vector container with tcpdump -i any -A -s 0 tcp port 443 I see the connection; however when I run vector validate triggering Openssl error I do not see a connection
• further attempt to witness connection during validate with tcpdump -i any -A -s 0, I see no external connection attempt when getting the OpenSSL Error

Perhaps I am interpreting the options for the component incorrectly and I do not have it set to use TLS

[wiperpaul]

root@vector-0:/tmp# vector validate vector4.yaml
2025-02-09T22:01:15.559070Z DEBUG vector::app: Internal log rate limit configured. internal_log_rate_secs=10
2025-02-09T22:01:15.559103Z  INFO vector::app: Log level is enabled. level="Trace"
2025-02-09T22:01:15.559142Z DEBUG vector::app: messaged="Building runtime." worker_threads=48
2025-02-09T22:01:15.559206Z TRACE mio::poll: registering event source with poller: token=Token(1), interests=READABLE
√ Loaded ["vector4.yaml"]
2025-02-09T22:01:15.593436Z DEBUG vector::topology::builder: Building new source. component=fake_source
2025-02-09T22:01:15.595229Z DEBUG vector::topology::builder: Building new sink. component=azure_monitor_logs_sink

Component errors
----------------
x Sink "azure_monitor_logs_sink": OpenSSL error`

[wiperpaul]

Concluded that it is in fact a malformed secret it seems that my secret configuration is not being picked up and is most likely actually the string "SECRET[1_secrets.shared_key]".

shared_key: SECRET[1_secrets.shared_key]

secret:
  1_secrets:
    type: directory
    path: /mnt/secrets-store/kv1-secrets
    remove_trailing_whitespace: true
  2_secrets:
    type: directory
    path: /mnt/secrets-store/kv2-secrets
    remove_trailing_whitespace: true

[wiperpaul]

#15796
dashes not supported

[pront]

Oh I see, great detective work. If only the error pointed us to the right direction... Can you workaround this by removing unsupported chars for now? I do think we can remove this limitation in the future.

[wiperpaul]

Yes, I got around it by changing my csi secretstore provider helm chart to replace - with _ in the secret name when pulling from azure keyvault vs what is stored on the /mnt. Thanks for the involvement.

Some validation on the Azure Monitor Logs Sink would be nice, though I understand I could have caught it in the log message that secrets were not loaded.