"splunk_hec_logs" Sink - Timestamp with Nano seconds not appearing in Splunk

[goatmale]

Hi,
I seem to be having an issue where if we specify a java running in a container timestamp format like below:
<timestampPattern>yyyy-MM-dd'T'HH:mm:ss.SSS'Z'</timestampPattern>
or
<timestampPattern>yyyy-MM-dd'T'HH:mm:ss.SSS</timestampPattern>
Printing to console shows the time as:
{"timestamp":"2023-01-06T19:37:15.128Z"}
However when sent to spunk sink (w/ encoding json), the event is shown as
2023-01-06T13:37:15.000-06:00 for the _time in the UI - seemingly dropping or not having sent the microseconds.

Oddly enough not setting the timestamp field and letting vector figure out the timestamp shows the ms properly, just i'd like to have the app timestamp as it's more accurate (maybe this isn't even a concern?).

Using the docker image timberio/vector:0.25.2-debian

config is:

data_dir: "somedir"
api:
  enabled: true
  address: "0.0.0.0:8686"
  playground: true
sources:
  json_logs:
    type: "docker_logs"
    include_containers:
      - somejavacontainer
transforms:
  json_logs_remap:
    type: remap
    inputs:
      - json_logs
    source: |-
      labels =
        .docker_image_sha = .label.COMMIT_SHA
        .alloc_id = .label."com.hashicorp.nomad.alloc_id"
        .job_name = .label."com.hashicorp.nomad.job_name"
        .node_id = .label."com.hashicorp.nomad.node_id"
        .node_name = .label."com.hashicorp.nomad.node_name"
        .task_group_name = .label."com.hashicorp.nomad.task_group_name"
        .task_name = .label."com.hashicorp.nomad.task_name"
        del(.label)
      . = merge(., parse_json!(.message)) ?? .
sinks:
  out:
    type: console
    inputs:
     - json_logs_remap
    encoding:
      codec: "json"
  splunk:
    type: splunk_hec_logs
    inputs:
      - json_logs_remap
    endpoint: ${SPLUNK_HEC_ENDPOINT}
    host_key: host
    compression: gzip
    index: ${SPLUNK_HEC_INDEX}
    tls:
      verify_certificate: false
    encoding:
      codec: json
    healthcheck: true
    default_token: ${SPLUNK_HEC_TOKEN}

I am unsure on how to continue to troubleshoot this. Is there a way I can see if this is something I'm doing wrong, or something I need to do on my end?
Thank you!

[jszwedko]

Hey @goatmale !

Could you share the Vector configuration and version?

[goatmale]

I have editing my original post to include this info. Thank you!

[jszwedko]

Thanks! Can you share the log output of Vector. I'm guessing that, since the timestamp is an unparsed string, that Vector is actually just ignoring it (it should print "Timestamp was an unexpected type. Deferring to Splunk to set the timestamp.").

[goatmale]

Drat that was it - I don't know why I didn't check stderr!

However it's still not working - possibly my own fault:

      labels =
        .docker_image_sha = .label.COMMIT_SHA
        .alloc_id = .label."com.hashicorp.nomad.alloc_id"
        .job_name = .label."com.hashicorp.nomad.job_name"
        .node_id = .label."com.hashicorp.nomad.node_id"
        .node_name = .label."com.hashicorp.nomad.node_name"
        .task_group_name = .label."com.hashicorp.nomad.task_group_name"
        .task_name = .label."com.hashicorp.nomad.task_name"
        del(.label)
      parsed_message = 
        parse_json!(.message)
        .timestamp = to_timestamp!(.timestamp)
      . = merge(., parsed_message) ?? .

[jszwedko]

Np! I think you want something like:

.docker_image_sha = .label.COMMIT_SHA
.alloc_id = .label."com.hashicorp.nomad.alloc_id"
.job_name = .label."com.hashicorp.nomad.job_name"
.node_id = .label."com.hashicorp.nomad.node_id"
.node_name = .label."com.hashicorp.nomad.node_name"
.task_group_name = .label."com.hashicorp.nomad.task_group_name"
.task_name = .label."com.hashicorp.nomad.task_name"
del(.label)

. = merge(., parse_json!(.message)) ?? .

.timestamp = parse_timestamp!(.timestamp)

In your snippet, you are assigning .timestamp but then, I believe, overwriting . with the later merge to put the string timestamp back that is in parsed_message. In mine, I'm doing the timestamp parsing last.

[goatmale]

Thanks so much!!!

…

On Fri, Jan 6, 2023, 15:04 Jesse Szwedko ***@***.***> wrote: Np! I think you want something like: .docker_image_sha = .label.COMMIT_SHA .alloc_id = .label."com.hashicorp.nomad.alloc_id" .job_name = .label."com.hashicorp.nomad.job_name" .node_id = .label."com.hashicorp.nomad.node_id" .node_name = .label."com.hashicorp.nomad.node_name" .task_group_name = .label."com.hashicorp.nomad.task_group_name" .task_name = .label."com.hashicorp.nomad.task_name" del(.label) . = merge(., parse_json!(.message)) ?? . .timestamp = parse_timestamp!(.timestamp) In your snippet, you are assigning .timestamp but then, I believe, overwriting . with the later merge to put the string timestamp back that is in parsed_message. In mine, I'm doing the timestamp parsing last. — Reply to this email directly, view it on GitHub <#15854 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACSG6Q7MLMVRYBEDIFH2GLDWRCCEJANCNFSM6AAAAAATTMWFGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>