Why is the log_to_metric transform failing with "Unable to convert null field"?

[jerrac]

I'm trying to get docker stats into my metrics via Vector. Currently I'm testing by just outputting to the console.

I can get the stats from the Docker api, but the log_to_metric transform keeps erroring.

If I leave fields with a null value in the data, it errors with "Unable to convert null field".

If I use compact() to remove the fields, it errors because those fields are not present.

The docs https://vector.dev/docs/reference/configuration/transforms/log_to_metric/#null-fields say that null fields are ignored. But I'm getting an error instead.

So, what am I missing?

Testing on Ubuntu 24.04 with Vector 0.47.0.

My vector.yaml where I'm leaving the null fields in the message:

sources:
  docker_stats:
    type: exec
    command:
      - bash 
      - -c 
      - "for c in $$(docker ps -q); do curl -s --unix /var/run/docker.sock http://localhost/containers/$$c/stats?stream=false; done"
    mode: scheduled
transforms:
  docker_stats_parser:
    type: remap
    inputs: [ "docker_stats" ]
    source: |
      data = parse_json!(.message)
      flat = flatten!(value: data, separator: "_")
      .message = flat
  docker_to_metric:
    type: log_to_metric
    inputs:
      - docker_stats_parser
    metrics:
      - type: gauge
        kind: absolute
        field: message.blkio_stats_io_merged_recursive
        name: blkio_stats_io_merged_recursive
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.blkio_stats_io_queue_recursive
        name: blkio_stats_io_queue_recursive
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.blkio_stats_io_service_bytes_recursive
        name: blkio_stats_io_service_bytes_recursive
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.blkio_stats_io_service_time_recursive
        name: blkio_stats_io_service_time_recursive
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.blkio_stats_io_serviced_recursive
        name: blkio_stats_io_serviced_recursive
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.blkio_stats_io_time_recursive
        name: blkio_stats_io_time_recursive
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.blkio_stats_io_wait_time_recursive
        name: blkio_stats_io_wait_time_recursive
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.blkio_stats_sectors_recursive
        name: blkio_stats_sectors_recursive
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.cpu_stats_cpu_usage_total_usage
        name: cpu_stats_cpu_usage_total_usage
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.cpu_stats_cpu_usage_usage_in_kernelmode
        name: cpu_stats_cpu_usage_usage_in_kernelmode
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.cpu_stats_cpu_usage_usage_in_usermode
        name: cpu_stats_cpu_usage_usage_in_usermode
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.cpu_stats_online_cpus
        name: cpu_stats_online_cpus
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.cpu_stats_system_cpu_usage
        name: cpu_stats_system_cpu_usage
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.cpu_stats_throttling_data_periods
        name: cpu_stats_throttling_data_periods
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.cpu_stats_throttling_data_throttled_periods
        name: cpu_stats_throttling_data_throttled_periods
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.cpu_stats_throttling_data_throttled_time
        name: cpu_stats_throttling_data_throttled_time
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.memory_stats_limit
        name: memory_stats_limit
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.memory_stats_stats_active_anon
        name: memory_stats_stats_active_anon
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.memory_stats_stats_active_file
        name: memory_stats_stats_active_file
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.memory_stats_stats_anon
        name: memory_stats_stats_anon
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.memory_stats_stats_anon_thp
        name: memory_stats_stats_anon_thp
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.memory_stats_stats_file
        name: memory_stats_stats_file
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.memory_stats_stats_file_dirty
        name: memory_stats_stats_file_dirty
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.memory_stats_stats_file_mapped
        name: memory_stats_stats_file_mapped
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.memory_stats_stats_file_writeback
        name: memory_stats_stats_file_writeback
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.memory_stats_stats_inactive_anon
        name: memory_stats_stats_inactive_anon
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.memory_stats_stats_inactive_file
        name: memory_stats_stats_inactive_file
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.memory_stats_stats_kernel_stack
        name: memory_stats_stats_kernel_stack
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.memory_stats_stats_pgactivate
        name: memory_stats_stats_pgactivate
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.memory_stats_stats_pgdeactivate
        name: memory_stats_stats_pgdeactivate
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.memory_stats_stats_pgfault
        name: memory_stats_stats_pgfault
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.memory_stats_stats_pglazyfree
        name: memory_stats_stats_pglazyfree
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.memory_stats_stats_pglazyfreed
        name: memory_stats_stats_pglazyfreed
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.memory_stats_stats_pgmajfault
        name: memory_stats_stats_pgmajfault
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.memory_stats_stats_pgrefill
        name: memory_stats_stats_pgrefill
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.memory_stats_stats_pgscan
        name: memory_stats_stats_pgscan
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.memory_stats_stats_pgsteal
        name: memory_stats_stats_pgsteal
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.memory_stats_stats_shmem
        name: memory_stats_stats_shmem
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.memory_stats_stats_slab
        name: memory_stats_stats_slab
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.memory_stats_stats_slab_reclaimable
        name: memory_stats_stats_slab_reclaimable
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.memory_stats_stats_slab_unreclaimable
        name: memory_stats_stats_slab_unreclaimable
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.memory_stats_stats_sock
        name: memory_stats_stats_sock
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.memory_stats_stats_thp_collapse_alloc
        name: memory_stats_stats_thp_collapse_alloc
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.memory_stats_stats_thp_fault_alloc
        name: memory_stats_stats_thp_fault_alloc
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.memory_stats_stats_unevictable
        name: memory_stats_stats_unevictable
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.memory_stats_stats_workingset_activate
        name: memory_stats_stats_workingset_activate
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.memory_stats_stats_workingset_nodereclaim
        name: memory_stats_stats_workingset_nodereclaim
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.memory_stats_stats_workingset_refault
        name: memory_stats_stats_workingset_refault
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.memory_stats_usage
        name: memory_stats_usage
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.name
        name: name
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.networks_eth0_rx_bytes
        name: networks_eth0_rx_bytes
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.networks_eth0_rx_dropped
        name: networks_eth0_rx_dropped
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.networks_eth0_rx_errors
        name: networks_eth0_rx_errors
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.networks_eth0_rx_packets
        name: networks_eth0_rx_packets
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.networks_eth0_tx_bytes
        name: networks_eth0_tx_bytes
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.networks_eth0_tx_dropped
        name: networks_eth0_tx_dropped
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.networks_eth0_tx_errors
        name: networks_eth0_tx_errors
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.networks_eth0_tx_packets
        name: networks_eth0_tx_packets
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.num_procs
        name: num_procs
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.pids_stats_current
        name: pids_stats_current
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.pids_stats_limit
        name: pids_stats_limit
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.precpu_stats_cpu_usage_total_usage
        name: precpu_stats_cpu_usage_total_usage
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.precpu_stats_cpu_usage_usage_in_kernelmode
        name: precpu_stats_cpu_usage_usage_in_kernelmode
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.precpu_stats_cpu_usage_usage_in_usermode
        name: precpu_stats_cpu_usage_usage_in_usermode
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.precpu_stats_online_cpus
        name: precpu_stats_online_cpus
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.precpu_stats_system_cpu_usage
        name: precpu_stats_system_cpu_usage
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.precpu_stats_throttling_data_periods
        name: precpu_stats_throttling_data_periods
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.precpu_stats_throttling_data_throttled_periods
        name: precpu_stats_throttling_data_throttled_periods
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
      - type: gauge
        kind: absolute
        field: message.precpu_stats_throttling_data_throttled_time
        name: precpu_stats_throttling_data_throttled_time
        tags:
          host: "{{ host }}"
          greptime_timestamp: "{{ timestamp }}"
          container_id: "{{ message.id }}"
sinks:
  out:
    type: console
    target: stdout
    encoding:
      codec: json
      json:
        pretty: false
    inputs:
      - docker_stats_parser

Thanks in advance!

[pront]

Hi @jerrac, I think you are observing the expected behavior.

vector/src/transforms/log_to_metric.rs

Lines 905 to 909 in a0ef7aa

	TransformError::PathNull { path } => {
	emit!(LogToMetricFieldNullError {
	field: path.as_ref()
	})
	}

The error is emitted to notify you there are invalid events and the component_errors_total is increased. No event is emitted but Vector will keep running.