Best practice to cut down op geoip lookup events

[driehuis]

I'm using this remap stanza to add GeoIP lookups to all records having a source_ip field:

  geoip:
    type: remap
    inputs:
      - parse_logs
    source: |-
      .geoip = get_enrichment_table_record!("geoip_table", { "ip": .source_ip })

This results in tons of messages from vector:

ERROR transform{component_kind="transform" component_id=geoip component_type=remap}: vector::internal_events::remap: Internal log [Mapping failed with event.] is being suppressed to avoid flooding.

I presume these are caused by RFC1918 addresses hitting GeoIP, but I'm not sure. The details seem to be left suppressed in the error message. I think these errors do not prevent the event being logged (I do see a ton of events with geoip_country: null), so using the dropped output handler would probably not help.

So my question is twofold:

1. What is the best way to deal with failed GeoIP lookups?
2. What is the best way to deal with error messages to allow tracking the source of the issue?

I tried replacing get_enrichment_table_record! with find_enrichment_table_records! and that got rid of the errors, but it requires unwrapping the returned array and manually ditching empty returns. I wouldn't mind doing that but it feels like I'm missing something obvious!

[driehuis]

To answer my own question, the lookup can be silenced by writing it as:

  geoip:
    type: remap
    inputs:
      - parse_logs
    source: |-
      geoip, err = get_enrichment_table_record("geoip_table", { "ip": .source_ip })
      if err == null {
        .geoip = geoip
      }

If you want to record the error with the data, you can simple store the error:

  geoip:
    type: remap
    inputs:
      - parse_logs
    source: |-
      geoip, err = get_enrichment_table_record("geoip_table", { "ip": .source_ip })
      if err == null {
        .geoip = geoip
      } else {
        .error = err
      }

All of this is in the docs, but these need to be re-read recursively a lot to start to make sense :-)