FIltering and Tests for Filters

[satellite-no]

Trying to write a simple filter and I think i'm over complicating things.

Goal: drop "unsupported" file types from s3. type examples are .snappy or .parquet.

tranform:

type: filter
inputs:
  - my-s3-bucket
condition:
  type: vrl
  source: |
    !ends_with(string!(.object), "snappy", case_sensitive: false) ||
    !ends_with(string!(.object), "parquet", case_sensitive: false)

test:

name: Drop unsupported S3 objects
inputs:
  - type: vrl
    insert_at: drop_my-s3-bucket
    source: |-
      . = {
        "bucket":"my-s3-bucket",
        "message":"gibberesh",
        "object":"raw/win/log_date=2025-04-14/3d52fdb4-d5d5-4f3f-b944-c7fa0820e51d.snappy",
        "region":"us-east-1",
        "source_type":"aws_s3",
        "timestamp":"2025-04-14T10:13:29Z"
      }
  - type: vrl
    insert_at: drop_my-s3-bucket
    source: |-
      . = {
        "bucket":"my-s3-bucket",
        "message":"gibberesh",
        "object":"raw/win/log_date=2025-04-14/3d52fdb4-d5d5-4f3f-b944-c7fa0820e51d.parquet",
        "region":"us-east-1",
        "source_type":"aws_s3",
        "timestamp":"2025-04-14T10:13:29Z"
      }
  - type: vrl
    insert_at: drop_my-s3-bucket
    source: |-
      . = {
        "bucket":"my-s3-bucket",
        "message":"gibberesh",
        "object":"raw/win/log_date=2025-04-14/3d52fdb4-d5d5-4f3f-b944-c7fa0820e51d.gz",
        "region":"us-east-1",
        "source_type":"aws_s3",
        "timestamp":"2025-04-14T10:13:29Z"
      }

outputs:
  - extract_from: core_s3
    conditions:
      - type: vrl
        source: |-
          # Validate .ch_database attributes.
          assert!(exists(.object))
          assert!(is_string(.object))
          assert!(!ends_with(string!(.object), "snappy", case_sensitive: false) ||
                      !ends_with(string!(.object), "parquet", case_sensitive: false),
                  message: "Unsupported S3 object format")

My hope would be 1 message makes it through to core_s3 and that event doesnt include the .snappy or .parquet file extenstions.

[pront]

transforms:
  my_filter:
    type: filter
    inputs:
      - my-s3-bucket
    condition:
      type: vrl
      source: |
        !ends_with(string!(.object), "snappy", case_sensitive: false) ||
        !ends_with(string!(.object), "parquet", case_sensitive: false)


tests:
  - name: Drop unsupported S3 objects
    inputs:
      - type: vrl
        insert_at: my_filter
        source: |-
          . = {
            "bucket":"my-s3-bucket",
            "message":"gibberesh",
            "object":"raw/win/log_date=2025-04-14/3d52fdb4-d5d5-4f3f-b944-c7fa0820e51d.snappy",
            "region":"us-east-1",
            "source_type":"aws_s3",
            "timestamp":"2025-04-14T10:13:29Z"
          }
      - type: vrl
        insert_at: my_filter
        source: |-
          . = {
            "bucket":"my-s3-bucket",
            "message":"gibberesh",
            "object":"raw/win/log_date=2025-04-14/3d52fdb4-d5d5-4f3f-b944-c7fa0820e51d.parquet",
            "region":"us-east-1",
            "source_type":"aws_s3",
            "timestamp":"2025-04-14T10:13:29Z"
          }
      - type: vrl
        insert_at: my_filter
        source: |-
          . = {
            "bucket":"my-s3-bucket",
            "message":"gibberesh",
            "object":"raw/win/log_date=2025-04-14/3d52fdb4-d5d5-4f3f-b944-c7fa0820e51d.gz",
            "region":"us-east-1",
            "source_type":"aws_s3",
            "timestamp":"2025-04-14T10:13:29Z"
          }

    outputs:
      - extract_from: my_filter
        conditions:
          - type: vrl
            source: |-
              # Validate .ch_database attributes.
              assert!(exists(.object))
              assert!(is_string(.object))
              assert!(!ends_with(string!(.object), "snappy", case_sensitive: false) ||
                          !ends_with(string!(.object), "parquet", case_sensitive: false),
                      message: "Unsupported S3 object format")

Running tests
test Drop unsupported S3 objects ... passed

The filter condition works as expected

[satellite-no]

Thank you for the second set of eyes!