How to delete parts of a parse_csv output?

[rcmv]

Hi!
I have a CSV like the one below which I want to transform into json.

Timestamp,country,duration,quality
06/09/2022 10:33,UAE,7.8470,3.5362
06/09/2022 10:30,Saudi_Arabia,10.0380,3.4051
06/09/2022 10:27,Switzerland,7.8720,2.4598
06/09/2022 10:24,Israel,9.9360,2.8292

I've the following piece of code to convert the CSV file into json using remap/parse_csv:

[sources.my_csv_files_id]
type = "file"
include = [ "/home/rcmv2/ftp/*.csv" ]
start_at_beginning = true

[transforms.remap]
inputs = ["my_csv_files_id"]
type = "remap"
source = """
     row = parse_csv!(.message)
    .Timestamp = row[0]
    .country = row[1]
   .duration = row[2]
   .quality = row[3]
"""

[sinks.console]
inputs = ["remap"]
type = "console"
encoding.codec = "json"

The output is something like this:

{
  "Timestamp": "Timestamp",
  "country": "country",
  "duration": "duration",
  "file": "/home/rcmv2/ftp/sample.csv",
  "host": "ip-172-31-28-49.ec2.internal",
  "message": "Timestamp,country,duration,quality\r",
  "quality": "quality",
  "source_type": "file",
  "timestamp": "2022-09-07T13:40:16.916340007Z"
}
{
  "Timestamp": "06/09/2022 10:33",
  "country": "UAE",
  "duration": "7.8470",
  "file": "/home/rcmv2/ftp/sample.csv",
  "host": "ip-172-31-28-49.ec2.internal",
  "message": "06/09/2022 10:33,UAE,7.8470,3.5362\r",
  "quality": "3.5362",
  "source_type": "file",
  "timestamp": "2022-09-07T13:40:16.916355689Z"
}
...

I would like to have instead something like this:

{
 "Timestamp": "06/09/2022 10:33",
  "country": "UAE",
  "duration": "7.8470",
  "quality": "3.5362",
}
{
  "Timestamp": "06/09/2022 10:30",
  "country": "Saudi_Arabia",
  "duration": "10.0380",
  "quality": "3.4051",
}
...

I've tried some changes on the source like the one below but got no joy.
source = """ row = parse_csv!(.message) .Timestamp = row[0] .country = row[1] .duration = parse_duration!(row[2], unit:"s") .quality = to_float!(row[3]) del(.source_type) del(.timestamp) del(.file) del(.host) del(.message) """
Output:

{
"file":"/home/rcmv2/ftp/sample.csv",
"host":"ip-172-31-28-49.ec2.internal",
"message":"06/09/2022 10:33,UAE,7.8470,3.5362\r"
,"source_type":"file",
"timestamp":"2022-09-07T14:38:23.509016393Z"
}

{
"file":"/home/rcmv2/ftp/sample.csv",
"host":"ip-172-31-28-49.ec2.internal",
"message":"06/09/2022 10:30,Saudi_Arabia,10.0380,3.4051\r",
"source_type":"file",
"timestamp":"2022-09-07T14:38:23.509022698Z"
}
...

Can someone help me with this?
BR

[jszwedko]

Hi @rcmv ,

Could you share a full Vector config? The VRL source you have there is on one line, which I don't think will work properly.

[rcmv]

Funny thing. To give you the full code properly formatted I removed the to_float and the parse_duration functions from the source and tested the version below:

[sources.my_csv_files_id]
type = "file"
include = [ "/home/rcmv2/ftp/*.csv" ]
start_at_beginning = true

[transforms.remap]
inputs = ["my_csv_files_id"]
type = "remap"
source = """
    row = parse_csv!(.message)
   .Timestamp = row[0]
   .country = row[1]
   .duration = row[2]
   .quality = row[3]
   del(.source_type)
   del(.timestamp)
   del(.file)
   del(.host)
   del(.message)
    """

[sinks.console]
inputs = ["remap"]
type = "console"
encoding.codec = "json"

It worked.
Output:

{"Timestamp":"Timestamp","country":"country","duration":"duration","quality":"quality"}
{"Timestamp":"06/09/2022 10:33","country":"UAE","duration":"7.8470","quality":"3.5362"}
...

Just have to figure out how to convert the string numbers (duration and quality) into float type.

[rcmv]

Btw, how can I remove the first line (the one that only have the headers)?

[jszwedko]

I would do something like:

if row[0] == "Timestamp" {
  abort # headers
}

[rcmv]

Thank you @jszwedko !
It worked!