parsing csv data and mapping values -> how ?

[StefanSa]

Hi there,
I have this syslog message, the message content is in CSV format.

Nov 30 16:44:36 PA-220 1,2018/11/30 16:44:36,012801096514,THREAT,url,2049,2018/11/30 16:44:36,192.168.15.224,152.195.55.192,192.168.1.63,152.195.55.192,new_outbound_from_trust,,,ssl,vsys1,trust,untrust,ethernet1/2,ethernet1/1,send_to_mac,2018/11/30 16:44:36,28191,1,52984,443,37679,443,0x403000,tcp,block-url,"consent.cmp.oath.com/",(9999),business-and-economy,informational,client-to-server,7726,0x2000000000000000,192.168.0.0-192.168.255.255,United States,0,,0,,,0,,,,,,,,0,0,0,0,0,,PA-220,,,,,0,,0,,N/A,unknown,AppThreat-0-0,0x0,0,4294967295,

Have these CSV mappings / array for it.

      mappings:
        client.ip: 7
        source.ip: 7
        source.address: 7
        server.ip: 8
        destination.ip: 8
        destination.address: 8
        source.nat.ip: 9
        client.nat.ip: 9
        destination.nat.ip: 10
        server.nat.ip: 10
        panw.panos.ruleset: 11
        client.user.name: 12
        source.user.name: 12
        server.user.name: 13
        destination.user.name: 13
        network.application: 14
        panw.panos.virtual_sys: 15
        panw.panos.source.zone: 16
        observer.ingress.zone: 16
        panw.panos.destination.zone: 17
        observer.egress.zone: 17
        panw.panos.source.interface: 18
        observer.ingress.interface.name: 18
        panw.panos.destination.interface: 19
        observer.egress.interface.name: 19
        panw.panos.flow_id: 22
        client.port: 24
        source.port: 24
        destination.port: 25
        server.port: 25
        source.nat.port: 26
        client.nat.port: 26
        destination.nat.port: 27
        server.nat.port: 27
        _temp_.labels: 28
        network.transport: 29
        panw.panos.action: 30
        panw.panos.threat.resource: 31
        url.original: 31
        panw.panos.threat.name: 32
        panw.panos.url.category: 33
        log.level: 34
        _temp_.direction: 35
        _temp_.srcloc: 38
        _temp_.dstloc: 39
        panw.panos.network.pcap_id: 42
        panw.panos.file.hash: 43
        user_agent.original: 46
        file.type: 47
        network.forwarded_ip: 48
        http.request.referer: 49
        source.user.email: 50
        panw.panos.subject: 51
        destination.user.email: 52
        observer.hostname: 59

Example client.ip and source.ip get the value from the position 7 from the CSV message.
What is the best way to do this with Vector?
Please give an example.

Thank you very much in advance.

[JeanMertz]

You should take a look at the remap transform, that will allow you to manipulate the data as you see fit.

See also:

• documentation on the remap transform
• documentation on Vector Remap Language
• the list of functions available in VRL

[StefanSa]

@JeanMertz
Hi Jean,
first of all thank you for your quick reply. I know that this (maybe) works with vrl.
Here there is also the function parse_csv, but as far as i understand it it doesn't provide a function to map the values, so my question about it.
There is an example with lua https://vector.dev/guides/advanced/parsing-csv-logs-with-lua/, but wanted to solve this with vrl.

So again, specifically the question.
How do i do the mapping with data previously extracted with the function parse_csv.
Or is that not possible with vrl ?

Thanks for your help and time and of course for vector.dev

[jszwedko]

Hi @StefanSa ,

What do you mean by "map the values"? Do you mean assign the headers? Assuming you assigned parse_csv to row you can do:

.client.ip = row[7]

[StefanSa]

Hi @jszwedko
just like that, really that simple (assign the headers) ?
Where can i find that in the documentary ?

[jszwedko]

Hi @StefanSa ,

You can assign the headers like I showed, after parsing the CSV row via parse_csv row.

row = parse_csv!(.some_field)
.client.ip = row[7]

Etc.

Hope that helps!

[StefanSa]

hi @jszwedko
You make my day.
You can mark the question as answered.
Thank you for your professional help Jesse.