Need Guidance on Ingesting Fortigate Firewall Logs from S3 to OpenSearch

[zahirktk78]

Hello GitHub Community,

I'm encountering an issue while attempting to ingest logs from a Fortigate Firewall stored in an S3 bucket into OpenSearch. Here's a breakdown of my setup and the steps I've taken:

Setup: I have two S3 bucket configured for my SIEM, which originally contained logs for AWS services like ELB, guard duty, CloudTrail, security hub etc). Recently, I added an additional bucket/prefix within the same S3 for FortiGate logs.
File Format: The logs in the S3 bucket are in .gz format.
Configuration Changes: I updated the aws.ini file with the following information:

[Fortigatefirewall]
index_name = log-aws-Fortigatefirewall
s3_key = firewall/
file_format = json
timestamp_key = datetime
#'2021-03-08T09:24:27.005808+0000'

ecs = destination.ip
destination.port
source.ip
source.port

destination.ip = dstip
destination.port = dstport
source.ip = srcip
source.port = srcport
Expected Outcome: After deploying these changes, I expected OpenSearch to create an index and ingest the logs automatically.
However, despite these configurations, I haven't seen any index created in OpenSearch, and there's no data being ingested.

I've tried creating my own index and index pattern, but still, no data is being ingested into OpenSearch.

Could someone please guide me on the necessary steps to successfully ingest additional logs or third-party logs from S3 into OpenSearch? Any insights or suggestions would be greatly appreciated.

Thank you in advance for your help!

[shpf1]

Hi 👋
how did you solve this?

[imcrimson777]

If the .gz log files are showing up in S3 but nothing is ingested in Opensearch, I would check the Lambda Function Logs: aes-siem-es-loader, either through Cloudwatch directly or in the Monitoring tab of the siem application. If there's a problem with loading the logs, this function will be outputting errors.

[shpf1]

thank you but they are not gz type, maybe this is the issue? the type is unknown it's - in the type column.
however they are custom json data in the format of simple text file
{}
{}
..
{}

[imcrimson777]

The console might display the "-" in the type even though it is actually a gz archive, it just depends on the file name. All my custom logs are in gz archives, but I don't know if that's a requirement.

Dig through the logs of the es_loader Lambda function, if there's a problem ingesting your logs it will very likely show errors there.