Migrate fastly log processor to pluto

[edolstra]

This makes the service that ingest raw fastly logs and generates summaries (see https://github.com/NixOS/infra/tree/master/metrics/fastly) to pluto. Currently it's running on one of my home machines.

This service runs once a week and kicks off some AWS Athena queries. It expects the credentials for the fastly-log-processor AWS user in /home/nix-metrics/.aws/config. That user has access to the nixos-metrics, nixos-athena and fastly logs buckets.

[mweinelt]

Do we need a statically allocated user (and uid), or can the service use the DynamicUser feature, which creates an ad hoc user, when the service runs?

[edolstra]

That's probably possible. But it does need to have the AWS credentials somewhere.

[mweinelt]

In which case we could probably use LoadCredential.

   systemd.services.process-raw-nix-logs.serviceConfig.LoadCredential = [
     "aws-cred:/path/to/actual/aws-cred-file"
   ];

The secret will then be provided at $CREDENTIALS_DIRECTORY/aws-cred.

We currently deploy secrets through agenix. I can help integrate the secret, if needed.

[mweinelt]

Maybe you can share the secret with me via Bitwarden and I'll create the age file? Feel free to reach out on Matrix to expedite this process.

[mweinelt]

@edolstra Can we pick this up?

[edolstra]

This seems to have stopped working:

$ AWS_PROFILE=fastly-log-processor ./ingest-raw-logs.sh 2025-02-24 2025-03-03

An error occurred (InvalidRequestException) when calling the StartQueryExecution operation: Unable to verify/create output bucket nixos-athena

[arianvp]

We'll need to create an IAM user with a policy that has sufficient access.
I can help with that. I dont currently see a user named fastly-log-processor defined in this PR.

Note that it's not enough to have just a bucket policy. The IAM user policy must also allow bucket access. It worked for the eelco.dolstra user as that user has admin policy and the Union of the user policy and the bucket policy allows access

We could maybe also consider moving this to a small EC2 instance as then we can use an IAM role instead of user. Which means we don't have to think about credentials rotation. Though not super string opinion in that. It's just nice not to have to think about credentials.