FGPA instances support

[chris-rands]

Thank you for nf-co2footprint. We would like to measure the footprint from running Illumina's DRAGEN pipelines, which run on AWS f1/f2 FPGA EC2 instances (or on-prem DRAGEN boxes). Can this be supported or will the existing reports be able to produce reasonable estimates for this infrastructure?

A related question, AWS claims they are working towards carbon neutrality so are these metrics accurate for AWS/the cloud? Thank you

[JosuaCarl]

Thanks for the request. We are working on making the plugin give meaningful estimates for cloud computation. In theory, it's pretty similar to just running it on any machine as we are also accessing the CPU model and infer energy usage from that as well as Carbon Intensity (CI) from the location. The second part is a bit more difficult for globally distributed datacenters, but doable.

With regards to the claim of working towards Carbon neutrality: We are working on including some information from the providers (like Power Usage Effectiveness (PUE) of datacenters), but do currently not rely on Carbon estimates from companies. We will compare our estimates with their reports (in particular AWS's) before the next release and see whether our estimates match in scale.
In general we often find companies like AWS and Azure,... to claim carbon neutrality / reduced emissions because of carbon offsetting. I would currently not consider this, as it is not in the scale of the plugin (providing emission estimates, not net-estimates) and the long-term effectiveness of current carbon capture methods is disputed.

[chris-rands]

Hello and thank you for the prompt reply. That's great that you are working to improve estimates for the cloud, I look forward to that! Would that also work for F1/F2 ( https://aws.amazon.com/ec2/instance-types/f2/ ) instance types?

I completely understand why capturing metrics from off-setting would be out-of-scope

[JosuaCarl]

It seems that F2 uses AMD EPYC processors with an acceleration technology. They are featured in our new TDP data table which will come in the next release, so the information should be available to us.
Apart form that we are tracking the average usage from the requested CPUs and the time it took to compute. So, yes I don't see why this shouldn't work as intended. I will however put a pin in this and follow up on it before the next release.

[chris-rands]

Hi @JosuaCarl , thank you again for this. Do you have a timeline for the next release? We are keen to use the plugin with the updates to provide improved estimates for workflows ran in the cloud, thanks

[JosuaCarl]

Hi @chris-rands, We are actually currently working on integrating AWS processors and just a short step ahead of the 1.0.0 release. But we are still facing issues reagarding the usage of CPU manufacturer data. We waited a long time for an answer by the respective companies and are currently doing a work-around in order to be able to relase. Our original aim was to release in June, but with the mentioned bumps, this could be shifted a bit. You are however on our agenda. I will notify you once we are in pre-release.

[chris-rands]

Thank you! Look forward to testing the release when it's ready

[JosuaCarl]

Just to give you a short update: CPUs are supported independent of instance, but we are still in talks with the manufacturers and cloud providers (including AWS) to be able to use all of the CPU data.

Accelerators and GPUs are a different story. Nextflow does not integrate them very well, resulting in little trace metrics being reported. We are working on integration, but this will probably take quite a while and require changes to Nextflows core code.