Providing a combined report for estimated counts table for Metaphlan

[apalleja]

Description of feature

When you run taxprofiler annotating with Metaphlan you can obtain the estimated counts by adding the following parameter: -t rel_ab_w_read_stats in db_params column in databases.csv. Then you obtain the profile files containing an extra column "estimated_number_of_reads_from_the_clade".

If you run the pipeline with --run_profile_standardisation then you get a combined report: metaphlan_db1_combined_reports.txt with the relative abundances of all the samples. I think it would be nice to also get a combined report of the estimated counts. Besides when you run the pipeline with -t rel_ab_w_read_stats and -run_profile_standardisation you get this error in the taxpasta_merge (extended report in the screenshot below):

Caused by:
Process NFCORE_TAXPROFILER:TAXPROFILER:STANDARDISATION_PROFILES:TAXPASTA_MERGE (metaphlan|db1) terminated with an error exit status (1)
...

Command output:
[15:19:40] CRITICAL Error in sample merge.py:419
'250703_S1_db1.metaphlan_profile'
with profile
'250703_S1_db1.metaphlan_profile.
txt'.
CRITICAL Unexpected report format. It has 5 columns but merge.py:424
only 4 are expected.

Discussed with @jfy133 at the taxprofiler Slack channel.

[jfy133]

Makes sense, We will get to this as soon as possible @apalleja ! Thanks for the issue!

[jfy133]

Note @Midnighter this will need a taxpasta update too I uess, would it be in scope there?

[Midnighter]

Yes, this will mainly be on taxpasta. I much prefer counts to percentage relative abundances. In fact, we currently multiply the percentages reported by a large number to arrive at a count. Can we make the count parameter mandatory/default?

P.S.: @apalleja pleased to "meet" someone from DTU Biosustain 😃.

[jfy133]

I much prefer counts to percentage relative abundances. In fact, we currently multiply the percentages reported by a large number to arrive at a count.

I dunno if that makes sense from the purpose of MetaPhlAn; the primary output is the abundance estimation (the calculation of which which they make a big point out of in their papers) that is default.

  -t ANALYSIS TYPE      Type of analysis to perform: 
                         * rel_ab: profiling a metagenomes in terms of relative abundances
                         * rel_ab_w_read_stats: profiling a metagenomes in terms of relative abundances and estimate the number of reads coming from each clade.
                         * reads_map: mapping from reads to clades (only reads hitting a marker)
                         * clade_profiles: normalized marker counts for clades with at least a non-null marker
                         * marker_ab_table: normalized marker counts (only when > 0.0 and normalized by metagenome size if --nreads is specified)
                         * marker_counts: non-normalized marker counts [use with extreme caution]
                         * marker_pres_table: list of markers present in the sample (threshold at 1.0 if not differently specified with --pres_th
                         * clade_specific_strain_tracker: list of markers present for a specific clade, specified with --clade, and all its subclades
                        [default 'rel_ab']

While I think it would be a nice idea to extend this by default, it would be difficult to implement 'cleanly' in the sense that we allow users almost complete freedom to customise the options of the tool (via teh db_params column).

I think it would be more for taxpasta to decide which columns to take from?

I think the three top options from the above would be the most common, so you can dynamically select which column depending on what taxpasta receives... maybe if taxasta gets a 5 column 'joint abundance plus reads' file, you could spit out two separate files: standardised rel_ab, and another for just the raw read counts?

[apalleja]

I agree that we should keep the abundance estimation and it would be nice if taxprofiler can offer the three top options (rel_ab, rel_ab_w_read_stats and reads_map). Definitely in favour of getting relative abundance AND counts tables. Then you have the two types of data and can run specific analyses for these two data types.

Another issue that I forgot to comment, but perhaps you have already noticed, is that if the pipeline ends in taxpasta for the 5 columns instead of 4 issue you do not get the multiQC report, so effectively you have to run taxprofiler twice: first one to get the relative abundance and multiQC and second one to get the counts (but wit a final summary).

[jfy133]

I agree that we should keep the abundance estimation and it would be nice if taxprofiler can offer the three top options (rel_ab, rel_ab_w_read_stats and reads_map). Definitely in favour of getting relative abundance AND counts tables. Then you have the two types of data and can run specific analyses for these two data types.

The (not really) issue here is that I don't think you can request all three modes in a single MetaPhlAn run, correct?

If that is the case, then you can already do this yourself: you specify the same metaphlan database three times in your database.csv but each entry with a different -t mode - so getting these raw files are possible - even if taxpasta will fail on some.

Another issue that I forgot to comment, but perhaps you have already noticed, is that if the pipeline ends in taxpasta for the 5 columns instead of 4 issue you do not get the multiQC report, so effectively you have to run taxprofiler twice: first one to get the relative abundance and multiQC and second one to get the counts (but wit a final summary).

I guess that's because the MultiQC module doesn't support that - so we would need someone to look into that... But that is somewhat out of our priorities currently