[BUG] Potential Issues in Code Paths for Non-Sars-CoV2 Organisms

[Michal-Babins]

Describe the bug
A clear and concise description of what the bug is. Be sure to check existing (and closed) issues before posting!

Hey @joshuailevy , thank you for your continued work and development on Freyja! I wanted to bring to your attention a few scenarios for non Sars-CoV-2 organisms that might be causing issues. Specifically when we were running for MPXV we were seeing a few scenarios error out:

When we ran freyja with the --pathogen flag set for MPXV and barcodes provided, freyja runs successfully.

However, when we started to add some layers we started running into issues:

When we add another layer and we run freyja with the --meta flag and set it to the lineage yml, we get a failure that arises from the buildLineageMap function that a json file can't be read. It seems that regardless, even if --lineageyml is specified, the buildLineageMap is within the code path no matter what and will lead to this failure.

We find another layer when we add the --depthcutoff and set it above the default of 0, the collapse_barcode function get's called and crashes when mapping to the pango_aliases. Specifically with mpox there seems to be mismatch between naming in barcode and lineage file. We also see that it's detecting duplicates from the barcodes that we can't identify maunally when we look at the barcodes csv file.

Here is a naming discrepancy we notice:

name: MPX-IIb.A but alias: MPX-II.b.A
name: MPX-Ia but alias: MPX-I.a

To Reproduce
Steps to reproduce the behavior:

For the scenario where --depthcutoff is set to default, but the lineage file is passed:

freyja demix \
  --meta /cromwell_root/fc-19055358-7daa-40be-bb7e-c11c0672326e/freyja/2024-09-12_mpox_lineages.yml \
  --barcodes /cromwell_root/fc-19055358-7daa-40be-bb7e-c11c0672326e/freyja/2024-09-12_barcode.csv \
  B29_VSP_freyja_variants.tsv \
  B29_VSP_freyja_depths.tsv \
  --output B29_VSP_freyja_demixed.tmp

The resulting error is:

Traceback (most recent call last):
  File "/opt/conda/envs/freyja-env/bin/freyja", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/opt/conda/envs/freyja-env/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/freyja-env/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/freyja-env/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/freyja-env/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/freyja-env/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/freyja-env/lib/python3.12/site-packages/freyja/_cli.py", line 121, in demix
    mapDict = buildLineageMap(meta)
              ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/freyja-env/lib/python3.12/site-packages/freyja/sample_deconv.py", line 49, in buildLineageMap
    dat = json.load(f0)
          ^^^^^^^^^^^^^
  File "/opt/conda/envs/freyja-env/lib/python3.12/json/__init__.py", line 293, in load
    return loads(fp.read(),
           ^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/freyja-env/lib/python3.12/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/freyja-env/lib/python3.12/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/freyja-env/lib/python3.12/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
tail: cannot open 'B29_VSP_freyja_demixed.tmp' for reading: No such file or directory
2025/03/31 23:58:06 Starting delocalization.

For where --depthcutoff is provided:

freyja demix \
  --barcodes 2024-09-12_barcode.csv \
  --depthcutoff 5 \
  B29_VSP_freyja_variants.tsv \
  B29_VSP_freyja_depths.tsv \
  --output B29_VSP_freyja_demixed.tmp

And here is the error:

Traceback (most recent call last):
  File "/opt/conda/envs/freyja-env/bin/freyja", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/opt/conda/envs/freyja-env/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/freyja-env/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/freyja-env/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/freyja-env/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/freyja-env/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/freyja-env/lib/python3.12/site-packages/freyja/_cli.py", line 116, in demix
    df_barcodes = collapse_barcodes(df_barcodes, df_depth, depthcutoff,
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/freyja-env/lib/python3.12/site-packages/freyja/utils.py", line 1008, in collapse_barcodes
    set([alias.split('.')[0] for alias in pango_aliases])) > 1
                                          ^^^^^^^^^^^^^
UnboundLocalError: cannot access local variable 'pango_aliases' where it is not associated with a value
tail: cannot open 'B29_VSP_freyja_demixed.tmp' for reading: No such file or directory

Expected behavior
We are expecting the same behavior for non sars-cov-2 pathogens as we are for sars-cov-2.

Output
If applicable, add screenshots or command line output to help explain your problem.

Additional info (please include the following information, if applicable):

• Freyja version [1.5.3]
• Barcode file version 2024-09-12

Additional context
Hopefully this summarized the issues well, we are happy to help and contribute to any development to Freyja!

[joshuailevy]

Hey @Michal-Babins!

Thanks for bringing this up - sorry for the delay in my response, I just got back from a few weeks of travel and there's a substantial backlog of things to catch up on.

I think this might actually be due to some lack of clarity on our end on what the different metadata types are, and how they're structured. So actually, you need to provide the lineage yml file as the argument following the --lineageyml option. --meta is a separate metadata file (a bit of relic from the original implementation) that encodes the VOC/VOI groups (e.g., BA.1 = BA.1, BA.1.1,...), that is only something we've really done for SARS-CoV-2.

As long as an accurate lineage yml is passed for the --lineageyml option, you should be good to go. It isn't too important to have that one from and older version of that file though, since in general groupings don't ever decrease in size (unless the definition is reclassified or something like that, but that's pretty rare).

Josh

[Michal-Babins]

Hey @joshuailevy following up and getting back to this. Sorry for the delay!

So it's worth deprecating the use of --meta all together in favor of --lineageyml and pointing each pathogen to it's corresponding lineage yaml file that can be found here:
https://github.com/andersen-lab/Freyja-barcodes

[joshuailevy]

Hey @Michal-Babins,

Good to hear from you! Agreed there's a lot we can offload from the --meta option to --lineageyml. We still will need to keep some additional metadata on things like WHO variant designations though (could be potentially be incorporated directly into the lineage yml) - this is predominantly a SARS-CoV-2 related concern though.

Josh