Does charcoal not support installation via mamba?

[SchwarzEM]

I tried a de novo installation and test of charcoal. I used to use conda for this but moved to mamba over the last two years. Trying to run commands that had worked for a conda-based installation, I seemed to be doing OK, but then got a crash at the test step:

cd $PROJECT/src ;
git clone https://github.com/dib-lab/charcoal charcoal_2023.10.19 ;

cd $PROJECT/src/charcoal_2023.10.19 ;

# This next step seems to work; at least, it does not emit shrieks to STDERR:
mamba env create -f environment.yml -n charcoal ;

# Now, try to test it:
mamba activate charcoal ;
python -m charcoal run demo/demo.conf -j 4 ;

[crashes, sigh:]
Traceback (most recent call last):
  File "/ocean/projects/mcb190015p/schwarze/mambaforge-pypy3/envs/charcoal/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,  
  File "/ocean/projects/mcb190015p/schwarze/mambaforge-pypy3/envs/charcoal/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/ocean/projects/mcb190015p/schwarze/src/charcoal_2023.10.19/charcoal/__main__.py", line 9, in <module>
    from charcoal.version import version
ModuleNotFoundError: No module named 'charcoal.version'

Is there some way to make charcoal installations with mamba work?

[SchwarzEM]

Today I tried one more time, making things as simple as possible in case somehow I was causing problems by using an environment name like charcoal_2023.10.19. Alas! Walking carefully through the entire official installation protocol seems to just fail.

Steps I followed, up to the point the test computation crashed:

cd $HOME/src ;
git clone https://github.com/dib-lab/charcoal charcoal ;

cd $HOME/src/charcoal ;
mamba env create -f environment.yml -n charcoal ;   # used to be conda
   
"""
Looking for: ["python[version='>=3.8,<3.10']", 'snakemake-minimal=6.5.1', 'screed', "click[version='>=7,<8']", 'pip', 'mamba', 'sourmash=4.4.3']
"""

mamba activate charcoal ;   # was conda
pip install -e . ;

"""
Obtaining file:///home/ems/src/charcoal
  Preparing metadata (setup.py) ... done
Collecting snakemake==6.4.1 (from charcoal-bio==0.1)
[... verify various requirements ...]
Installing collected packages: snakemake, charcoal-bio
  Attempting uninstall: snakemake
    Found existing installation: snakemake 6.5.1
    Uninstalling snakemake-6.5.1:
      Successfully uninstalled snakemake-6.5.1
  Running setup.py develop for charcoal-bio
Successfully installed charcoal-bio-0.1 snakemake-6.4.1
"""

# turn the environment off and then turn it back on
mamba deactivate ;
mamba activate charcoal ;

# Now try to run the test program:
python -m charcoal run demo/demo.conf -j 4 ;

# Get the same damn error as before, sigh.
Traceback (most recent call last):
  File "/home/ems/mambaforge-pypy3/envs/charcoal/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/ems/mambaforge-pypy3/envs/charcoal/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/ems/src/charcoal/charcoal/__main__.py", line 9, in <module>
    from charcoal.version import version
ModuleNotFoundError: No module named 'charcoal.version'

mamba deactivate ;

[ctb]

The installation works for me with mamba; it looks like you missed running the actual charcoal install command from the README:

pip install -e .

Now, on to running the demo... I'll let you know if and when it all works ;)

[SchwarzEM]

When was I supposed to run pip install -e . ?

I did, in fact, run it -- immediately after running mamba activate charcoal ;, which in turn I did after running mamba env create -f environment.yml -n charcoal ;. Did I get the order of line-commands wrong?

(See my second post on this thread, not the first post, which I agree overlooked pip install -e . ).

[ctb]

ahh, I see the second post! Sorry!

Per the README, the full order is:

git clone https://github.com/dib-lab/charcoal
cd ./charcoal/
conda env create -f environment.yml -n charcoal
conda activate charcoal
pip install -e .

This is because you need to install charcoal inside the conda/mamba environment that you so lovingly created for it ;). If you do the pip install before conda activate then you are running pip in whatever conda environment you were in before - probably base - and that conda environment won't have all the stuff from environment.yml in it.

What install docs are you using? I want to make sure to fix 'em.

I have now successfully completed the install and am running the demo, but that is failing for different reasons, which I will now proceed to fix ;). #endlessrecursivefailures

[ctb]

Now failing with #237, so that's progress!

[SchwarzEM]

The main page's install documentation is what I ultimately derived my line-commands from. As you can see, what I did was almost identical, with minor modifications: I did pip install, then turned off the conda environment, then turned it back on before trying to do the test run. I did that to make absolutely sure that anything installed by pip into the charcoal conda environment would be fully up and running for the test installation (which failed in the manner shown in my second post).

[SchwarzEM]

"If you do the pip install before conda activate then you are running pip in whatever conda environment you were in before - probably base - and that conda environment won't have all the stuff from environment.yml in it."

Yes, that's true -- I was indeed careful not to run pip in my non-conda-charcoal-activated general environment.

[ctb]

huh, interesting. The README-based version is working for me (up until a certain point).

Let me dig a little bit. I have some ideas.

[SchwarzEM]

Sure, have at it!

If at some point you come up with a walkthrough that completely works in your hands on UCD farm, let me have a set of line commands to use, and I'll just give them a fresh try (again, on UCD farm). My being able to try again there should control for some loose variables.

[ctb]

quick question: what does echo $PYTHONPATH show in your hands? Not urgent, but it could help explain the behavior you're seeing.

[SchwarzEM]

In my hands, echo $PYTHONPATH shows no directories -- either before, or after, I run mamba activate charcoal.

[ctb]

Finally fixed enough tests that I recapitulated your error over in #239! Evaluating a potential fix now.

[SchwarzEM]

Good!

(At any rate, good that my error can finally be reproduced ... progress!)

[quliping]

Finally fixed enough tests that I recapitulated your error over in #239! Evaluating a potential fix now.
Unfortunately I got the same problem. Has this issue been resolved now?

[SchwarzEM]

@quliping: I have not tried working with this since my initial tickets in late October; but if there emerges a version of charcoal that can be installed with mamba and will Just Work, I will be happy to try installing it! My impression from your post 2 days ago is that today is not that day. ;^] But please correct me if and when I'm wrong!