Microsoft OneDrive overriding file download locations

[ChrisMantegna]

Has anyone used our regular programs/platforms (git, jupyter, blast) with success while maintaining files in OneDrive?

I have a Microsoft windows computer and use OneDrive to save files. It's a newer computer that will not allow me to pick and choose which folders not to sync (as has been the case in the past). I thought I fixed the duplicate Desktop, Documents, Download file locations, but I haven't. The result is toggling directories when I use command line and that is a disaster...
My concern is that I'll have to completely uninstall OneDrive as "pause sync" is temporary.

Please & Thank You :)

[kubu4]

Sorry for the delay in responding to this.

The result is toggling directories when I use command line

I'm not totally clear on what you mean by this. Can you please describe your workflow a bit more? Specifically, how you're using Windows Subsystem for Linux (WSL; Ubuntu) along with the Windows file system?

With that said, one solution to this is to avoid saving files in a location monitored by OneDrive. Instead, utilize GitHub (along with git) to backup files/data. And, since you're using WSL, I don't think OneDrive would monitor any directories found in WSL (unless you tell it to).

Create a GitHub repo for whatever analysis/project your working on in a location in your WSL and organize it. E.g.:

• data directory
• analysis directory
• scripts directory
• notebooks directory

Then, work within the repo and sync to GitHub using git.

This would allow you to avoid OneDrive entirely.

Anyway, fell free to post any more questions/problems you might have/encounter.

Also, this might be easier to walk through via Zoom. If you're available for Science Hour this week, that would be a good time for us to look at this together.

[ChrisMantegna]

Thanks - I've tried through Git but I must be doing something wrong. The science hour would be perfect for this.
Thanks

[sr320]

@ChrisMantegna to confirm are you in fact using WSL?

[kubu4]

@ChrisMantegna - Any luck with this? Sounds like Science Hour assessment was that you should install WSL. My apologies earlier, for some reason I thought you had installed this previously (vague recollection of talking about this aspect of things awhile ago; looks like I was wrong).

Here are instructions for installing WSL:

https://docs.microsoft.com/en-us/windows/wsl/install-win10

I'd recommend following the instructions for "Simplified install".

At the step to "open a command line as Adminstrator", using the Windows search box, search for PowerShell and then "Run as Administrator":

Let us know how it goes!

[ChrisMantegna]

@kubu4 @mattgeorgephd
I'm not 100% sure what you need to be able to help, so please let me know what else you need. I have installed Ubuntu and WSL/ Powershell.
Next I went to this [website] (https://www.digitalocean.com/community/tutorials/how-to-set-up-jupyter-notebook-with-python-3-on-ubuntu-18-04) and installed Jupyter and followed the steps to create my virtual environment. The SSH Tunneling did not work.
Question 1 - should I be opening Jupyter directly from the command line (picture 2) or from the virtual environment (picture 1)? Only difference are the commands to get to the same place.

Question 2- where should i be installing blast? I have it in no less than 4 places, none of which are correct. (picture 3).

[kubu4]

Thanks for posting all of that.

1. I would create a directory structure in WSL. Run the following commands in WSL:

cd
mkdir programs data analyses scripts

That will change to your home directory and then make directories called programs, data, analyses, and scripts.

2. Change to the programs directory, download and install miniconda:

cd programs
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh

Follow the prompts on the installer screens.

You'll probably want to change the install location. Set it to:

• /home/chris/programs/miniconda

If you are unsure about any setting, accept the defaults. You can change them later.

To make the changes take effect, close and then re-open your terminal window.

3. Install mamba (a faster package manager for Anaconda stuff):

conda install mamba -n base -c conda-forge

4. Install Jupyter Lab (fancier version of Jupyter Notebook):

mamba install -c conda-forge jupyterlab

5. Close/re-open Ubuntu WSL to have changes take effect.

6. Start Jupyter Notebook with the following command:

jupyter-lab

---

Now, on to the BLAST issues. The issue you're encountering is that you're trying to run the Windows BLAST program from Ubuntu. You should think of your Ubuntu WSL as a totally separate computer with a totally different operating system (because, it sort of is). So, here're the instructions for getting BLAST set up in Ubuntu WSL:

1. Download and install BLAST:

cd ~/programs
https://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/LATEST/ncbi-blast-2.12.0+-x64-linux.tar.gz
tar -xzvf ncbi-blast-2.12.0+-x64-linux.tar.gz

2. Test BLAST install:

~/programs/ncbi-blast-2.12.0+/bin/blastx -h

The above should print out the expected BLASTx help menu.

So, going forward, that's the path you'll use to access BLAST programs.

---

Give it all a whirl and let me know how things work out!

[ChrisMantegna]

I am now in syntax jail. I spaced the commands out so you could see the error locations - the "// " for the first line and lack of "/ " in the second. This occurs with and without the " !curl " command.
When run, if I put "/ " in front of ncbi - blast... the error then moves down the text thread to the period at 12.0+-... not the first period in that string.

[mattgeorgephd]

@ChrisMantegna, Sam's comments are right on - I'd follow through with his suggestions and then see where you are at. In reference to question 1, definitely go with Jupyter lab. It can be installed using conda as Sam mentioned. After that, you should be able to launch Jupyter lab in Ubuntu this way:

and copy and pasting the link in your browser of choice.

[kubu4]

@ChrisMantegna - Alrighty, there are a few things going on here, but first we'll focus on getting BLAST installed.

All instructions posted below should be done in Ubuntu WSL, not in a Jupyter Notebook (we'll tackle this and the nuances of Jupyter Notebooks in a bit). Reviewing my instructions above, I clearly see my mistake; sorry! I also noticed a critical omission (bad copy/paste job on my part) that would screw things up; sorry again!

1. Download and install BLAST:

cd ~/programs
wget https://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/LATEST/ncbi-blast-2.12.0+-x64-linux.tar.gz
tar -xzvf ncbi-blast-2.12.0+-x64-linux.tar.gz

2. Test BLAST install:

~/programs/ncbi-blast-2.12.0+/bin/blastx -h

The above should print out the expected BLASTx help menu.

After this is set, we'll tackle some of the Jupyter Notebook idiosyncracies!

[ChrisMantegna]

No apologies - when I'm unsure I read instructions a bit too literally!
I'm going to assume db setup is also in WSL so I'm going to do that.

[ChrisMantegna]

@kubu4
This is my newest hurdle. I was able to get my query file and my db file. When I went to actually run blast it told me nothing existed. I erased (rm -r) the files and verified I was putting everything in the correct place and re-retrieved. Now I cannot make the db...

[kubu4]

To unzip the uniprot_sprot.fasta.gz file you need to use this command:

gunzip uniprot_sprot.fasta.gz

[ChrisMantegna]

Alright @kubu4 ... there has been some success, but not completely. Here is what the powershell looks like and then the file in jupyter notebook. The first go around this file looked like 8-10 rows of the same kind of thing as you see below in jupyter.

[kubu4]

What output were you expecting (wanting)?

Looking at your command (and the resulting output file), everything looks totally normal to me.

[kubu4]

And, I can explain in more detail why you ended up with what you ended up with, but I'd like to wait to find out what you were expecting/wanting so I can address both things at the same time.

[ChrisMantegna]

I was expecting something much more I guess, or at least a longer list.
Also, I don't actually know when it is finished - is there a way to see the progress? Each time I've done this I've had to force the prompt line back just to see if it actually worked.

[kubu4]

In regards to output quantity, there are many variables involved, so it's going to depend on the number of query (input) sequences, the number of query (database) sequences, BLAST stringency (e. g. max target seqs, max hsps, e-value cutoff, etc).

Also, this takes time. So, if you waitd longer you'd likely have more output.

In regards to runtime, if your command prompt has not returned (like in your case), it means things are still running. You can also check this by opening an additional terminal and running top from that terminal. You can then look at the screen and see which programs are running and actively using CPU and memory.

As far as speeding things up, your only options are to use a different BLAST program (DIAMOND BLAST) which will take minutes instead of hours on your laptop, or you can run the commands remotely on a more powerful computer. If you opt for the former, you'll need to create a DIAMOND BLAST database from your SwissProt FastA. From there, I think the BLASTx command is identical to what you ran before (but you'll need to call DIAMOND BLASTx instead of NCBI BLASTx).

[ChrisMantegna]

Success with diamond blast! Thank you @kubu4