Skip to content
This repository was archived by the owner on Apr 25, 2023. It is now read-only.

Commit c70177d

Browse files
astrojuanluastrojuanlu
authored
Merge pull request #54 from Juanlu001/third-session
Add session on conda vs pip
2 parents f13f261 + 83d236d commit c70177d

File tree

3 files changed

+273
-0
lines changed

3 files changed

+273
-0
lines changed

‎img/conda-forge.png

6.19 KB
Loading

‎img/python_comrades.png

270 KB
Loading

‎sessions/03_pip-vs-conda.ipynb

Lines changed: 273 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,273 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {
6+
"slideshow": {
7+
"slide_type": "slide"
8+
}
9+
},
10+
"source": [
11+
"![IE](../img/ie.png)\n",
12+
"\n",
13+
"# Sessions 4 & 5: pip vs conda\n",
14+
"\n",
15+
"### Juan Luis Cano Rodríguez <jcano@faculty.ie.edu> - Master in Business Analytics and Big Data (2019-04-05)"
16+
]
17+
},
18+
{
19+
"cell_type": "markdown",
20+
"metadata": {},
21+
"source": [
22+
"## Managing Python environments\n",
23+
"\n",
24+
"![Python Comrades](../img/python_comrades.png)\n",
25+
"\n",
26+
"Bootstrapping a working Python installation in a future-proof way can be tough. Linux distributions carry a ([sometimes crippled](https://github.com/pypa/pip/issues/4222#issuecomment-417646535)) Python _that should **never** be used for development_, OS X does the same thing with even older version, and getting the `%PATH%` to work on Windows is not exactly newcomers-friendly.\n",
27+
"\n",
28+
"Furthermore, even with a correct Python installation (let's assume Python >= 3.4 for sanity), the native [environment creation utility](https://docs.python.org/3/library/venv.html) has two problems:\n",
29+
"\n",
30+
"1. It's tied to the originating Python version (e.g. if I install Python 3.6, I can't create a Python 3.7 environment)\n",
31+
"2. For packages that require compiled extensions (mostly scientific/data packages, but not only), sometimes admin intervention is needed to install some system dependency\n",
32+
"\n",
33+
"There are three popular solutions for these problems nowadays:\n",
34+
"\n",
35+
"* Use [pyenv](https://github.com/pyenv/pyenv/) to manage Python environments. It solves (1) by allowing several Python versions, and doesn't handle (2). The main advantage is that several Python interpreters can be installed seamlessly.\n",
36+
"* Use [Docker](https://docs.docker.com/engine/). It solves (1) and (2) by giving you _an entire operating system_, which is a bit overkill, and some people claim [it's optimized for deployment, not development](https://github.com/moby/moby/issues/7198#issuecomment-230965019). The main advantage is its total flexibility.\n",
37+
"* **Use [conda](https://conda.io/)**. It solves (1) and (2) by providing a somewhat language-agnostic package and environment manager that does not require admin privileges. We will use this solution, even though it requires some care when mixing it with pip and its performance is (at the time of writing) not very good.\n",
38+
"\n",
39+
"### Summary\n",
40+
"\n",
41+
"> For the user, the most salient distinction is probably this: pip installs _python_ packages within _any_ environment; conda installs _any_ package within _conda_ environments\n",
42+
">\n",
43+
"> —Jake Vanderplas\n",
44+
"\n",
45+
"To mix the best of both worlds and minimize risk, our (default) approach will be\n",
46+
"\n",
47+
"* Use **conda** to _manage environments_ and _install non-Python dependencies_\n",
48+
"* Use (recent versions of) **pip** inside conda environments to install Python dependencies\n",
49+
"\n",
50+
"This way we will:\n",
51+
"\n",
52+
"* Avoid the cumbersome multi-step process of pyenv\n",
53+
"* Avoid incompatibilities between conda and pip\n",
54+
"* Avoid the performance issues of conda while leveraging the improved dependency handling of modern pip (>= 19.0.3) \n",
55+
"\n",
56+
"### References\n",
57+
"\n",
58+
"* Conda: Myths and Misconceptions https://jakevdp.github.io/blog/2016/08/25/conda-myths-and-misconceptions/\n",
59+
"* Using pip in a conda environment https://www.anaconda.com/using-pip-in-a-conda-environment/"
60+
]
61+
},
62+
{
63+
"cell_type": "markdown",
64+
"metadata": {},
65+
"source": [
66+
"## conda and conda-forge\n",
67+
"\n",
68+
"### Installation\n",
69+
"\n",
70+
"1. Download Miniconda3 https://docs.conda.io/en/latest/miniconda.html (version is not important, _but_ ignore Python 2 😉)\n",
71+
"2. Accept the license terms ([BSD 3-clause](https://tldrlegal.com/license/bsd-3-clause-license-(revised)))\n",
72+
"3. Specify a location (recommendation: `/home/userXX/.miniconda37`, although the default one is fine)\n",
73+
"4. Do **NOT** initialize Miniconda3 in `.bashrc` (we will learn and understand how it works instead)\n",
74+
"\n",
75+
"If we follow the steps correctly we will see this message and we _won't_ be able to run conda, yet:\n",
76+
"\n",
77+
"```\n",
78+
"...\n",
79+
"installation finished.\n",
80+
"Do you wish the installer to initialize Miniconda3\n",
81+
"in your /home/user00/.bashrc ? [yes|no]\n",
82+
"[no] >>> no\n",
83+
"\n",
84+
"You may wish to edit your /home/user00/.bashrc to setup Miniconda3:\n",
85+
"\n",
86+
"source /home/user00/miniconda3/etc/profile.d/conda.sh\n",
87+
"\n",
88+
"Thank you for installing Miniconda3!\n",
89+
"user00@ns3003537:~$ conda --version\n",
90+
"conda: command not found\n",
91+
"```\n",
92+
"\n",
93+
"Notice though that Miniconda3 was correctly installed!\n",
94+
"\n",
95+
"```\n",
96+
"user00@ns3003537:~$ ~/miniconda3/bin/conda --version\n",
97+
"conda 4.5.12\n",
98+
"```\n",
99+
"\n",
100+
"To properly initialize conda and avoid having to use the full path every time, we can do:\n",
101+
"\n",
102+
"```\n",
103+
"user00@ns3003537:~$ source /home/user00/miniconda3/etc/profile.d/conda.sh\n",
104+
"user00@ns3003537:~$ conda --version\n",
105+
"conda 4.5.12\n",
106+
"```\n",
107+
"\n",
108+
"And, to make it permanent, add it to our `.bashrc`:\n",
109+
"\n",
110+
"```\n",
111+
"user00@ns3003537:~$ echo 'source /home/user00/miniconda3/etc/profile.d/conda.sh' >> .bashrc\n",
112+
"user00@ns3003537:~$\n",
113+
"```\n",
114+
"\n",
115+
"<div class=\"alert alert-warning\">Depending on how the machine was configured, the <code>~/.bashrc</code> trick might not work. There should be a <code>~/.profile</code> or <code>~/.bash_profile</code> telling the system to load it on login.</div>"
116+
]
117+
},
118+
{
119+
"cell_type": "markdown",
120+
"metadata": {},
121+
"source": [
122+
"### Basic usage\n",
123+
"\n",
124+
"Even if we initialized conda already (remember: **not** on installation, but right after) we won't have access to the Python that comes with it:\n",
125+
"\n",
126+
"```\n",
127+
"user00@ns3003537:~$ which python\n",
128+
"user00@ns3003537:~$ which python3\n",
129+
"/usr/bin/python3\n",
130+
"```\n",
131+
"\n",
132+
"The reason is that we have to **activate an environment**. conda comes with an environment called `base` by default, and we can activate it using `conda activate <environment>`:\n",
133+
"\n",
134+
"```\n",
135+
"user00@ns3003537:~$ conda info -e # Lists environments\n",
136+
"# conda environments:\n",
137+
"#\n",
138+
"base * /home/user00/miniconda3\n",
139+
"\n",
140+
"user00@ns3003537:~$ conda activate base # Notice the prompt change!\n",
141+
"(base) user00@ns3003537:~$ which python\n",
142+
"/home/user00/miniconda3/bin/python\n",
143+
"(base) user00@ns3003537:~$ which python3\n",
144+
"/home/user00/miniconda3/bin/python3\n",
145+
"(base) user00@ns3003537:~$ which pip\n",
146+
"/home/user00/miniconda3/bin/pip\n",
147+
"(base) user00@ns3003537:~$ echo $PATH # Here's the trick!\n",
148+
"/home/user00/miniconda3/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/snap/bin\n",
149+
"```\n",
150+
"\n",
151+
"And we can go back to \"normal\" using `conda deactivate`:\n",
152+
"\n",
153+
"```\n",
154+
"(base) user00@ns3003537:~$ conda deactivate\n",
155+
"user00@ns3003537:~$ echo $PATH\n",
156+
"/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/snap/bin\n",
157+
"```\n",
158+
"\n",
159+
"<div class=\"alert alert-warning\"><strong>Note:</strong> Avoid using the <code>base</code> environment: you will pollute it, and eventually break it. Instead, get used to creating one environment per project.</div>"
160+
]
161+
},
162+
{
163+
"cell_type": "markdown",
164+
"metadata": {},
165+
"source": [
166+
"### Environment creation\n",
167+
"\n",
168+
"To create an environment we use `conda create --name <name> <list-of-packages>`. We don't need to specify all the packages we will need, but it's customary to set the Python version, and sometimes also NumPy ([if you want extra performance](https://www.anaconda.com/tensorflow-in-anaconda/)).\n",
169+
"\n",
170+
"For example, to create a environment for our ie-nlp-utils project using Python 3.7:\n",
171+
"\n",
172+
"```\n",
173+
"user00@ns3003537:~$ conda create -n nlp37 python=3.7\n",
174+
"Solving environment: /\n",
175+
"...\n",
176+
"Proceed ([y]/n)? y\n",
177+
"...\n",
178+
"Preparing transaction: done\n",
179+
"Verifying transaction: done\n",
180+
"Executing transaction: done\n",
181+
"#\n",
182+
"# To activate this environment, use\n",
183+
"#\n",
184+
"# $ conda activate nlp37\n",
185+
"#\n",
186+
"# To deactivate an active environment, use\n",
187+
"#\n",
188+
"# $ conda deactivate\n",
189+
"\n",
190+
"user00@ns3003537:~$ conda activate nlp37\n",
191+
"(nlp37) user00@ns3003537:~$ \n",
192+
"```"
193+
]
194+
},
195+
{
196+
"cell_type": "markdown",
197+
"metadata": {},
198+
"source": [
199+
"### Channels\n",
200+
"\n",
201+
"![conda-forge](../img/conda-forge.png)\n",
202+
"\n",
203+
"Anaconda (formerly Continuum Analytics), the company behind the Anaconda product and conda, uploads all conda packages to [their main repository](https://repo.anaconda.com/), so `conda` knows where to look for them. However, there is a [_community repository_](https://anaconda.org/) as well, where anyone can upload any packages.\n",
204+
"\n",
205+
"To decide where to download the packages from, `conda` has the concept of **channels**. The `defaults` channel is implicit, but we can decide to install a specific package from a specific channel. The most imporant of these channels (and almost the only one you should care about) is **conda-forge**:\n",
206+
"\n",
207+
"https://conda-forge.org/\n",
208+
"\n",
209+
"> A community led collection of recipes, build infrastructure and distributions for the conda package manager.\n",
210+
"\n",
211+
"The `defaults` channel does not have _all_ the packages available out there, and also doesn't usually have the latest versions. The reason is that they are a bit more conservative to please corporate users.\n",
212+
"\n",
213+
"To install a package from conda-forge, we can especify the channel in two ways:\n",
214+
"\n",
215+
"* `$ conda install numpy --channel conda-forge` (or `-c conda-forge`)\n",
216+
"* `$ conda install conda-forge::numpy`\n",
217+
"\n",
218+
"To configure `conda` to use `conda-forge` first:\n",
219+
"\n",
220+
"```\n",
221+
"user00@ns3003537:~$ conda config --prepend channels conda-forge\n",
222+
"user00@ns3003537:~$ cat ~/.condarc\n",
223+
"channels:\n",
224+
" - conda-forge\n",
225+
" - defaults\n",
226+
"```\n",
227+
"\n",
228+
"(See [the tips and tricks](http://conda-forge.org/docs/user/tipsandtricks.html) section of conda-forge documentation for more information)"
229+
]
230+
},
231+
{
232+
"cell_type": "markdown",
233+
"metadata": {},
234+
"source": [
235+
"## pip and PyPI\n",
236+
"\n",
237+
"pip is the default Python installer. By default, it fetches packages from https://pypi.org/, which is the community repository for Python packages. As its Anaconda counterpart, it's not curated so anyone can upload anything - however, the concept of channels doesn't exist, so **there can't be name clashes**.\n",
238+
"\n",
239+
"Several considerations must be taken into account while using `pip`:\n",
240+
"\n",
241+
"* **Never, ever use `sudo pip install`**. You will break your system in very ugly ways. Create a conda environment instead.\n",
242+
"* Check the pip version. The latest releases were:\n",
243+
" - 19.x (optimal)\n",
244+
" - 18.x\n",
245+
" - 10.x (yes, they switched to [calendar versioning](http://calver.org/) right after)\n",
246+
" - 9.x (between \"old\" and \"very old\")\n",
247+
" - <8.x (avoid like the plague!)\n",
248+
"* As a general rule, _don't upgrade straight away_ - the developers iron the issues after each release"
249+
]
250+
}
251+
],
252+
"metadata": {
253+
"kernelspec": {
254+
"display_name": "Python 3",
255+
"language": "python",
256+
"name": "python3"
257+
},
258+
"language_info": {
259+
"codemirror_mode": {
260+
"name": "ipython",
261+
"version": 3
262+
},
263+
"file_extension": ".py",
264+
"mimetype": "text/x-python",
265+
"name": "python",
266+
"nbconvert_exporter": "python",
267+
"pygments_lexer": "ipython3",
268+
"version": "3.7.1"
269+
}
270+
},
271+
"nbformat": 4,
272+
"nbformat_minor": 2
273+
}

0 commit comments

Comments
 (0)