diff --git a/README.md b/README.md index 7f77141b..25c723e8 100644 --- a/README.md +++ b/README.md @@ -6,63 +6,75 @@ PyJulia Experimenting with developing a better interface to [Julia language](https://julialang.org/) that works with [Python](https://www.python.org/) 2 & 3 and Julia v0.6+. -to run the tests, execute from the toplevel directory - -```shell -tox -``` - -See [Testing](#testing) below for details. - -**Note** You need to explicitly add julia to your `PATH`, an alias will not work. - -`pyjulia` is tested against Python versions 2.7, 3.6, and 3.7. Older versions of Python (than 2.7) are not supported. +PyJulia is tested against Python versions 2.7, 3.6, and 3.7. Older versions of Python (than 2.7) are not supported. Installation ------------ + +**Note:** If you are using Python installed with Ubuntu or `conda`, +PyJulia may not work with Julia ≥ 0.7. For workarounds, see +[Troubleshooting](#troubleshooting) below. Same caution applies to +any Debian-based and possibly other GNU/Linux distributions. + You will need to install PyCall in your existing Julia installation ```julia -Pkg.add("PyCall") +julia> using Pkg # for julia ≥ 0.7 +julia> Pkg.add("PyCall") ``` -Your python installation must be able to call Julia. If your installer -does not add the Julia binary directory to your `PATH`, you will have to -add it. +Your python installation must be able to call command line program +`julia`. If your installer does not add the Julia binary directory to +your `PATH`, you will have to add it. _An alias will not work._ -Then finally you have to install pyjulia. +Then finally you have to install PyJulia. + +**Note:** If you are not familiar with `pip` and have some troubles +with the following installation steps, we recommend going through the +[Tutorials in Python Packaging User Guide](https://packaging.python.org/tutorials/). To get released versions you can use: -``` -pip install julia +```console +$ python3 -m pip install --user julia +$ python2 -m pip install --user julia # If you need Python 2 ``` -You may clone it directly to your home directory. +where `--user` should be omitted if you are using virtual environment +(`virtualenv`, `venv`, `conda`, etc.). -``` -git clone https://github.com/JuliaPy/pyjulia +If you are interested in using the development version, you can +install PyJulia directly from GitHub: +```console +$ python3 -m pip install --user 'https://github.com/JuliaPy/pyjulia/archive/master.zip#egg=julia' ``` -then inside the pyjulia directory you need to run the python setup file +You may clone it directly to (say) your home directory. + +```console +$ git clone https://github.com/JuliaPy/pyjulia ``` -[sudo] pip install [-e] . + +then inside the `pyjulia` directory you need to run the python setup file + +```console +$ cd pyjulia +$ python3 -m pip install --user . +$ python3 -m pip install --user -e . # If you want "development install" ``` -The `-e` flag makes a development install meaning that any change to pyjulia +The `-e` flag makes a development install, meaning that any change to PyJulia source tree will take effect at next python interpreter restart without having to reissue an install command. -`pyjulia` is known to work with `PyCall.jl` ≥ `v0.7.2`. - -If you run into problems using `pyjulia`, first check the version of `PyCall.jl` you have installed by running `Pkg.installed("PyCall")`. +See [Testing](#testing) below for how to run tests. Usage ----- -`pyjulia` provides a high-level interface which assumes a "normal" -setup (e.g., `julia` is in your `PATH`) and a low-level interface +PyJulia provides a high-level interface which assumes a "normal" setup +(e.g., `julia` program is in your `PATH`) and a low-level interface which can be used in a customized setup. ### High-level interface @@ -70,61 +82,182 @@ which can be used in a customized setup. To call a Julia function in a Julia module, import the Julia module (say `Base`) with: -```python -from julia import Base +```pycon +>>> from julia import Base ``` and then call Julia functions in `Base` from python, e.g., -```python -Base.sind(90) +```pycon +>>> Base.sind(90) ``` Other variants of Python import syntax also work: -```python -import julia.Base -from julia.Base import LinAlg # import a submodule -from julia.Base import sin # import a function from a module +```pycon +>>> import julia.Base +>>> from julia.Base import Enums # import a submodule +>>> from julia.Base import sin # import a function from a module ``` The global namespace of Julia's interpreter can be accessed via a special module `julia.Main`: -```python -from julia import Main +```pycon +>>> from julia import Main ``` You can set names in this module to send Python values to Julia: -```python -Main.xs = [1, 2, 3] +```pycon +>>> Main.xs = [1, 2, 3] ``` which allows it to be accessed directly from Julia code, e.g., it can be evaluated at Julia side using Julia syntax: -```python -Main.eval("sin.(xs)") +```pycon +>>> Main.eval("sin.(xs)") ``` ### Low-level interface -If you need a custom setup for `pyjulia`, it must be done *before* +If you need a custom setup for PyJulia, it must be done *before* importing any Julia modules. For example, to use the Julia executable named `custom_julia`, run: -```python -from julia import Julia -jl = julia.Julia(runtime="custom_julia") +```pycon +>>> from julia import Julia +>>> jl = julia.Julia(runtime="custom_julia") ``` You can then use, e.g., -```python -from julia import Base +```pycon +>>> from julia import Base ``` +### IPython magic + +In IPython (and therefore in Jupyter), you can directly execute Julia +code using `%%julia` magic: + +``` +In [1]: %load_ext julia.magic +Initializing Julia interpreter. This may take some time... + +In [2]: %%julia + ...: Base.banner(IOContext(stdout, :color=>true)) + _ + _ _ _(_)_ | Documentation: https://docs.julialang.org + (_) | (_) (_) | + _ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help. + | | | | | | |/ _` | | + | | |_| | | | (_| | | Version 1.0.1 (2018-09-29) + _/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release +|__/ | +``` + +### Virtual environments + +PyJulia can be used in Python virtual environments created by +`virtualenv`, `venv`, and any tools wrapping them such as `pipenv`, +provided that Python executable used in such environments are linked +to identical libpython used by PyCall. If this is not the case, +initializing PyJulia (e.g., `import julia.Main`) prints an informative +error message with detected paths to libjulia. See +[PyCall documentation](https://github.com/JuliaPy/PyCall.jl) for how +to configure Python executable. + +Note that Python environment created by `conda` is not supported. + +Troubleshooting +--------------- + +### Your Python interpreter is statically linked to libpython + +If you use Python installed with Debian-based Linux distribution such +as Ubuntu or install Python by `conda`, you might have noticed that +PyJulia cannot be initialized properly with Julia ≥ 0.7. This is +because those Python executables are statically linked to libpython. +(See [Limitations](#limitations) below for why that's a problem.) + +If you are unsure if your `python` has this problem, you can quickly +check it by: + +```console +$ ldd /usr/bin/python + linux-vdso.so.1 (0x00007ffd73f7c000) + libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f10ef84e000) + libc.so.6 => /usr/lib/libc.so.6 (0x00007f10ef68a000) + libpython3.7m.so.1.0 => /usr/lib/libpython3.7m.so.1.0 (0x00007f10ef116000) + /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f10efaa4000) + libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f10ef111000) + libutil.so.1 => /usr/lib/libutil.so.1 (0x00007f10ef10c000) + libm.so.6 => /usr/lib/libm.so.6 (0x00007f10eef87000) +``` + +in Linux where `/usr/bin/python` should be replaced with the path to +your `python` command (use `which python` to find it out). In macOS, +use `otool -L` instead of `ldd`. If it does not print the path to +libpython like `/usr/lib/libpython3.7m.so.1.0` in above example, you +need to use one of the workaround below. + +The easiest workaround is to use the `python-jl` command bundled in +PyJulia. This can be used instead of normal `python` command for +basic use-cases such as: + +```console +$ python-jl your_script.py +$ python-jl -c 'from julia.Base import banner; banner()' +$ python-jl -m IPython +``` + +See `python-jl --help` for more information. + +Note that `python-jl` works by launching Python interpreter inside +Julia. If you are comfortable with working in Julia REPL, you can +simply do: + +```julia +julia> using PyCall + +julia> py""" + from julia import Julia + Julia(init_julia=False) + + from your_module_using_pyjulia import function + function() + """ +``` + +Alternatively, you can use [pyenv](https://github.com/pyenv/pyenv) to +build +[`--enable-shared` option](https://github.com/pyenv/pyenv/wiki#how-to-build-cpython-with---enable-shared). +Of course, manually building from Python source distribution with the +same configuration also works. + +```console +$ PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.6.6 +Downloading Python-3.6.6.tar.xz... +-> https://www.python.org/ftp/python/3.6.6/Python-3.6.6.tar.xz +Installing Python-3.6.6... +Installed Python-3.6.6 to /home/USER/.pyenv/versions/3.6.6 + +$ ldd ~/.pyenv/versions/3.6.6/bin/python3.6 | grep libpython + libpython3.6m.so.1.0 => /home/USER/.pyenv/versions/3.6.6/lib/libpython3.6m.so.1.0 (0x00007fca44c8b000) +``` + +For more discussion, see: +https://github.com/JuliaPy/pyjulia/issues/185 + +### Segmentation fault in IPython + +You may experience segmentation fault when using PyJulia in old +versions of IPython. You can avoid this issue by updating IPython to +7.0 or above. Alternatively, you can use IPython via Jupyter (e.g., +`jupyter console`) to workaround the problem. + How it works ------------ @@ -139,12 +272,86 @@ when reference count drops to zero, so that the Julia object may be freed). Limitations ------------ -Not all valid Julia identifiers are valid Python identifiers. Unicode identifiers are invalid in Python 2.7 and so `pyjulia` cannot call or access Julia methods/variables with names that are not ASCII only. Additionally, it is a common idiom in Julia to append a `!` character to methods which mutate their arguments. These method names are invalid Python identifers. `pyjulia` renames these methods by subsituting `!` with `_b`. For example, the Julia method `sum!` can be called in `pyjulia` using `sum_b(...)`. +### Mismatch in valid set of identifiers + +Not all valid Julia identifiers are valid Python identifiers. Unicode +identifiers are invalid in Python 2.7 and so PyJulia cannot call or +access Julia methods/variables with names that are not ASCII only. +Although Python 3 allows Unicode identifiers, they are more +aggressively normalized than Julia. For example, `ϵ` (GREEK LUNATE +EPSILON SYMBOL) and `ε` (GREEK SMALL LETTER EPSILON) are identical in +Python 3 but different in Julia. Additionally, it is a common idiom +in Julia to append a `!` character to methods which mutate their +arguments. These method names are invalid Python identifers. +PyJulia renames these methods by subsituting `!` with `_b`. For +example, the Julia method `sum!` can be called in PyJulia using +`sum_b(...)`. + +### Pre-compilation mechanism in Julia 1.0 + +There was a major overhaul in the module loading system between Julia +0.6 and 1.0. As a result, the "hack" supporting the PyJulia to load +PyCall stopped working. For the implementation detail of the hack, +see: https://github.com/JuliaPy/pyjulia/tree/master/julia/fake-julia + +For the update on this problem, see: +https://github.com/JuliaLang/julia/issues/28518 + +### Ctrl-C does not work / terminates the whole Python process + +Currently, initializing PyJulia (e.g., by `from julia import Main`) +disables `KeyboardInterrupt` handling in the Python process. If you +are using normal `python` interpreter, it means that canceling the +input by Ctrl-C does not work and repeatedly providing +Ctrl-C terminates the whole Python process with the error +message `WARNING: Force throwing a SIGINT`. Using IPython 7.0 or +above is recommended to avoid such accidental shutdown. + +It also means that there is no safe way to cancel long-running +computations or I/O at the moment. Sending SIGINT with +Ctrl-C will terminate the whole Python process. + +For the update on this problem, see: +https://github.com/JuliaPy/pyjulia/issues/211 + +### No threading support + +PyJulia cannot be used in different threads since libjulia is not +thread safe. However, you can +[use multiple threads within Julia](https://docs.julialang.org/en/v1.0/manual/parallel-computing/#Multi-Threading-(Experimental)-1). +For example, start IPython by `JULIA_NUM_THREADS=4 ipython` and then +run: + +```julia +In [1]: %load_ext julia.magic +Initializing Julia interpreter. This may take some time... + +In [2]: %%julia + ...: a = zeros(10) + ...: Threads.@threads for i = 1:10 + ...: a[i] = Threads.threadid() + ...: end + ...: a +Out[3]: array([1., 1., 1., 2., 2., 2., 3., 3., 4., 4.]) +``` + +### PyJulia does not release GIL + +PyJulia does not release the Global Interpreter Lock (GIL) while +calling Julia functions since PyCall expects the GIL to be acquired +always. It means that Python code and Julia code cannot run in +parallel. Testing ------- +PyJulia can be tested by simply running [`tox`](http://tox.readthedocs.io): + +```console +$ tox +``` + The full syntax for invoking `tox` is ```shell @@ -165,13 +372,13 @@ The full syntax for invoking `tox` is For example, -```shell -PYJULIA_TEST_REBUILD=yes JULIA_EXE=~/julia/julia tox -e py37 -- -s +```console +$ PYJULIA_TEST_REBUILD=yes JULIA_EXE=~/julia/julia tox -e py37 -- -s ``` means to execute tests with -* `pyjulia` in shared-cache mode +* PyJulia in shared-cache mode * `julia` executable at `~/julia/julia` * Python 3.7 * `pytest`'s capturing mode turned off diff --git a/julia/core.py b/julia/core.py index 2f946910..4cbf6226 100644 --- a/julia/core.py +++ b/julia/core.py @@ -407,7 +407,7 @@ def is_compatible_exe(jlinfo, _debug=lambda *_: None): See `python-jl --help` for more information. For other available workarounds, see: - https://github.com/JuliaPy/pyjulia/issues/185 + https://github.com/JuliaPy/pyjulia#troubleshooting """ diff --git a/julia/fake-julia/README b/julia/fake-julia/README deleted file mode 100644 index 604381ee..00000000 --- a/julia/fake-julia/README +++ /dev/null @@ -1,37 +0,0 @@ -This directory contains a python script that pretends to be the julia executable -and is used as such to allow julia precompilation to happen in the same environment. - -When a Julia module Foo marked with `__precompile__(true)` is imported in Julia, it gets "precompiled" to -a Foo.ji cache file that speeds up subsequent loads. See: - https://docs.julialang.org/en/stable/manual/modules/#Module-initialization-and-precompilation-1 -A key thing to understand is that this precompilation works by *launching a new Julia process* -that loads the module in a special "output-ji" mode (by running `julia --output-ji`) that creates -the cache file. - -A second key thing to understand is that pyjulia is using PyCall configured in a different way than -when PyCall is called from with a `julia` process. Within a `julia` process, PyCall works by loading -`libpython` to call the CPython API. Within a `python` process (for `pyjulia`), at least if -`python` is statically linked to `libpython`, PyCall works instead by loading CPython API symbols from -the `python` process itself. This difference affects how PyCall functions are compiled, which means -that *pyjulia cannot use the same PyCall.ji cache file* as julia. This extends to any Julia module -*using* PyCall: every such module needs to have a precompiled cache file that is different from the ordinary -Julia module cache. - -The combination of these two facts mean that when PyCall, or any Julia module that uses PyCall, -is loaded from pyjulia with a statically linked `python`, we have to precompile a separate version of it. -Since "normal" precompilation launches a new `julia` process, this process would create the wrong -(`libpython`) version of the PyCall cache file. So, we have to force precompilation to launch -a `python` process, not a `julia` process, so that PyCall is compiled correctly for running inside `python`. - -That is what `fake-julia` does. By changing the `JULIA_HOME` (v0.6) or `JULIA_BINDIR` (v0.7+) environment variable, we trick Julia -into launching `fake-julia/julia` instead of the "real" `julia` process during precompilation. `fake-julia/julia` -is actually a Python script, but it links `libjulia` and uses `libjulia` to process the command-line arguments, -so it mimics the behavior of the `julia` process. Since `fake-julia/julia` is running from within the `python` -process, PyCall configures itself correctly. - -(From the above discussion, it should be clear that the fake-julia trick is only really necessary for -compiling PyCall and other Julia modules that use PyCall. For other Julia modules, the compiled code -should be identical to the normal Julia cache, so as an optimization `fake-julia/julia` shares the same cache -file with the real `julia` in that case.) - -See also the discussion in https://github.com/JuliaPy/PyCall.jl/pull/293 and https://github.com/JuliaPy/pyjulia/pull/54 diff --git a/julia/fake-julia/README.md b/julia/fake-julia/README.md new file mode 100644 index 00000000..95cf653a --- /dev/null +++ b/julia/fake-julia/README.md @@ -0,0 +1,48 @@ +This directory contains a python script that pretends to be the julia executable +and is used as such to allow julia precompilation to happen in the same environment. + +When a Julia module `Foo` is imported in Julia, it gets "precompiled" to +a Foo.ji cache file that speeds up subsequent loads. See: +[Module initialization and precompilation](https://docs.julialang.org/en/stable/manual/modules/#Module-initialization-and-precompilation-1) +in Julia manual. PyCall uses this precompilation mechanism to reduce +JIT compilation required during its initialization. This results in +embedding the path to `libpython` used by PyCall to its precompilation +cache. Furthermore, `libpython` ABI such as C struct layout varies +across Python versions. Currently, this is determined while +precompiling PyJulia and cannot be changed at run-time. Consequently, +PyJulia can use the precompilation cache of PyCall created by standard +Julia module loader only if the PyCall cache is compiled with the +`libpython` used by the current Python process. This, of course, +requires the Python executable to be dynamically linked to +`libpython` in the first place. Furthermore, it also applies to any +Julia packages using PyCall. + +If `python` is statically linked to `libpython`, PyJulia has to use +PyCall in a mode that loads CPython API symbols from the `python` +process itself. Generating a precompilation cache compatible with +this mode requires to do it within a _`python`_ process. A key thing +to notice here is that the precompilation in Julia works by *launching +a new process* that loads the module in a special "output-ji" mode (by +running `julia --output-ji`) that creates the cache file. Thus, we +need to configure Julia in such a way that it uses our custom +executable script that behaves like `julia` program for the +precompilation. + +That is what `fake-julia` does. By changing the `JULIA_HOME` (v0.6) we trick Julia +into launching `fake-julia/julia` instead of the "real" `julia` process during precompilation. `fake-julia/julia` +is actually a Python script, but it links `libjulia` and uses `libjulia` to process the command-line arguments, +so it mimics the behavior of the `julia` process. Since `fake-julia/julia` is running from within the `python` +process, PyCall configures itself correctly. + +(From the above discussion, it should be clear that the fake-julia trick is only really necessary for +compiling PyCall and other Julia modules that use PyCall. For other Julia modules, the compiled code +should be identical to the normal Julia cache, so as an optimization `fake-julia/julia` shares the same cache +file with the real `julia` in that case.) + +Unfortunately, this "hack" does not work for Julia 0.7 and above due +to the change in the module loading system. For ongoing discussion, +see: https://github.com/JuliaLang/julia/issues/28518 + +For the discussion during the initial implementation, see also: +https://github.com/JuliaPy/PyCall.jl/pull/293 and +https://github.com/JuliaPy/pyjulia/pull/54