mkitti
February 27, 2023, 7:00am
1
@MilesCranmer discovered a bug in pyjulia that only manifested in Julia 1.9 due to a correction in argument processing.
Suppose we have a file printargs.jl with the following content.
println(ARGS)
If you placed a -- after the printargs.jl on the command line, you will get different results between Julia 1.8 and Julia 1.9:
$ julia-1.8.5/bin/julia --startup=no printargs.jl -- 1 2 3
["1", "2", "3"]
$ julia-1.9.0-beta4/bin/julia --startup=no printargs.jl -- 1 2 3
["--", "1", "2", "3"]
Julia 1.9 is processing the arguments correctly. Looking at the command line help, the -- is clearly suppose to come before the programfile. If it comes after, then it should be an argument.
$ julia-1.9.0-beta4/bin/julia --help
julia [switches] -- [programfile] [args...]
...
Putting the -- before the programfile results in consistent behavior between the versions.
$ julia-1.8.4/bin/julia --startup=no -- printargs.jl 1 2 3
["1", "2", "3"]
$ julia-1.9.0-beta2/bin/julia --startup=no -- printargs.jl 1 2 3
["1", "2", "3"]
JuliaLang:master ← JuliaLang:kc/dashdash
opened 11:15AM - 17 May 22 UTC
Rebase of https://github.com/JuliaLang/julia/pull/37010 since that PR does not a… llow updating the author's branch.
> This is a proposed fix for #37009. It removes manual processing of `--` and relies on `getopt()` to handle it correctly. The expected behavior:
> 1. If `--` appears before the first non-option argument, it forces the end of option processing and does not appear in `ARGS`.
> 2. If `--` appears after a non-option argument, it is preserved in `ARGS`. This `--` has no effect on option processing since it's already stopped on the first non-option argument.
opened 02:06AM - 27 Feb 23 UTC
closed 09:47AM - 27 Feb 23 UTC
I can't seem to get PyJulia working on Julia 1.9 (beta 4), although it works fin… e on 1.8.5. It seems to be some issue with the Python interpreter and libpython set by PyCall.jl. It could be interference from Conda.jl's installed copy of Python and libpython?
First, I set up a completely fresh Julia install with juliaup (note that this code will delete your `~/.julia` folder, only saving `startup.jl`):
```bash
mv ~/.julia/config/startup.jl ~/startup.jl && \
rm -rf ~/.julia && \
mkdir -p ~/.julia/config && \
mv ~/startup.jl ~/.julia/config/startup.jl && \
juliaup add 1.9 && \
juliaup default 1.9 && \
julia --startup-file=no -e 'using Pkg; Pkg.add(["OhMyREPL", "Revise"])'
```
Then, I run:
```bash
python -c 'import julia; julia.install()'
```
Which succeeds. However, when I try to start Julia via PyJulia:
```bash
python -c 'from julia import Main'
```
<details><summary> despite the exact same set of environment variables as used in the `install`, this fails: </summary>
```bash
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 672, in _load_unlocked
File "<frozen importlib._bootstrap>", line 632, in _load_backward_compatible
File "/Users/mcranmer/venvs/main/lib/python3.10/site-packages/julia/core.py", line 247, in load_module
JuliaMainModule(self, fullname))
File "/Users/mcranmer/venvs/main/lib/python3.10/site-packages/julia/core.py", line 149, in __init__
self._julia = loader.julia
File "/Users/mcranmer/venvs/main/lib/python3.10/site-packages/julia/core.py", line 239, in julia
self.__class__.julia = julia = Julia()
File "/Users/mcranmer/venvs/main/lib/python3.10/site-packages/julia/core.py", line 489, in __init__
raise UnsupportedPythonError(jlinfo)
julia.core.UnsupportedPythonError: It seems your Julia and PyJulia setup are not supported.
Julia executable:
julia
Python interpreter and libpython used by PyCall.jl:
/Users/mcranmer/.julia/conda/3/aarch64/bin/python
/Users/mcranmer/.julia/conda/3/aarch64/lib/libpython3.10.dylib
Python interpreter used to import PyJulia and its libpython.
/Users/mcranmer/venvs/main/bin/python
/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/Python
In Julia >= 0.7, above two paths to `libpython` have to match exactly
in order for PyJulia to work out-of-the-box. To configure PyCall.jl to use
Python interpreter "/Users/mcranmer/venvs/main/bin/python",
run the following code in the Python REPL:
>>> import julia
>>> julia.install()
For more information, see:
https://pyjulia.readthedocs.io/en/latest/troubleshooting.html
```
</details>
In particular, the Python interpreter and libpython seem to be set incorrectly:
```
Python interpreter and libpython used by PyCall.jl:
/Users/mcranmer/.julia/conda/3/aarch64/bin/python
/Users/mcranmer/.julia/conda/3/aarch64/lib/libpython3.10.dylib
Python interpreter used to import PyJulia and its libpython.
/Users/mcranmer/venvs/main/bin/python
/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/Python
```
I'm assuming these new `python` and `libpython` are packaged by the `PyCall.jl` install. But I think ideally we would want them to be set by the user from their Python/libpython, for compatibility with Python libraries, right?
**I can run this entire workflow with `juliaup add 1.8 && juliaup default 1.8` instead, and it works fine.**
cc @mkitti @ngam @stevengj @marius311
---
edit: I should mention I'm not using conda at all. This is a homebrew-installed version of Python running in a virtualenv.
4 Likes