I use VS CODE 1.58.0 (Insiders) and SSH connection to connect to Ubuntu 18.04.5 LTS and conda environment (env name: base). I have several conda environments. One of them is destined for Julia with Julia 1.6 (env name: julia) installed from conda-forge channel.
I set VS CODE interpreter to specified Python conda environment where Julia 1.6 is installed, set Julia executable path in REMOTE part of Julia extension to Julia 1.6 version installed in conda environment.
When I start Julia REPL (CTL + SHIFT + P, Julia: Start REPL) REPL starts correctly with Julia 1.6 version, however, it seems that Julia REPL is operating in (base) conda environment, not in desired environment where Julia 1.6 is installed. Executing conda env list in Julia REPL shell ( ; ) informs that I am in (base) environment, not in (julia).
On the other hand, when I start separate terminal, activate (julia) environment, start Julia form the terminal everything works fine. The problem is related to Julia REPL when starting it from VS CODE Command Palette.
I would like to ask how to start Julia REPL in VS CODE (Insiders) from VS CODE Command Palette in specified conda environment? I am operating over SSH. Is it possible at all?
Hi JordiBolibar, the problem I described was about executing Julia code in VS Code on remote machine in specified (non base) conda environment with the use of i.e. Alt+Enter. Lets call this specified non base conda environment Julia16CondaEnv.
In other words, when I wanted to execute Julia code, after hitting Alt+Enter VS Code was opening new Julia REPL and this REPL was operating in the base conda environment where I had a different Julia version than in Julia16CondaEnv (it was opening REPL in a different environment that I wanted).
How I am solving this problem is with the use of VS Code Julia extension that since version 1.2 is offering option to connect to external REPL. Those are the steps I follow:
I am connecting through ssh to my machine. I am finding myself in the base environment.
In VS Code I am opening Command Pallete (Ctl+Shift+P) and typing “Julia: Connect external REPL”. I am copying the code from the lower right part of the screen next to “Start a Julia session, and execute the code copied into your clipboard by the button below into it.” into the clipboard.
In the terminal I am changing the environment to Julia16CondaEnv (typing “conda activate Julia16CondaEnv”).
I am starting julia (i.e. JULIA_EXCLUSIVE=1 julia -t 24) in this desired Julia16CondaEnv conda environment.
I am pasting and executing the code from Point 2 into this Julia session.
And now when I hit Alt+Enter the code is executing in the REPL that is operating in my desired Julia16CondaEnv. By using “Julia: Connect external REPL” I can even operate on two local computers connected to my remote machine, on one using VS Code and on the second using Julia REPL and everything works.
I have not been exploring this topic further since the time I found this solution. Maybe there is an option to make VS Code and Julia automatically connecting to Julia16CondaEnv, without all this typing. Maybe @pfitzseb or somebody who is fluent with VS Code / conda can provide some additional advice.
I understand that the subject is not directly related to PyCall. However, I have to admit that I had also problems with running PyCall in various anaconda environments on remote machines particularly when trying to use some external software packages. If this is not the answer you were looking for, please feel free to provide more details in this thread or a separate one. I found some solutions directly related to PyCall, not sure if ideal but maybe I would be able to help.
Rather use the official Julia binaries from Download Julia than install Julia from Conda. Julia does not need Conda (except for Python interaction) and the Conda version is not officially supported.
For using PyCall, set the environment variable ENV["PYTHON"] = "... path of the python executable ..."
before building PyCall (see Readme · PyCall.jl ).
I am not sure if this fixes your VSCode issues, but there is a fair chance because this is the recommended setup.
I am not sure if this fixes your VSCode issues […]
I provided my solution and described it briefly on the Jun 26th as well as today. The use of the option to connect to external REPL of VS Code Julia Extension (>1.2) makes it possible to connect to specified conda environment and to execute code directly from VS Code. At the time I was writing my first post in this thread VS Code Julia Extension v.1.2 was a new or a very new thing. Are you aware of a better approach to execute Julia code (i.e. Alt+Enter) from VS Code on remote machine in Julia REPL in specified conda environment (not base)?
Rather use the official Julia binaries from Download Julia than install Julia from Conda.
I fully agree. However at that time in this particular case it was easier for me to use conda install.
Julia does not need Conda (except for Python interaction) and the Conda version is not officially supported.
I rather prefer to use separate conda environment for Julia setup and PyCall interaction. I have global conda on the remote machine.
AFAIK, Julia comes with its own conda. In theory this should make things easier, in practice I am recalling I spent more then a day in order to set all things up. This included PyCall interaction with as I understand “Python wrapper” of Fortran and MPI package installed through Conda. Its working, however, I am not sure if I would be able to replicate the setup again on the first approach. I am happy to try it again with a new conda environment and Julia 1.8 as 1.8 supports more OpenBLAS threads than 8. I got slightly more experience than a few months ago but I would welcome some additional advice.
For using PyCall, set the environment variable ENV["PYTHON"] = "... path of the python executable ..."
before building PyCall (see Readme · PyCall.jl ).
To which Python?
What Python should I put into a new conda environment?
<conda create -n “Julia18”> and not specify any Python? Than I guess the one from global conda would be installed?
The same version that comes with Julia Conda? As far as I remember for Julia 1.6 it was Python 3.8.5 so <conda create -n “Julia18” python=3.8.5 ipython>
Or maybe the one that “Python wrapper” of Fortran and MPI package was used? (This is a guess, I do not know if this makes any difference or if this assumption is correct at all.)
“conda info” returns:
python version : 3.7.11.final.0,
my base environment is “read only”,
I got 2 env directories that include /home/xxxxxx/.conda/envs,
user-agent is: conda/4.10.3 requests/2.25.1 CPython/3.7.11 Linux/5.4.0-80-generic ubuntu/20.04.3 glibc/2.31
I would like to install Julia 1.8 from official binaries (I understand its currently a nightly build from the link provided by you) . As stated above I am on Ubuntu.
I would suggest the following steps for setting up Julia and Python:
install Julia from binaries (the directory does not really matter)
create a separate Conda environment for Python-Julia interactions (or even one for each of your projects) and install the Python version and all your Python dependencies there with Conda as you need them.
set ENV['PYTHON'] to the Python executable in your project-specific Python environment
In Julia, use a project-specific Pkg environment and ] add PyCall and all Julia dependencies you need.
] build PyCall if your Python path has changed (e.g. if you are working with multiple Conda envs from Julia)
This setup is imho more flexible and better reproducable than using the default Conda env from Julia.
This setup is imho more flexible and better reproducable than using the default Conda env from Julia.
As I understand it you are suggesting using global Conda (not Conda that comes with Julia) to manage environments and Python packages which as I understand is different than mostly recommended use of Conda that comes with Julia. As I understand it your suggestions might also provide a way to execute the code directly from VS Code with the use of i.e. Alt+Enter as well as provide access to Julia versions that are not present at conda / conda-forge.
Based on this understanding I tried to implement your suggestion. I created new Conda environment “julia18” [“conda create --name julia18”] which is located at /home/xxxxxx/.conda/envs/julia18. I installed Julia from julia-latest-linux64.tar.gz official binaries. I tried two options in julia to set ENV['PYTHON']: one was ENV["PYTHON"]="" and the second ENV["PYTHON"]="/home/xxxxxx/.conda/envs/julia18/bin/python" and in both cases I added PyCall v1.92.3 [Updating `~/.julia/environments/v1.8/Project.toml.
When:
julia> using PyCall
[ Info: Precompiling PyCall [438e738f-606a-5dbb-bf0a-cddfbfd45ab0]
ERROR: LoadError: IOError: open("/glob/development-tools/versions/[...]python/latest/condarc-julia.yml", 577, 33261): read-only file system (EROFS)
in expression starting at /home/xxxxxx/.julia/packages/Conda/sNGum/src/Conda.jl:1
ERROR: LoadError: Failed to precompile Conda [8f4d0f93-b110-5947-807f-2305c1781a2d] to /home/xxxxxx/.julia/compiled/v1.8/Conda/jl_2w0VUw.
in expression starting at /home/xxxxxx/.julia/packages/PyCall/BD546/src/PyCall.jl:1
ERROR: Failed to precompile PyCall [438e738f-606a-5dbb-bf0a-cddfbfd45ab0] to /home/xxxxxx/.julia/compiled/v1.8/PyCall/jl_WmUWT5.
I encountered errors as above. As I informed in my previous post my base environment is “read only”. I think it is a similar situation to descibed here: https://github.com/JuliaPy/Conda.jl/issues/166. It was more than a few months ago when I was setting up my environment, however, if I am recalling correctly this was one of the reasons that I decided to install Julia through conda in a separate conda environment. I took a look at my notes and I think those might be the steps I followed:
# Create julia env with python=3.8.5
conda create --name julia python=3.8.5
# OPTIONAL: Add conda-forge channel
conda config --add channels conda-forge
# Install Julia 1.6.0
conda install -c mjohnson541 julia
# Start julia and
julia> ENV["PYTHON"]=""
(@v1.6) pkg> add PyCall
(@v1.6) pkg> add Conda
julia> using Conda
julia> Conda.add("choosen conda package name")
or
use global Conda to add packages to Conda environment.
Should you have any comments or suggestions I would be happy to try them. Thank you once again.
I am also recalling that after changing ENV["PYTHON"] = " path to python" and building PyCall, one should exit Julia and start it again before using PyCall.
In addition, I would like to mention PythonCall.jl [https://github.com/cjdoris/PythonCall.jl] that is an alternative to PyCall. As written at PythonCall documentation, in contrary to PyCall, it uses a separate Conda environment for each Julia environment/project/package and installs Python (and other Python packages) into that. This means that different Julia projects can maintain an isolated set of Python dependencies (including the Python version itself). [https://cjdoris.github.io/PythonCall.jl/dev/pycall/].
I would like to provide you with more precise and detailed info related to PyCall, however, as I mentioned several times, all this I was doing a few months ago and I am recalling that I spent more than a day to make it work exercising different options. I also mentioned that I was not sure if I could replicate it on the first approach. Never less, I hope that you find the info useful, particularly general approaches presented by @lungben and @Balinus as well as you find the trick with External REPL interesting.
Indeed, it is a tricky subject, since the configuration is quite error prone. I first was working with Julia with a conda environment in a Jupyter notebook and I had no issues. However, when trying to run the code from a .jl file, things started to get messier. I spent yesterday trying different things out and I more or less isolated the smoothest way for me to work with a conda environment in VSCode:
# First of all I activate the Julia environment
using Pkg; Pkg.activate("../.");
Pkg.instantiate()
# And then, before importing any other Julia modules (I had conflicts with fiona and other libraries), specify the conda environment
# Use same path as "which python" in shell
ENV["PYTHON"] = "/Users/xxxxx/miniconda3/envs/oggm_env/bin/python3.9"
Pkg.build("PyCall")
using PyCall
# Then import Python packages
np = pyimport("numpy")
xr = pyimport("xarray")
# And finally import Julia packages and external files
using SparseArrays
using Statistics
using LinearAlgebra
include("helpers/parameters.jl")
include("helpers/types.jl")
With this order, everything runs fine from the Julia REPL instantiated within VSCode (both using SHIFT + ENTER to run the whole file or OPTION + ENTER to run a few lines). It was a matter of order, since sometimes there might be some conflicts between Julia and Python libraries. I’m not sure if this will be the same for your case, but if you encounter some strange behaviour, verify the order to the commands.
Would you mind providing also the relevant output of conda env list with related paths? I would be happy to compare it with my output to try to better understand potential differences.
I would welcome some additional advice as it seems that my notes I used to replicate the working setup are not complete.
To recap:
My working setup consists of a) julia 1.7 beta3 installed in base environment and b) julia 1.6 installed through conda.
I would like to replicate the installation I did through conda or even better install julia from official binaries and use PyCall, however, it seems that I am unable to do it. When I try to follow instruction and my notes I am receiving similar ERRORS to the ones below:
julia> using PyCall
[ Info: Precompiling PyCall [438e738f-606a-5dbb-bf0a-cddfbfd45ab0]
ERROR: LoadError: IOError: open("/glob/development-tools/versions/[...]python/latest/condarc-julia.yml", 577, 33261): read-only file system (EROFS)
in expression starting at /home/xxxxxx/.julia/packages/Conda/sNGum/src/Conda.jl:1
ERROR: LoadError: Failed to precompile Conda [8f4d0f93-b110-5947-807f-2305c1781a2d] to /home/xxxxxx/.julia/compiled/v1.8/Conda/jl_2w0VUw.
in expression starting at /home/xxxxxx/.julia/packages/PyCall/BD546/src/PyCall.jl:1
ERROR: Failed to precompile PyCall [438e738f-606a-5dbb-bf0a-cddfbfd45ab0] to /home/xxxxxx/.julia/compiled/v1.8/PyCall/jl_WmUWT5.
I see differences related to location of Python in my working environment that I want to replicate which seem to be not in line with information presented in this thread, however, again, the setup seems to work correctly. When in julia environment [this is the environment with julia 1.6 that I installed the way I described in post # 8 above], the differences I see are:
which python
/home/xxxxxx/.conda/envs/julia/bin/python
I am providing additional information, hope in a consistent way.
WORKING SETUP WITH JULIA 1.6
conda env list
julia * /home/xxxxxx/.conda/envs/julia
/home/xxxxxx/.conda/envs/julia/share/julia/site/conda/3
which python
/home/xxxxxx/.conda/envs/julia/bin/python
conda env list provides also two other environments, I understand they are related to julia 1.7 beta3 installed through jill from official binaries at base environment.
What is /home/xxxxxx/.conda/envs/julia/share/julia/site/conda/3 environment? How can I recreate it and provide PyCall with the path to Python located there?
You got the same paths to Python as Conda environment paths, I got two different. I think that in order to call Python from Julia in environment with global non-writeable Conda depot one can use Conda that comes with Julia or at least this is how I currently understand it. I got Julia installed through jill at base which created Conda ROOTENV in /home/xxxxxx/.julia/conda/3 [how exactly I do not remember, maybe its standard, maybe its not].
Now I can:
- download Julia binnaries as suggested by @lungben,
- set ENV["CONDA_JL_HOME"] = "/home/xxxxxx/.julia/conda/3",
- ]add Conda
- (and maybe set ENV["CONDA_JL_HOME"] = "/home/xxxxxx/.julia/conda/3" again,
- ]build Conda,
- ]add PyCall
- ]build Pycall as location of /home/xxxxxx/.julia/conda/3/condabin/bin/python has moved to /home/xxxxxx/.julia/conda/3/bin.
Without those steps I was not able to use Conda v1.5.2 nor PyCall v1.92.3 due to ERROR of read-only file system (EROFS). Now it works ok with any Julia version I have installed which is 1.8.0-DEV.772, 1.7beta3 and 1.6. Frankly as for now, I am not sure if I am using any different Conda environment that the root one that comes with Julia or to be more precise that sometimes I am probably using two at the same time. One from Julia and the other from global Conda. Should it be the case probably PythonCall can be safer to use than PyCall as it is written that “it uses a separate Conda environment for each Julia environment/project/package and installs Python (and other Python packages) into that” whereas " Python packages for use with [PyCall] must use the root environment".
Should there be any additional suggestions I would be happy to try them. Hope it might be of any interest to somebody. Thank you.
