Build PyCall with error in installing miniconda

General Usage
,
1

When I built PyCall, I found that miniforge can’t be installed on the path with non-ascii chararcters which is my username. So I manually installed the miniforge and set the env. But in this case, julia can’t manage python and I have to install packages manually.
Is there any way to avoid this?

2

Not a solution to your exact problem but have you tried PythonCall.jl instead? It’s a more modern take on PyCall.jl which sometimes simplifies installation

1 Like
3

when I used CondaPkg to install packages, it showed the critical libmamba and failed to extract and open.

here the part of error report:

critical libmamba Failed to open 'C:\Users\%username%\.julia\scratchspaces\0b3b1443-0f03-428d-bdfb-f27f9c1191ea\root\pkgs\vc14_runtime-14.40.338'
4

I thought the problem maybe same as before. non-ascii username may result this.

5

Don’t use a username with non-ascii characters. Alternatively, it might help to set the variable JULIA_DEPOT_PATH to a path without non-ascii characters.

export JULIA_DEPOT_PATH=/juliadepot

This obviously works only if you have the permissions to create a directory outside of your home directory in the first place.

6

In windows, there doesn’t seem to be a clean way to change username. If it has, please tell me! So right now I can only choose the latter one.

7

Not exactly true. How it can be done is explained here:

8

It just change the info about user(binding to microsoft), but it can’t change the username path etc.

9

Are you using Windows 10 or Windows 11?

10

Windows11.

I followed the steps, it does work but flawed. I took a while to restore things like onedrive. There’s still other things aren’t mentioned, for example terminal fonts missing and some path problem.(I previously installed fonts for terminal and after that it’s missing)

11

I install PythonCall now, and it works fine.

If I had to choose, which one is better? (based on gaddle, PythonCall is better), but it doesn’t have any interface which means all methods should be memorized.

12

You mean you confirm PythonCall works with non-ASCII username?

Then it’s better, in almost every way including that. And ideally all software, including PyCall, shouldn’t be restricted to ASCII [username]. PyCall has (still) faster startup, the only advantage I recall over PythonCall’s:

julia> @time using PythonCall
  1.243336 seconds (381.06 k allocations: 28.079 MiB, 7.42% gc time, 55.30% compilation time)

Note, that’s in 1.10.5 and since it’s a one-time cost, I wouldn’t worry too much, and the time will hopefully come down in future versions of PythonCall, or future Julia’s but actually, in 1.11.0-rc3 (can other confirm slowdown, or might be be related to something other than version change?):

julia> @time using PythonCall
  5.406764 seconds (3.24 M allocations: 176.386 MiB, 7.58% gc time, 80.11% compilation time: 18% of which was recompilation)

While it can be mitigated with changing (to lower) optimization:

$ julia +1.11 -O0

julia> @time using PythonCall
  3.512657 seconds (3.24 M allocations: 176.386 MiB, 9.10% gc time, 69.22% compilation time: 19% of which was recompilation)

or even (you likely do not want to…):

$ julia +1.11 --compile=min

julia> @time using PythonCall
  1.527100 seconds (881.98 k allocations: 51.569 MiB, 9.37% gc time, 13.26% compilation time)

note in 1.10.5:

$ julia --compile=min
julia> @time using PythonCall
  0.809859 seconds (230.96 k allocations: 17.532 MiB, 13.11% compilation time)
13

Sadly, I still change the username folder name to manage this(when I install miniforge3, it warns me that I can’t install with non-ascii path due to the incompatibility of certain packages, that’s the reason I post the question).

Thanks for providing info about time cost evaluation of PythonCall!

1 Like
14

Well, PyCall has a number of advantages for me:

15

Are you building it to use it directly from your code, i.e. to call some Python [library] code, or for using an existing python-wrapping package? I.e. are “forced” to use PyCall, with no alternative package?

Correct, though SymPy.jl, and more are sort of an exception. There’s also SymPyPythonCall.jl to do the same (and they share common code in a Core package). PythonPlot is another corresponding to PyPlot.

[That said, yes, JuliaControl/SymbolicControlSystems.jl depends on SymPy, not SymPyPythonCall, but likely could, maybe such a package already exists, or if not it should just switch?]

Even if you have a package using PyCall, you likely should be using PythonCall for your own code, or for other package dependencies [you’re making], since different dependencies can use either. See docs about using both together.

Yes, PyCall.jl has 257 direct (plus indirect) dependent python wrapping packages, currently according to juliahub, more than PythonCall.jl’s 69. But that is only because the latter is newer.

[Some packages are utilities, such as PyCallChainRules.jl.]

1 Like
16

I tried to rewrite it for using PythonCall and failed. Which does not mean it is not possible, it just means it is more than a few days of work.

1 Like
17

Using Python in the context of a Julia package is a pain. Using both PyCall and PythonCall is twice the pain. I would advice against it.

Just as an example the section of a bash script that I use to install Python and Matplotlib if required:

PYTHON_PATH=$(which python3)
if [ -x "$PYTHON_PATH" ]; then
    echo "Python is found at $PYTHON_PATH"
    if $PYTHON_PATH -c "import matplotlib" &> /dev/null; then
        echo "Matplotlib found. Using existing installation."
        export PYTHON=$PYTHON_PATH
    else
        echo "Matplotlib is not found."
        read -p "Do you want to install matplotlib with Conda? (y/n): " choice
        case "$choice" in 
          y|Y ) 
            export PYTHON=""
            ;;
          n|N ) 
            echo "Exiting without installing matplotlib."
            exit 1
            ;;
          * ) 
            echo "Invalid choice. Exiting."
            exit 1
            ;;
        esac
    fi
else
    echo "Python is not found."
    exit 1
fi

julia --startup-file=no -e "using Pkg; Pkg.add(\"TestEnv\")"

if [[ $update == true ]]; then
    echo "Updating packages..."
    if test -f "Manifest.toml"; then
        mv Manifest.toml Manifest.toml.bak
    fi
    julia --startup-file=no --pkgimages=no --project -e "using TestEnv; TestEnv.activate(); using Pkg; Pkg.add(\"PyCall\"); Pkg.build(\"PyCall\")"
    if [[ $PYTHON == "" ]]; then
        julia --startup-file=no --pkgimages=no --project -e "using TestEnv; TestEnv.activate(); using Pkg; Pkg.add(\"Conda\"); using Conda; Conda.add(\"matplotlib\"); using ControlPlots"
    fi
    julia --startup-file=no --pkgimages=no --project -e "using Pkg; Pkg.update()"
fi

By the way, this code works very well. No more complaints that installing PyPlot (well, I use ControlPlots, but that currently relies on PyPlot) fails since I have this script.

1 Like
18

FYI: PythonCall isn’t likely inherently slower on 1.11, only its dependency 16x slowdown for CondaPkg v0.2.23 on 1.11.0-rc3, and therefore JuliaCall much slower to start · Issue #145 · JuliaPy/CondaPkg.jl · GitHub

and it’s in part because Pkg is no longer in the sysimage, so possibly worth it to lazily load it.

EDIT: These are slower:

               ┌ 1839.1 ms PythonCall.C.__init__() 90.91% compilation time  # why this slow, though not much slower?
               ├ 0.0 ms PythonCall.GC.__init__() 
               ├ 771.2 ms PythonCall.Core.__init__() 96.96% compilation time (85% recompilation)  # why this 23x slower?

Can someone help with the “recompilation” (that only happens on 1.11)? >The above “90.91% compilation time” is slightly worse on 1.11, but happens on both it and 1.10. This is mentioned in the above issue (in wrong repository).

19

Are you sure? You had to take care of Python dependencies yourself with PyCall, but you can let PythonCall, actually CondaPkg, install them (including python itself), e.g. matplotlib for you.

It seems you need no bash script, only before for PyCall.

I would use PythonCall, PyPlot’s replacement, but yes, ControlPlots relies on it.

EDIT EDIT: ControlPlot relies on Plots.jl, GR by default (here I mixed up with the other Control package, that does have plotting…, and the rest is true), so I suppose it could use PythonPlot already, even though docs do not mention it yet: ControlSystems.jl/docs/src/man/introduction.md at efac62fe6c1e917e10b1acda7e288e847b1d4c5a · JuliaControl/ControlSystems.jl · GitHub

Should PyPlot (and/or Plots) be changed to rather rely on PythonCall, so that dependents like ControlPlots just work? Alternatively ControlPlots could change to use PythonPlot.

[Using PythonCall, and PyCall is supported, explained in docs of former package. And I mentioned in an open PyCall issue yesterday, on them not working together, and then it was closed, so I assume I’m not wrong about them actually working together. Did you rely on old info?]

Can someone help with “recompilation”? I edited, added to my above post.

20

Well, I need a bash script to create the system image anyways. And it is quite sophisticated: create_sys_image.

And I was unhappy with my experience when using PythonPlot: It was updating the Python dependencies every time I run my scripts, which is perhaps 10 or 20 times per day. Now this happens only if I compile a new system image, which happens perhaps once per week.

1 Like