# Using artifacts to install python dependencies?

**URL:** <https://discourse.julialang.org/t/using-artifacts-to-install-python-dependencies/36006>\
**Category:** General Usage\
**Tags:** question\
**Created:** [March 15, 2020, 11:36am UTC](https://discourse.julialang.org/t/using-artifacts-to-install-python-dependencies/36006 "2020-03-15T11:36:10Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![jbrea](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jbrea/32/3879_2.png) [@jbrea](https://discourse.julialang.org/u/jbrea)\
**Post date:** [March 15, 2020, 11:36am UTC](https://discourse.julialang.org/t/using-artifacts-to-install-python-dependencies/36006/1 "2020-03-15T11:36:10Z")

</div>

I used to install python dependencies with `run(`$(PyCall.python) $(proxy_arg) -m pip install --user $(PACKAGES)`)` in a build script. Does it make sense to install them instead with [artifacts](https://julialang.org/blog/2019/11/artifacts/)? If yes, does somebody have an example?

---

<div class="post-metadata">

**Author:** ![staticfloat](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/staticfloat/32/34_2.png) [@staticfloat](https://discourse.julialang.org/u/staticfloat)\
**Post date:** [March 15, 2020, 7:34pm UTC](https://discourse.julialang.org/t/using-artifacts-to-install-python-dependencies/36006/2 "2020-03-15T19:34:01Z")

</div>

Hey! I actually did some experiments on this a little while back. My repository [`PyEnv.jl`](https://github.com/staticfloat/PyEnv.jl) attempts to bring the advantages of Julia Manifest files and Artifacts to python packages. Quick TL;DR for those that don’t read the whole README: This package **IS NOT** good for production usage yet; it’s an experiment and it showcases weaknesses in both its own implementation as well as difficulties with working in the Python ecosystem.

The package essentially uses a `Python_jll` artifact to get a python interpreter, uses `pip` to download/install python packages to a local environment, then you can “artifactify” that local environment, which freezes the python source tree into artifacts that get baked into an `Artifacts.toml` file. Those artifacts could then be uploaded somewhere to provide a nice, portable and completely reproducible python environment. The `with_artifacts_pyenv()` wrapper function will set up all the environment variables and whatnot needed for `python` to be able to find its packages, and everything should be happy. In theory.

In practice, there are still significant barriers to this. For instance, python packages, in general, must run arbitrary python code at install-time, up to and including resolving what other packages to install. This means that, without being able to run the python code during install, we may not know exactly what needs to be installed. I spent a good chunk of time trying to install Python packages straight from the metadata in PyPI (and not using `pip` at all; see the `direct.jl` file in the repo) but in the end decided it was too much work, and there were not enough python packages using solely `.whl` files (which give a much nicer interface for this than the competing formats).

A consequence of this is that it makes the python environments completely platform-specific, and it’s difficult to generate python environments for foreign architectures, because we’re not running on those foreign architectures. This is solvable by generating a separate pyenv on each platform you want, but this is a substandard experience for us, where we’re used to being able to create a manifest on one machine, then instantiate it on another, and as long as that other machine is running a supported platform, everything should “just work”.

My opinion on this is that there is promise, but we need to wait for the Python packaging ecosystem to settle more fully on wheels and a fully declarative interface for installing packages. If the entire subtree of dependencies that you require have wheels for all major platforms, the strategy in the `direct.jl` code will allow us to generate cross-platform lists of archives that must be downloaded and unpacked, without needing to run python code at install time (which would then lock us to installing only for the current platform). Once we can do that, we will be able to provide a Julia-like experience for downloading and unpacking Python packages, and more than that, recording them in an Artifacts.toml (such as [this one](https://github.com/staticfloat/PyEnv.jl/blob/master/data_science.artifacts.toml) which gets generated only for the current platform; note how there’s only platform-specific entries for the single platform it was built for) so that things are nicely reproducible.

---

<div class="post-metadata">

**Author:** ![jbrea](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jbrea/32/3879_2.png) [@jbrea](https://discourse.julialang.org/u/jbrea)\
**Post date:** [March 15, 2020, 8:11pm UTC](https://discourse.julialang.org/t/using-artifacts-to-install-python-dependencies/36006/3 "2020-03-15T20:11:41Z")

</div>

Thanks a lot for the detailed answer!