just informed myself about diamond dependency problems, i.e. the problem when a package A depends on package B and C which both depend on another package D, however with different versions. I.e. package B uses D v1.2 and C uses D v1.3.
The crux in such a setup is that you cannot easily pass results from B which use data structures from D v1.2 to C, which expects D v1.3 datastructures and hence may fail because of incompatibilities between the versions.
( See for instance this explanation for illustration and details The dreaded diamond dependency problem - Well-Typed: The Haskell Consultants )
More concretely, in julia, the functionality in C may not work at all with data from B, because it depends on function dispatches which uses types defined in D v1.3 (which are different from types from D v1.2)
Such problems look perfectly possible in julia 1.0 as every package is its own environment with individual dependency versions. How to solve this class of problems in julia?
just to mention the only reference I found about julia and diamond dependencies: https://news.ycombinator.com/item?id=17719489 (but as you see, there is only the question, without an answer)
Thanks for your answer, can you please elaborate? What am I overlooking?
How should it be impossible for package B and C to use different versions of D?
They have their own “virtual environments” (in Python terms) and can use whatever they want to
At any given moment, you only have one “virtual environment” active. The requirements of A will determine the versions of B and C, and their requirements will in turn determine the version of D. In your example, C requires D v1.3 and so this is the version of D to be installed, not the older version D v1.2 required by B.
Spot on. I also believe @schlichtanders may be misunderstanding how Python virtual environments work. You can have different versions of D in different environments but only one environment is active for a given Python process—virtual environments are implemented by manipulating various environment variables which are shared by the whole process—so you will only load one version of D in a given Python session. This is also how it works in Julia but the environment mechanism is built in and does not require messing around with environment variables to simulate different environments.
Note that the “dreaded diamond dependency problem” is not all that dreaded in dynamic languages. This is because dynamic languages can fairly easily perform load time reflection to work with different incompatible APIs for different versions of dependencies. So dependency coupling is much looser than in static languages, where incompatible API changes are checked and enforced by the compiler and therefore cannot be programmed around without some kind of fairly complex API adaptation system.
Julia packages program around API differences all the time. Compat for example provides a consistent API across many underlying Julia versions by checking the Julia version and implementing the consistent API in terms of whatever Julia version is running. This is one of the reasons why Julia version bounds are usually pretty loose.