Hey,
My notebook was working fine, but after some changes, I’m now getting an error.
Please help me to fix this problem.
The first few cells of my Jupyter Notebook are shown below:
Charles
You’re on an ancient version of CSV, the current release is 0.9.8. This looks like a classic case of default environment pollution to me, do you really need all of those packages at the same time?
As a new user of Julia, I’m remined of how much pain I went through with Anaconda Python package incompatibilities. It took years for Anaconda to help users install packages that were compatible with each other. I predict that Julia will go through the same long pain period in dealing with package incompatibilities. I’ll come back to Julia when you guys solve this problem.
Welcome to dependency hell 
When a developer works on a package which has dependencies (quite common if you use a language with a decent package manager), they need to put some constraints on versions to ensure that the package is working as intended. Julia (and many other language package managers) offers a standardised way to define those constraints – see the docs about Compatibility – which relies on a silent meta-contract that every other package developer respects Semantic Versioning
This is a problem which is simply there, you cannot do anything about it and every single language faces it. At some point, package developers will improve their software and introduce breaking changes, so they release new major versions and everyone else who relies on that package needs to test their software with the new versions to see if it works as intended.
There are packages which are not (actively) maintained anymore because person X has moved on, has other things to do, or died, you name it…
If you decide to install at least 26 packages in your global environment, you are calling for trouble. So do you when you install that much packages in a Python environment via pip or conda, or in a Haskell env with Cabal, with npm for JavaScript you’ll find yourself in a yet another awkward situation but basically the problem is always the same: things move on, the Earth rotates.
Julia has a built-in package manager and gives you the – in my opinion best – possibility to manage environments easily, see the docs of Pkg.jl. For Python it took a couple of years to put virtualenvs into the standard library and there are a bunch of alternative ways to manage environments, including conda and whatnot. It’s a mess and it does not get better with a new kid around the block every other year.
Long story short: what can you do about?
Simple: try to focus on projects and only install what’s needed, don’t use one environment for everything, you are calling for trouble. I use isolated environments for each project and analysis I do, no matter if Julia, Python, Haskell, Go, JavaScript, or whatever language I use. Yes, this means I have hundreds of environments, but guess what: they are all independent and controllable and I don’t have to worry that I update one package and suddenly my analysis from last year which I now want to hand out to a new master student doesn’t run anymore!
Which means: “my notebook was working fine, but […] now getting an error” is totally your fault. You changed the state of your system and you decided to do so. Project.toml and Manifest.toml are there to pin all dependencies and these are described in the docs. Additionally, this should be managed by a version control system like Git and you should be able to recreate the environment in which your notebook has worked before. It’s not the fault of the “guys” who work on Julia. Those “guys” created something which is way ahead of Anaconda or other package managers capabilities. Trust me, I am using quite a few languages.
If you don’t have your old Manifest.toml, everyone can only guess what has happened. You can try to force updated CSV.jl and see which package is holding it back or simply delete the global environment and reinstall everything. Again, hard to tell if you have not followed some basic “best practices”, which are not Julia specific, just common sense when working with computers.
7 Likes
well, Julia learned from Python’s failure and has a much more robust system. Just to name a few:
- not
setup.pybs - Compatibility (both ways) is stored in the General registry, which means you don’t need to download multiple versions of the same package to “try” out the compatibility
- binary dependencies are managed in the exact same way as normal package
Just add Pkg.activate(".") at the top, and this will all sort itself right out.
Can you try again where you un-comment the Pkg.add steps that you have commented out in the post above?
Please let me respond. You wrote: “before I do so: I have the feeling that you have never contributed a package to the public”. True! I’m part of the unwashed masses. I have a problem I need to solve, and I tried Julia to see if it could solve it.
You talk about “constraints”, “toml” files etc. I interpret these kind of statements as “You should not be using a Jupyter Notebook!”
You guys accused me of loading too many packages. When I first started the Jupyter Notebook, I had only four package adds:
using DataFrames
using CSV
using DecisionTree
using Lathe
It worked great. However, Lathe did not have the functionality I needed, so I added “Pkg.add(“MLJ”)”. Big mistake! That’s when all those packages were added, and it broke my Jupyter Notebook.
I don’t mean to be critical, I’m just frustrated.
There’s a bit of confusion here. using a package and adding it to your environment are two different things. The results of Pkg.status() in your initial screenshots show that you actually had many more packages in your environment, which is where your incompatibilities came from.
This is not to say it’s okay that CSV errored. It’s a problem that adding it to your environment works and using it fails. I know that specific problem was fixed in more recent releases of CSV.jl, but I think it also indicates some larger issue with the way the resolver works, which should be addressed. Though I don’t know what the actual issue is.
As a general rule, whenever you start a new project in Julia, for any reason, always have Pkg.activate(".") somewhere so you aren’t adding packages to your “global” 1.6 environment.
Confusingly, you can use packages in your “global” environment even after Pkg.activate("."), which is why it looks like DataFrames.jl etc are in your environment, but they are not. This is definitely confusing and can cause frustration.
Here’s an MWE which will fix your problem
(@v1.6) pkg> activate --temp # make a temporary environment
Activating new environment at `/var/folders/hw/wyk0x5g92cb9w8xy3v7mm9lm0000gn/T/jl_493nZP/Project.toml`
(jl_493nZP) pkg> add DataFrames, CSV, DecisionTree, Lathe
# Lots of output
(jl_493nZP) pkg> st
Status `/private/var/folders/hw/wyk0x5g92cb9w8xy3v7mm9lm0000gn/T/jl_493nZP/Project.toml`
[336ed68f] CSV v0.9.9
[a93c6f00] DataFrames v0.22.7
[7806a523] DecisionTree v0.10.11
[38d8eb38] Lathe v0.1.8
(jl_493nZP) pkg> add MLJ
# More output
(jl_493nZP) pkg> st
Status `/private/var/folders/hw/wyk0x5g92cb9w8xy3v7mm9lm0000gn/T/jl_493nZP/Project.toml`
[336ed68f] CSV v0.9.9
[a93c6f00] DataFrames v0.22.7
[7806a523] DecisionTree v0.10.11
[38d8eb38] Lathe v0.1.8
[add582a8] MLJ v0.16.5
2 Likes
Hi Peter,
Your last suggestion worked. Many thanks. I’m back in business.
3 Likes