ANN: mybinder.org support for Julia 1.x and Project.toml

Community
1

mybinder.org now supports julia 1.x, including Project.toml and Manifest.toml files, out of the box.

How do you use it? You create a github repo, put a Project.toml and Manifest.toml in the root and add some Jupyter Notebooks to the repo. Then you go to https://mybinder.org/ and enter the URL of your github repository. What you get there is a link that you can for example embed in a web page, share with someone else etc.

What happens if someone clicks on that link? It will automatically spin up a VM instance in the cloud (I believe Google hosted), provision it with exactly the julia environment that is specified in the Project.toml and Manifest.toml, and then it takes you to a hosted version of IJulia notebook where you can run code and do whatever you want. It is like a small, free Julia Box, with the additional option to easily share a pre-configure environment with others! It supports both the traditional Jupypter Notebook, and JupyerLab.

What are some scenarios where this might be useful? Here are three scenarios: 1) You want to provide replication code for a scientific paper, with this setup you can create something that replicates the computational environment exactly that one needs to run your code, and provide a one-click link to a fully provisioned computing environment to run your code. 2) You want to teach a course. You can provision a computational environment that has all the packages pre-installed that you need, and then share that link with the participants of your course, so that they don’t need to install anything to run the course content. 3) you want to provide example code or tutorials for your package, not as static documentation, but as runnable notebooks. I’m sure there are many more use-cases!

All the new Julia support we added over the last couple of weeks should also be available for groups that host their own JuptyerHub and have integrated that with their own BinderHub (for example UC Berkeley provides that kind of infrastructure on campus). In that case you wouldn’t be using the free, public hosted VMs to run stuff, but can use your own computational resources.

This was all exceptionally pleasant joint work between @NHDaly, the awesome Jupyter Binder team and myself.

58 Likes
2

I probably should have included an example of this kind of link :slight_smile: Here we go:

And here is the repo that powers this example:

1 Like
3

It can take a “long” time to spin up such an environment because the installations of packages needs to happen every time a docker image is spun up?

4

Yes, but sometimes it is also super fast. I think there is some caching of docker images going on, including with all packages installed. So if you are lucky and the cached image is still around, you might not have to wait at all. At least that is what I think is going on, I’m actually not sure, though :slight_smile:

1 Like
5

As I wrote here, I think that it takes an awfully long time to create the Docker image and install all the packages. This happens basically after every commit to the repository. Once the image is done, it’s cached and will be immediately ready to use. As I suggested in the linked post, show the log to see what’s going on :wink:

1 Like
6

Minor nitpick: we only build the image once, if the repository doesn’t change the image stays the same and launch times depend on how popular the repository is, how big the image is (5GB images take longer to transfer than 200MB images), etc.

I just wrote a post on how to create Binder launch links, I think a practice that is becoming a “best practice” is to link people to a particular tag/branch and not to master to avoid them potentially having to wait for rebuilds.

10 Likes
7

Thanks for clarifying this, @betatim! I should add that @betatim was the Jupyter Binder team member who shepherded and helped us with all of this, thanks again :slight_smile:

2 Likes
8

Thanks to you and @NHDaly for taking the time to be our Julia experts! We need the contributions from people who use a particular language every day to help get things right and make them feel “Julia-stic” (Is there a Julia equivalent to Pythonic?)

4 Likes
9

Some people use “Julian”.

2 Likes
10

I think I’m actually most excited about the potential to at some point use PackageCompiler.jl to create a custom sysimage with mybinder.org, see this issue.

If we can pull that off, the whole compiled sysimage (that includes the packages that are added to Package.toml) would be cached in the docker image, and I would expect that one could get really good responsiveness that way.

Right now PackageCompiler.jl is probably not stable enough to run it by default, but hopefully that will improve over time. @arnavsood and @jlperla are currently investigating slightly more involved manual ways to pull this off (so this would be a bit more involved on the side of the person that creates the mybinder link, not on the user side).

5 Likes
11

It’s bound to happen at some point. I love using it to compile Plots and GR and DataFrames. Actually I maintain a DataBundle of packages where I save the sys.dll.

12

Would it make sense to search for Project.toml from the notebook directory and up towards the root and use the first one you encounter? Having the project file in the root would then still work, but it makes it possible to have some more fine grained control.

1 Like
13

It might well make sense to change this line so that JULIA_PROJECT=@., I’m just not sure, so any further thoughts on that would be great.

I do think that if we go towards a model where we try to use PackageCompiler.jl by default on mybinder, then it would be easier if there was just the one project in the root, right?

14

Yes. I just think that for the foreseeable future it will require whitelisting packages.

I also like the idea of recursively going up the tree until it finds a manifest. Makes for organizing notebooks in directories easier.

15

Maybe, but then all notebooks in the repo needs to be compatible with the same Julia version and set of packages.

Right, but you can still have just one project file if you want. I’d be in favor of @..

16

Thanks a lot!
Could you help me with the use case:
I have notebook folder in my Pkg and notebooks require some additional packages which are not listed in [deps] part of Project.toml. Can I add those additional packages to mybinder environment , for example, with [extras], [targets] fields in Project.toml? I can add those packages inside of the notebooks with Pkg.add but it makes code less stable to pkg changes

17

Yeah, right now mybinder only instantiates any Project.toml that it finds in the root. I think what you could do is add a Project.toml and Manifest.toml in the notebook folder, and then at the top of the notebook you do something like

using Pkg
pkg"activate ."
pkg"instantiate"

and that might work?

18

Thanks! It solves my problem

19

@davidanthoff Is it possible to configure resources for provisioned mybinder environments. The question is related to this issue
https://github.com/JuliaMatrices/BlockBandedMatrices.jl/issues/45

20

Not that I’m aware of.