Using Julia in Binder: interactive web environment for running your code

Update: Binder now fully supports Julia v1 and its Project.toml and Manifest.toml, which make creating reproducible environments even more easy and reliable. See here for the announcement.


Today I’ve been introduced to Binder. I found a few references to this tool on Discourse but not a topic on its own devoted to Binder, so I thought that it could be useful for some people to explain what it is and how to use Julia with it.

Introduction

Binder is a website where you can run an interactive environment (basically a Jupyter server) created out of a repository. Under the hood, it creates for you a Docker image of your repository that you can use remotely. This is meant to be as a tool to deliver reproducible research.

HOWTO

Binder workflow

You only need to create a repository (e.g., on GitHub, but probably any other public source hosting website should work) with your code. The REQUIRE lets Binder know that you want to use Julia. I’m not sure what exactly is the syntax expected by Binder, for example it doesn’t seem to need a julia line (I tried to use it but it failed to create the environment, not sure this was the problem but didn’t try again because it’d have taken too long).

Once you create a repository with your code, go to https://mybinder.org/, insert the information about your repository and click on the “launch” button. And wait. That’s all.

Very simple example

I’ve created an almost useless example here: GitHub - giordano/my-first-binder at julia. The Binder is at Binder A Jupyter server will be opened. From there, you can either open a terminal where to run the code (well, the hello.jl does very little) or create a new notebook.

Personal comments

First of all, following the current documentation (2019-03-12) I was able to run only Julia v0.6.4 with Binder. I see that support for Julia v1+ is coming, hopefully quite soon. This should include also support for the new Project.toml and Manifest.toml files. Probably there is already something in place but the documentation needs to be updated?

One of the main goals of Binder is to create a fixed environment where to run your code in a reproducible way. This is achieved by enforcing the use of REQUIRE file for Julia or the requirements.txt file for Python. Julia v1 already does a great job at creating fixed environments with the use of the TOML files which are truly part of code loading, contrary to, e.g., requirements.txt for Python. However, the nice feature of Binder is to provide an interactive environment where to run the code remotely.

Note that this service comes free of charge, so do expect occasionally long waiting times, especially when creating for the first time an image for a given revision of you repository, this operation can take several minutes (more than 10). Open the log when loading a Binder environment, at least you can see what’s happening.

Despite some annoying issues with slow creation of the image, Binder looks like a great option for sharing your code with other people and let them play with it, even without having to install Julia themselves. With full Julia v1 support this can be a great resource for doing reproducible research.

Further reading

You can find Binder documentation at Binder Documentation — Binder 0.1b documentation. Several examples of Binder repository are at Binder Examples · GitHub

12 Likes

There are a few tools like this now. I am tempted to test all of them out and write up something. But I got hands full for the next year or so. But the others I found interesting are

  • Kaggle kernel
  • Google colab
  • Nextjournal
  • repl.it
  • binder
  • juliabox.com
3 Likes

Does anyone know if the binder architecture might support a custom system build (a la https://github.com/JuliaLang/PackageCompiler.jl)?

@NHDaly, the incredibly awesome jupyter binder team and I recently got the whole julia 1.0 story pretty much figured out and merged for binder ([MRG+1] Split julia support into pre julia 1.0 and post julia 1.0 by davidanthoff · Pull Request #595 · jupyterhub/repo2docker · GitHub). There are still a few minor things we are ironing out right now (mostly waiting for Set JULIA_PROJECT="@." for kernel by davidanthoff · Pull Request #820 · JuliaLang/IJulia.jl · GitHub to merge so that we can adopt that model for binder as well), and then we were going to deploy it on the public binder infrastructure and I was going to make a big announcement :slight_smile:

I think I would actually recommend that folks don’t adopt binder at this stage using REQUIRE files, but instead just wait a few more days/weeks until we have the proper Project.toml/Manifest.toml story deployed. That will be much more robust, in particular when it comes to reproducability.

Docs for the new Project.toml support are all written, but I think they will only deploy when the software is also deployed to the public binder infrastructure.

Compile custom julia sysimage · Issue #601 · jupyterhub/repo2docker · GitHub. I don’t see any reason why this shouldn’t work, assuming that PackageCompiler.jl is stable enough.

15 Likes

Hooray for Project.toml! Hooray for reproducability! Hooray for free notebook hosting!

And hooray for your imminent announcment!! :smile: Thanks again for all your awesome work on that @davidanthoff! I think it’ll be really useful!

Thanks also to @kristoffer.carlsson for reviewing a lot of the julia 1.0 stuff in the binder PRs.

4 Likes

Yeah, I saw great activity in the repo2docker repository about support for Julia, thank you so much! I’m happy to see that Julia v1 is landing on Binder.

https://julialang.github.io/Pkg.jl/v1/environments/

1 Like

Hello,

I’m wondering what is current status of Julia support in Binder.

I have a Jupyter notebook which I uploaded as a Github Gist to https://gist.github.com/scls19fr/172c351557721e8851b828018738b4c0

it’s viewable “statically” in nbviewer
https://nbviewer.jupyter.org/gist/scls19fr/172c351557721e8851b828018738b4c0

I can see in nbviewer an “Execute on Binder” button…
but the kernel Julia 1.3.1 can’t be used there (currently only Python 3 kernel can be used).

Any news?

Kind regards

In my latest Julia workshop Binder was working just fine with Julia 1.3.1 (apart from the fact that it takes a really long time to rebuild the docker image after a new commit). You should be able to click on the “launch binder” badge and should see a functioning jupyter notebook environment in less than a minute.

I suspect that since your code doesn’t come with a Project.toml/Manifest.toml Binder assumes that it is a python project and only shows you the python kernel.

1 Like

Yes, that is the key: you need a Project.toml and Manifest.toml in the root of your repo, that triggers the Julia install!

Are both files required? I’m wondering if just Project.toml can be enough
Should I get a uuid for my notebook? (I thought that was only required for package) how should I get it?

Dependencies are CSV.jl, Polynomials.jl, Plots.jl.
How can I get there uuid? (except by going to there Github repository) How can I know the uuid of a registered package? - #6 by Raf doesn’t really answer that.

I can’t answer the first question, but David’s answer suggests you need both. Fortunately this is easy with Pkg!

You don’t need a uuid for your notebook, nor do you ever need to worry about uuids – julia’s Pkg handles all that complexity for you.

The right workflow here is:

  1. Start julia in your project directory (so that Julia’s working directory is your project folder; you could alternatively just cd from within Julia, whatever is easier).
  2. Change to Pkg repl (type ])
  3. Run activate . – this creates a new project & environment in this directory… You’ll notice your Pkg REPL prompt will now be the name of the folder, i.e.
(foobar) pkg>
  1. Add the packages you need for your project:
add CSV, Polynomials, Plots

Pkg will do it’s thing for a little while.

You now have a Project.toml and Manifest.toml in that folder! And they have all the entries and uuids you need.

Unless you are developing a package, you never need to manually edit a Project.toml (and no one ever needs to manually edit Manifest.toml).

This is the work flow I use for every project. Each project is in it’s own folder, and I activate . each time I start julia so that each project has its own “environment” of packages.

I like having per-project environments. This is very different to e.g. R, where packages are installed globally. It has some frictions - you have to remember to add a package before you can use it in a project. But I find it actually makes my life much better (once you get used to the idea that ‘installing’ a package for one project doesn’t mean it’s available for every project). Installing a package for one project doesn’t cause havoc with other unrelated projects (had this happen with R, where installing or updating a package for one project ruins another one. R has come up with packrat to try to solve this – but Julia is greedy, so Pkg just built-in this in from the start.)

2 Likes

As it’s a notebook that I wrote in JuliaBox I haven’t used that workflow but I created manually a Project.toml file
A uuid is not required for notebook but I have found uuid of dependencies in Github repository of each dependency.
Such a Project.toml seems to be enough to be able to execute notebook with MyBinder (ie a Manifest.toml file doesn’t seems to be required).
See Binder