Jupyter Magics Proposal/Feature Request

Tooling Jupyter-Notebook
1

Background

I am aware of IJulia’s decision on magics. From their docs:

One difference from IPython is that the IJulia kernel does not use “magics”, which are special commands prefixed with % or %% to execute code in a different language. Instead, other syntaxes to accomplish the same goals are more natural in Julia, work in environments outside of IJulia code cells, and are often more powerful

I do agree we do not need jupyter magics in julia. However, I do think they provide a nicer syntax, and as a result, they are easier to read. I am trying to develop a better calc process for many who do not know how to program and do not want to learn how either.

In my experience, as soon as these sorts of people see indented code, they tend to disengage.

Example - Python vs Julia

image
image1435×552 25.8 KB

You can see the python version of the notebook on the left is “easier” or “less codey” compared to the julia version on the right. In fact, if you exclude the import cells, to someone who doesn’t know how to code, the cells don’t even really look like code.

That is unfortunately not the case for the julia version, and I fear that it could hinder adoption.

Proposal

My request would be to include a magics syntax for the %% magic. This would be a cell based magic and would be more of a syntactic sugar. The %%name magic (I am fine with another symbol but it would be nice for those python users who are comfortable with this syntax) would essentially be a @macro_name begin ... end. See below:

Example 1

# imagine this as a jupyter cell with julia magic
%%handcalcs
a = 2
b = 5
c = 2

would be equivalent to:

# imagine this as a jupyter cell without julia magic
@handcalcs begin
    a = 2
    b = 5
    c = 2
end

Example 2

# imagine this as a jupyter cell with julia magic
%%handcalcs cols=3
a = 2
b = 5
c = 2

would be equivalent to:

# imagine this as a jupyter cell without julia magic
@handcalcs begin
    a = 2
    b = 5
    c = 2
end cols=3

This example may be a little harder to do since it is out of order compared to macro order. It may also be a sort of unique to me issue since not all macros are like this.

Conclusion

I just wanted to put this out there and see what people think. I know it could be more niche to certain use cases, but I can see it being somewhat nice for the @chain macro as well so I thought I would put this out there.

I know this would bring more complexity too for jupyter to .jl conversions and may not be worth the extra complexity. I am mainly just trying to get more adoption in the company, and I would rather them choose julia over python.

1 Like
2

Just some thoughts …

Well, for that use case I would consider Pluto.jl to be nicer than Jupyter as it automatically updates whenever you change something – you could even sell it as being “like Excel, just better” :wink:

This can already be written as

@handcalcs begin
a = 2
b = 5
c = 2
end

In contrast to Python, whitespace/indentation is (mostly) not required in Julia.

Disclaimer: I do like Lisp and cannot understand the obsession with superficial syntax, so take my word with a grain of salt.

3

Oh yes that is definitely the main goal. I would rather use Pluto (and plan to compare it to excel haha). However, currently there is no quarto support and not a great way to automate things (I think) in Pluto. Whereas in Jupyter I can just use papermill and easily insert parameters and print things off using quarto and have multiple notebooks.

And yeah I have tried the non-indent way, and it still doesn’t look as good as a Jupyter magic. I will most likely be demonstrating the process using both python and Julia, and I just know this is a +1 for python.

I will do my best to show the advantages of Julia but I will just have to see. I still have months before I do any deonstrations though.

4

If slight changes in spelling from what your audience is used to will cause them to walk away, then you are sunk no matter what you do. Julia syntax is not going to look like Python.

(Not that “magics” are Python syntax, either, which to me is a strike against Python.)

(Hangups over surface syntax are part of why I tell my colleagues at MIT that it’s a disservice to students for us standardize on any one programming language for undergraduate education. Learning their first programming language was so hard, students assume that learning any second language will be equally hard. If someone knows only one programming language, I basically assume they don’t know how to program.)

11 Likes
5

They are used to excel, so it’s not really a slight change but a big one to them.

I have demonstrated this tool for a small handful of people, and there is a Wow Factor that I have seen on their faces when I show them the python version. They really do see it as magic (Jupyter magics seem to have that affect I guess).

When I led with the Julia version, I can tell they are impressed but are resistant. It has been a very noticeable difference that sort of sets the tone for the rest of the demo. I can usually make them lean slightly toward Julia after I bring up some of the great things about Julia, but it can take some convincing!

These people don’t have a bias toward python or Julia, because they don’t use either. I just want to hide away some of the code and make it feel more like magic. That way users who don’t know programming can still feel comfortable using the tool. I think Jupyter magics and smart cells from livenotebooks are doing things with these sorts of things in mind.

1 Like
6

We could debate this to infinity, but it does not sound very hard to do. Just do it.

The function to intercept the the magic is here:

7

The obstacle isn’t the implementation, it’s having that implementation accepted.

For what it’s worth, I think adding magics that just expand to macros would be a nice touch and I’d give it a thumbs up, but also it’s not my choice if it makes it in or not.

8

IJulia is not a fast moving package, and I’m quite sure that a decent implementation of this is shorter than this thread.

On one hand, I don’t think we need it, but I also do not see it doing much harm. Other kernels than IPython also have magic functions. I occasionally maintain a fork of the IJava kernel where magics are used.

9

The implementation isn’t the obstacle, that’s the easy part. The hard part is having such a PR accepted. The IJulia owners are not interested in accepting magics. The package could be forked, but then we’d end up splitting community resources on an already borderline inactive package.

10

Bearing in mind that I don’t know anything about how Jupyter/Pluto cells are scoped, a possible way forward would be to avoid magics altogether and just create an IJulia specific macro. For instance, take your Example 2 and replace %%handcalcs with something like @handcalc_cell where the new macro wraps everything in a begin ... end block.

The advantages to this approach would be: 1) it uses standard Julia macro syntax and would be a more natural extension, 2) it would be possible to implement entirely within HandCalcs.jl so it wouldn’t rely on convincing others to make major changes to their packages, and 3) if others want to do “magic”, then maybe interested devs could design a notebook-agnostic API where the same syntax could work in Pluto and Jupyter (and others?) instead of a %% syntax for IJulia and a different one for Pluto magics (if that were a desired thing).

I don’t know if this is reasonable to code or not, but seems doable on the surface.

1 Like
11

Cool. Maybe I will work up a PR when I get some time just to at least give it a shot.

I would like an opinion on Example 2. I think that is where things aren’t quite correct or could be better.

Compare this:

With what I can imagine how @chain could be used

%%chain df
dropmissing
filter(:id => >(6), _)
groupby(:group)
combine(:age => sum)

which is equivalent to:

@chain df begin
  dropmissing
  filter(:id => >(6), _)
  groupby(:group)
  combine(:age => sum)
end

Now chain could be written as

@chain begin
  df
  dropmissing
  filter(:id => >(6), _)
  groupby(:group)
  combine(:age => sum)
end

so maybe it is not a very difficult since you can just do:

%%chain 
df
dropmissing
filter(:id => >(6), _)
groupby(:group)
combine(:age => sum)

but you can see in first chain example how that would be different and you would need to parse that differently as a result. After writing this though, I do think Example 2 is the best way, since passing a n number of kwargs… is probably pretty common. I have to admit my bias there though.

12

I am not sure if I follow. Are you suggesting something like this?

@handcalc_cell
a = 2
b = 5
c = 2

I can not write code for this (in my package) to see what is beneath it in julia. IJulia would have to code for this since it has knowledge of cells.

13

All I’m saying is that I would be interested in seeing the implementation. Whether the PR is accepted, or a fork is needed seems premature. OP is asking for community interest, not just PR prospects.

Maybe the causation is flipped here? Maybe the lack of activity and community resources is because we are too quick to dismiss ideas like this.

As a member of the JuliaPy Github organization and a conda-forge maintainer, would be interested in seeing an implementation.

Considering whether to accept the PR or fork does not seem like the immediate decision nor does it seem to me that the OP is necessarily asking about that. That said because of the current pace of IJulia development, a fork with this feature would not be difficult to maintain.

I would hope that we can could consider an extension mechanism that might permit this to be implemented as a plugin.

Note that we currently have _str and _cmd suffixes for string and command macros. He is recommending we create a new class of macro such that %%magicnamehere maps to @magicnamehere_cell. While this might require some glue macros, it does seem like a good suggestion. It helps with the discovery problem. We can now ask does @magicnamehere_cell exist, if so proceed with it; otherwise, default to the current help behavior.

For @handcalc we could just create an alias with a _cell suffix.

4 Likes
14

Oh I see now. Thanks for the explanation. I was aware of _str but not _cmd (I will have to look into that one). Yeah that would be very cool if not better. If it would work in pluto too that would be awesome! This method seems a little bit above my expertise to look into though :sweat_smile:

1 Like
15

Yeah, as @mkitti described also, I was essentially suggesting that instead of cell magics, just trying to do it with Julia macros.

^^ This is the part in which I have no knowledge, so I don’t know if my suggestion is feasible.

1 Like
16

Imho, when they know neither Python nor Julia why show both? In the end, they can’t make an informed decision which to use anyways – especially not based on the syntax of some first examples alone. Appears that showing Python and Julia side by side is just a distraction taking away time from your presentation.

Regarding tooling, both Pluto and Jupyter will have their merits:

  • More Excel like, +1 for Pluto
  • Larger community and handy infrastructure, +1 for Jupyter
  • …

In any case, many things that require magic in Jupyter are just a macro away in Julia. For what it’s worth, here is a small Pluto notebook which can also be run as a script with parameters passed via the command line:
pluto.jl (2.3 KB)

2 Likes