My mental load using Julia is much higher than, e.g., in Python. How to reduce it?

[jules]

Hi,

I’m new to the language, and although I like so many aspects of Julia, I am struggling to start using it. I come from Python mostly, but have been looking for an alternative in which low-level function speed can be accomplished easily if needed, but high level implementation speed is also high. Julia seemed to fit the bill, at least on paper. In Swift, I loved how clean my code became because XCode helped my by scanning all my arguments and complaining beforehand that something wouldn’t fit. Also I made tons of use of extending existing objects, for chains like line.mirror(along: axis).drawInContext(ctx) and I thought this would be similarly possible in Julia, when you are already annotating types and often know exactly what to expect in a certain place. (Of course keeping it general wherever possible.)

I mostly do data analysis, experiment design and plotting in Python, or specifically in Jupyter Lab. With all the packages that there are, it’s hard to keep track of all the available functionality. Especially using matplotlib, numpy, and comparable big modules.

So my usual process is iterative. I might plot a figure like this:

fig, ax = plt.subplots(1)
scat = ax.scatter(x, y)

Now I have the fig and ax, and scat objects, which of course have tons of methods bundled. So often, I will type scat. and then tab to give autocomplete suggestions. This way I can really quickly scan the available functionality, basically only stuff that has to do with this specific object. I might learn that there is a function to set marker color, or whatever.

In Julia, this is much harder. Because methods never come bundled with the kinds of objects they’re meant to be used on, I see myself in front of a sea of functions without an idea what I have available. I know that there is methodswith, but this gets tiresome pretty fast if you have to use it a lot. My mental load is always relatively high, trying to remember, what was this function that I could call with a DataFrame as the first argument that did something related to aggregating, instead of doing df. and filtering the suggestions quickly.

In many code examples I see that people are pulling tons of functions into the global namespace with using, which is comparable to do from xxx import * in Python. This is pretty bad practice in my opinion, because reading the code you can often not know where a certain function being called is from, you would have to execute and trace which method was dispatched.

How do you deal with these issues, maybe there are better workflows that I’m not aware of? This together with the sometimes very long precompilation times make Julia hard to justify for myself, even if I love the type mechanisms and would like to harness the high performance.

Thanks for your suggestions!
Julius

[pdeffebach]

Are you using ? liberally in the repl to get help?

[chakravala]

@jules

In many code examples I see that people are pulling tons of functions into the global namespace with using , which is comparable to do from xxx import * in Python. This is pretty bad practice in my opinion, because reading the code you can often not know where a certain function being called is from, you would have to execute and trace which method was dispatched.

You can do import PackageName instead of using PackageName, then all the methods have to be prefixed with PackageName.__method_name.

[cstjean]

@jules

In Julia, this is much harder. Because methods never come bundled with the kinds of objects they’re meant to be used on, I see myself in front of a sea of functions without an idea what I have available. I know that there is methodswith , but this gets tiresome pretty fast if you have to use it a lot. My mental load is always relatively high, trying to remember, what was this function that I could call with a DataFrame as the first argument that did something related to aggregating, instead of doing df. and filtering the suggestions quickly.

Yeah, the problem of method discovery has been discussed before, but I don’t think that a solution has been found. You just have to get used to reading the documentation. It’s one of the drawbacks of multiple dispatch.

[rdeits]

I think these are all reasonable points, but I hope I can help a little bit.

@jules

I made tons of use of extending existing objects, for chains like line.mirror(along: axis).drawInContext(ctx) and I thought this would be similarly possible in Julia

I think what you’re talking about is “fluent interfaces”, and it’s true that we don’t really do that in Julia, at least not in the same way. You might find the |> operator useful, since you can do:

x |> f |> g

as an alternative notation for g(f(x)). This should become even better when we (eventually) get RFC: curry underscore arguments to create anonymous functions by stevengj · Pull Request #24990 · JuliaLang/julia · GitHub which would allow you to write:

line |> mirror(_, along=axis) |> drawInContext(_, ctx)

which is pretty close to the fluent interface but is in some ways even better. Notably, in a fluent interface, there is no way to chain a function that isn’t a method of the returned object. So if sort() is a generic function (and not a method of whatever drawInContext returns), then you can’t do:

line.mirror.drawIncontext(ctx).sort()

but in Julia, you certainly can:

line |> mirror |> x -> drawInContext(x, ctx) |> sort

or, in the future:

line |> mirror |> drawInContext(_, ctx) |> sort

This actually gets at something that I definitely struggled with when coming from Python, and which you’ve mentioned as well, which is that it can be hard to figure out what you’re supposed to do with a given object in Julia. It’s true that in Python I would often type ax.<tab> to get some sense of what I can do with an object. But there’s a downside of that: the things you see when you do ax.<tab> are almost never a good representation of all the things you can do with ax, they’re just the methods it happens to define.

For example, given a list in python, we can sort it with its sort method and reverse it with its reverse method. But what if we want to enumerate it? Well, that’s actually enumerate(x) instead of x.enumerate(), but you’d never find that from x.<tab>. And what if we want a reversed copy of the list, instead of doing it in-place? That’s reversed(x), but not x.reverse(). In Julia things are more consistent: we have enumerate(x) and sort(x) and sort!(x). One isn’t somehow more special than the other by virtue of being a method rather than a generic function.

I do agree that methodswith is kind of an awkward tool to use–this seems like exactly the kind of thing that better editors and interfaces will help with. We’re not there yet, but I think it can be done.

@jules

In many code examples I see that people are pulling tons of functions into the global namespace with using , which is comparable to do from xxx import * in Python. This is pretty bad practice in my opinion, because reading the code you can often not know where a certain function being called is from, you would have to execute and trace which method was dispatched.

Yeah, I don’t love this. Inside packages I try to have between 0 and 1 plain using Foo statements, with all other imports done explicitly as:

using Foo: bar, baz

which makes it easier to see where things come from. However, it’s worth noting that using is not nearly as bad as from Foo import * would be in Python. from ... import * in Python is bad because:

1. It makes it hard to tell where things come from
2. It can silently replace things in your current namespace and break your code in crazy ways. If package Foo has a function called sin and you do: from Foo import *, you’d better hope you weren’t relying on some other definition of sin.

Issue 1 still applies to Julia, but issue 2 does not. Julia won’t let you accidentally blow away things in your current namespace just by using some package:

julia> module Foo
       export sin
       sin(x) = 1
       end
Main.Foo

julia> using .Foo

julia> sin(5)
WARNING: both Foo and Base export "sin"; uses of it in module Main must be qualified
ERROR: UndefVarError: sin not defined

for that reason, I think using in Julia is perfectly acceptable, but I agree that we should limit our use of it in packages.

[PetrKryslUCSD]

Excellent advice!

[ssfrr]

@rdeits

for that reason, I think using in Julia is perfectly acceptable, but I agree that we should limit our use of it in packages.

Also in code examples. When an example has several using Foo statements up top, it’s really hard to know which parts of the example come from which package. Sometimes you can figure it out from context, but if you’re using a few packages that are related to each other it’s often ambiguous.

[airpmb]

@jules

Now I have the fig and ax , and scat objects, which of course have tons of methods bundled. So often, I will type scat. and then tab to give autocomplete suggestions. This way I can really quickly scan the available functionality, basically only stuff that has to do with this specific object. I might learn that there is a function to set marker color, or whatever.

Same here, I miss that functionality from Python (or even other languages in IDEs). I wonder if a simple improvement could be to make tab in the REPL show the results of methodswith, at least if done right after the mention of a defined symbol. When one of the offered methods is chosen the REPL then it would insert the method (including parens) around the symbol.

[rdeits]

Yes, I think that could absolutely be done and would be quite helpful. All that’s needed is for someone to want it badly enough to do it

[ssfrr]

Juno already does something similar (shows a list of possible argument types for a given function). It even re-displays the list (filtered by the type of the first argument) after you type , and are ready to write the 2nd argument. Very slick.

[stevengj]

@airpmb

I wonder if a simple improvement could be to make tab in the REPL show the results of methodswith, at least if done right after the mention of a defined symbol. When one of the offered methods is chosen the REPL then it would insert the method (including parens) around the symbol.

I’m not sure I would want to see this every time I hit tab, but one option would be to do it when you tab complete (symbol, or maybe just symbol, … requiring a comma would cut down on accidental calls to methodswith, since you are unlikely to want to tab-complete a comma normally.

[airpmb]

Which is great but you still need to know the function name in the first place. What I suggested would leverage the existing methodswith in a way that would help discovery even more as it would as has been pointed out provide methods that work with the type of the symbol at the cursor and not just those that belong to a class instance.

Another related idea: in the REPL if you type “something?” (Or maybe just hit return) then a combined summary of something’s type as well as its methods pops up. A lot more work than the fist suggestion to be sure, just brainstorming here. I’d be willing to help implement some of this stuff with guidance btw.

[airpmb]

Fair point. My suggestion would be since this is a newbie oriented feature to make the default whatever is most useful for such a user but configurable to the needs of more experienced users.

[antoine-levitt]

I agree that the absence of easy tab completion really hurts discoverability. See https://github.com/JuliaLang/julia/issues/30052. I had a working patch that did method completion, but a problem is that many methods just accept Any, and it’s difficult to know what the user wants from just “tab”. The code to get completions is not complicated, but someone needs to sit down and figure out a proper UI for this.

[tkf]

@jules

So often, I will type scat. and then tab to give autocomplete suggestions.

I did Python programming a lot before using Julia so I missed this feature too. That’s why I made ANN: InteractiveCodeSearch.jl --- Interactively search Julia code (sorry, no intention for a plug). I don’t think object.<hit tab> is the only interactive method discovery UI. InteractiveCodeSearch is a quick hack at it but I think we can do more. For example, it would be nice to have a keyboard shortcut to find a minimal subexpression surrounding a cursor, inference the type of it, and then run @searchmethods on it.

(As for the star import, @rdeits did a perfect answer so I have nothing important to add.)

[Ken-B]

@jules

Now I have the fig and ax , and scat objects, which of course have tons of methods bundled. So often, I will type scat. and then tab to give autocomplete suggestions. This way I can really quickly scan the available functionality, basically only stuff that has to do with this specific object. I might learn that there is a function to set marker color, or whatever.

Obviously not a real substitute, but I sometimes type e.g. Plots.+TAB to see what methods and types are defined in a particular package.

[Tamas_Papp]

@ssfrr

When an example has several using Foo statements up top, it’s really hard to know which parts of the example come from which package.

In “scripts”, ie non-reused, non-packaged code, mostly for interactive exploration, I prefer just plain using Foo, and stick to using Foo: bar in packages (yes, even with LinearAlgebra ). I think both are useful, and looking up the module of a function is rather easy.

[jules]

Thanks for the responses everybody, I see that I’m not alone with my struggles

I think that an autocomplete feature could suggest methods on variable. + TAB (to mimick the common form of acessing bound methods) and then a list appears that shows all methods that operate on objects of this specific type, or maybe supertypes other than Any. The Any methods could come after the others, so they don’t crowd everything. And upon selection this would be transformed to method(variable. Of course this is more of an IDE thing anyway…

The underscore anonymous function idea is great! I hope this gets merged soon, it seems to be stuck in discussion currently… Right now piping is not so useful, it being restricted to the first argument.

[Mason]

The underscore anonymous function thing will be nice, but in the mean time you should maybe check out Lazy.jl for its threading macros. They can be quite useful. In particular, the @as macro essentially does this with more control:

# @as lets you name the threaded argmument
@as _ x f(_, y) g(z, _) == g(z, f(x, y))

[tk3369]

This is really a tooling issue. I’m no expert in jupyter or Juno, but theoretically it would be possible to make jupyter/ide to capture <variable>.TAB the same way and display methods that can operate on the variable, and when it’s chosen it just need to replace the whole expression? Someone please correct me if I’m wrong.