Interactive prototyping workflows

New to Julia
,
21

Whenever I run into this problem, I realize the solution is basically always to use Pkg.generate and make an actual package, then have a scripts folder in addition to src which has scripts. That way you can just have a single using (which itself has lots of exports). You can also do this somewhat hacky way to export all names from a module

julia> module MyMod
           w = 5
       end;

julia> using .MyMod

julia> function exportall(mod)
           for n in names(mod; all=true)
               if Base.isidentifier(n) && n ∉ (Symbol(mod), :eval, :include)
                   @eval mod export $n
               end
           end
       end;

julia> w
ERROR: UndefVarError: `w` not defined in `Main`
Suggestion: check for spelling errors or missing imports.

julia> exportall(MyMod)

julia> w
5

You can also do REPL.activate(MyMod) and work within your module, obviating the need for too many MyMod.xxx prefixes.

2 Likes
22

I think Pluto.jl is the best way to go. If you need write modules, you can use Revise.jl in another session, then reload your module in Pluto.jl. It will work following your specs.

2 Likes
23

I didn’t understand these so I just repeated them for myself.

julia> run(`cat up.jl`);

function foo(x::Vector{Float64}, a::Float64)
    return x .+ a
end


julia> include("up.jl")
foo (generic function with 1 method)

help?> foo     # there is one binding for foo
search: foo for floor Bool fdio

  No documentation found for private binding Main.foo.

  foo is a Function.

  # 1 method for generic function "foo" from Main:
   [1] foo(x::Vector{<:Real}, a::Real)
       @ /tmp/up.jl:2
       
julia> foo([1.0], 1.0)
1-element Vector{Float64}:
 2.0

julia> run(`cat up.jl`);    # change the type signature

function foo(x::Vector{<:Real}, a::Real)
    return x .+ (a / 2)
end


julia> include("up.jl")
foo (generic function with 2 methods)

help?> foo    # now there are *two* bindings named foo.
search: foo for floor Bool fdio

  No documentation found for private binding Main.foo.

  foo is a Function.

  # 2 methods for generic function "foo" from Main:
   [1] foo(x::Vector{Float64}, a::Float64)
       @ /tmp/up.jl:2
   [2] foo(x::Vector{<:Real}, a::Real)
       @ /tmp/up.jl:2

julia> foo([1.0], 1.0) # calls the older "hidden-state" definition because Float64 is more specific than Real
1-element Vector{Float64}:
 2.0

julia> foo([1.0], 1) # calls the intended function
1-element Vector{Float64}:
 1.5

I think I can live with this. Overwriting definitions works without magic, I just have to remember that the type signature is an essential part of the name of a function, and Julia’s polymorphism means you can fall into calling a ghost if you’re not paying close attention.

Thanks for the clarification!

Iiiiinteresting. That’s the sort of insight I came here for. I’m curious how common that is. The workflow tips mention writing a package as an option but I figured that was for, you know, writing utilities to share, not a specific data analysis.

If you use Pkg.generate then I guess Revise works perfectly? Or, not, I guess it still doesn’t re-evaluate variables whose definitions haven’t changed even if the inputs to those definitions have.

Thanks for all your contributions everyone!

24

If you use Revise with this Pkg.generate workflow, it will re-load all functions that are defined, but global variables in your script will not be updated. So you would still need to do include("scripts/my_script.jl") to do re-run, and this will pollute your namespace (which can be good or bad depending on your needs).

PS: If someone sent me a Pluto.jl notebook I would be pretty annoyed. Its a pretty bespoke development environment and I would much rather just get something I can easily run in the terminal.

1 Like
25

I envy you if you sure you can. I have on many occasions teared my hair out before “just remembering” that I constructed that trap with my own hands.

The behavior of Revise is the same whether you using a package, or includet a script.

I don’t know how common, but that’s what I generally do if I have anything more than a few lines. Just I’d use PackageMaker.jl (being author of it) instead of Pkg.generate; see also general considerations concerning workflows in it’s documentation as to why it’s better to start with package right from the beginning. There is one more bulletpoint to be added: With a package for every task, all dependencies are installed there and not into the main environment, reducing the risk of dependencies conflict.

If you need to play around with global variables, you can use the trick as proposed above by lmiq - here one more variation:

# in the package body
my_data() = (;a=1, b="b", c=[1,2,3])

# in a script
using MyPackage
using MyPackage: foo

(;a) = my_data()
answer = foo(a)
1 Like
26

Ooo okay. Noted. I’ll see how far I get. Maybe I will find myself bald in six months.

27

The hair grows back :grin:

1 Like
28

Thank you for launching this discussion. I am also in search of a better workflow. I apologize for the long message, but I thought it would help to explain how I struggle with basic things - after all, this is a “New to Julia” forum.

I mostly work in VSCode, building scripts and testing them interactively by executing lines or sections one at a time, and from time to time restarting the kernel to make sure I am not in some parallel universe. When something gets to a state where it needs to be “properly” documented or shared, I tend to switch to Jupyterlab, but I never use it for my own work, I find VSCode scripts with the REPL always ready more convenient. I also transferred from Matlab the habit of saving intermediate results when one part of the analysis works, so I will sometimes end-up with scripts like “step1”, “step2”, etc. Very occasionally, I think some functions I created could be useful to someone else or the future me, so I try to code them properly and perhaps put them in a module.

I have played with precompiled sysimages in the past, when it was (for me) reducing the frustration a lot, but I find that they are unnecessary now (again, to me!).

The main problem for me to switch to a “better” workflow that would not do everything in Main is debugging. I have given up on emulating C/Fortran debugging (as in “F10 / F11” in Visual Studio). Infiltrate is working well for me, but I find it’s polluting my code with commented out (or just “forgotten”) @infiltrate instructions, plus it requires me to switch from running lines or sections to running the whole file with include(), so it’s not super convenient.

One strong limitation of my “fully interactive approach” is loops, do blocks, etc. that create their own scope and cannot be evaluated line-by-line. What I am doing - which seems stupid - is to replace loops by manual iterations (as in having a line that says i = i+1, running that, running the section of code that should be in the loop, and then finding the problems with the next value of i). Once the code works for “many values of i”, I put back the actual loop, perhaps move the code to a function, and move on to the next part of the analysis. I typically have to change some stuff because of the scope behaviour but that’s OK.

I have not been able to find instructions that explain to someone like me how to adopt a better workflow. Back to the feeling that I need a “(really) New to Julia” forum…

If a generous soul has read this far and would have ideas for a “light approach” to adopting a better workflow, going one step at a time, i.e. adopting something that is not yet “good” but at least “better”, that would be really great!