What's the best practice in Julia for research (often changed codes)?

Hi, I’ve used Julia about one year and I still struggle with how to use it.

First of all, as a researcher, I often change my code including code structure and Julia types.
When redefining types, Julia REPL needs to be restarted and I should spend quite a long time for precompilation.
In my experience, the actual development speed is fast only for almost-completed codes.
For some projects on progress, I always restart and precompile the whole code, which is so annoying.

What’s the best practice for often changed codes?

Note: I usually create my own package and add almost every functionality in the package for each project. Also, I use Revise.jl.

1 Like

Some options to deal with the problem of Revise not being able to track type changes are discussed in some threads. For example:

2 Likes

Well, I don’t think it is a cool solution.
It sounds like, I have to replace all structs to NamedTuple for often-changed codes.

Of course, struct may not be changed frequently when the development becomes mature.
Ironically, if the project becomes mature and huge, precompilation would take a long time when I realise that the struct should be changed.

EDIT: Apart from Revise, is it impossible to enable type redefinition in Julia in principle?
It would be really great if someone tells me something about this functionality :slight_smile:

Theoretically in the future, revise should be able to support this. The hard part is figuring out what to do with the old type. If you have variables that are of the old type, do you delete them? What about specialized functions?

2 Likes

I don’t know how to “delete” certain objects. When I’ve searched it, many people recommended to just restart REPL, but my memory might be wrong.

If you don’t mind, can you describe what you meant in detail? For instance, with an example.

And I really hope that Revise once supports this functionality.

Here’s the basic problem. Suppose I define

struct Foo
    x::Float64
end
bar = Foo(1.0)
baz(x::Foo) = Foo(x.x -1.0)

If I now redefine Foo as

struct Foo
    y::Int
end

what happens when I try to call baz(bar)?

2 Likes

Well, I’ve tried it but encountered the redefinition error.
First, run the following (by include("the_file_name.jl").

## Type redefinition test
struct Foo
    x::Float64
    # y::Int
end
bar = Foo(1.0)
baz(x::Foo) = Foo(x.x - 1.0)
@show baz(bar)

and changed the code to the following and re-run it.

## Type redefinition test
struct Foo
    x::Float64
    # y::Int
end
bar = Foo(1.0)
baz(x::Foo) = Foo(x.x - 1.0)
@show baz(bar)

It gives

julia> include("test/tmp.jl")
ERROR: LoadError: invalid redefinition of constant Foo
Stacktrace:
 [1] top-level scope
   @ ~/.julia/dev/MyPackage/test/tmp.jl:2
 [2] include(fname::String)
   @ Base.MainInclude ./client.jl:444
 [3] top-level scope
   @ REPL[1]:1
in expression starting at /Users/jinrae/.julia/dev/MyPackage/test/tmp.jl:2

Sorry for the confusion, my point wasn’t describing something that currently works. It was about what would happen if redefinition were supported. Basically, it’s unclear how redefinition should behave, so it currently errors.

4 Likes

My approach:

  • Don’t use types for “interactive” coding unless you really have to. I’ve come to appreciate that the types provided by Base / other people’s packages are sufficient much more often than one might think.
  • If you do have to create your own types, put them into a module MyModule and don’t do using MyModule. Instead, either introduce an abbreviation M = MyModule and type M.my_func() while prototyping, or do something like in the following MWE:
julia> # Initial definition
       module MyModule
       export MyType
       struct MyType
           x::Int
       end
       end;

julia> module Foo
       using ..MyModule
       @show MyType(1)
       end;
MyType(1) = Main.MyModule.MyType(1)

julia> # Redefinition
       module MyModule
       export MyType
       struct MyType
           x::Float64
       end
       end;
WARNING: replacing module MyModule.

julia> module Foo
       using ..MyModule
       @show MyType(1)
       end;
WARNING: replacing module Foo.
MyType(1) = Main.MyModule.MyType(1.0)  # Look ma, I changed Int to Float64!
5 Likes

Another option if you don’t like named tuples is to just call your struct MyStruct_1 and if you need to change it, do a find replace on your code base to Mystruct_2.

1 Like

One slightly simpler version of this is to use something like MyStruct=MyStruct_1, and then you can redefine what struct MyStruct is an alias of.

3 Likes

Could a macro do that?

@numbered_struct struct Foo
x::Int
y::Int
z::Int
end

abc = Foo(1,2,3)

expanding to

struct Foo_1
x::Int
y::Int
z::Int
end
Foo = Foo_1

abc = Foo(1,2,3)
2 Likes

Redefinition of structs works fine in Pluto.jl, you can use it as your IDE - notebooks are valid .jl files.

3 Likes

In principle, I think that is obvious. Your code should be equivalent to:

struct Foo1
    x::Float64
end
Foo = Foo1
bar = Foo(1.0)
baz(x::Foo) = Foo(x.x -1.0)

struct Foo2
    y::Int
end
Foo = Foo2
baz(x::Foo) = Foo(x.x -1.0)

Although other methods for Foo2 could work, the method baz(::Foo2) will throw an error. No doubt you will eventually get sick of that and redefine it.

It isn’t obvious how the compiler would find all the methods like baz that need to be recompiled for Foo2. It might be hard to find a compromise that could actually be implemented, but where the semantics were still clean and code did what people expected it to.

I’m not claiming my workflow is good, but here’s how I do it:

  1. Write code in package
  2. Have a REPL open near by
  3. Write everything I want to try in the test and use ]test
4 Likes

TLDR: I used to have such a macro, but there were quite a few edge cases that I didn’t handle, so I finally gave up using it. (Might still be useful to some people I guess, but I’m not sure if I kept the code somewhere)

First, note that you need to handle not only type definitions, but also type constraints in methods. Taking a simple example with an hypothetic numbered_struct macro:

@versioned_struct v1 struct Foo
  x::Int 
end

bar(::Foo) = 1

A simple expansion like this won’t work well with revise:

struct Foo_v1
  x::Int 
end
Foo=Foo_v1

bar(::Foo) = 1

this is because the value of Foo is taken into account when bar gets compiled, producing a method working on Foo_v1 instances. But when Foo_v2 comes into play, no new bar method is introduced (unless Revise also detects changes to the definition of bar itself, in which case it redefines the correct method)

One way to circumvent that could involve abstract types, making the above expand to:

abstract type Foo end
struct Foo_v1 <: Foo
    x :: Int
end
Foo(args...) = Foo_v1(args...)

bar(::Foo) = 1

This way, methods constraining their arguments types to Foo actually support any (concrete) version of the (abstract) type. So there is no need to introduce a new method each time a new version of the type is defined.
This is where I stopped at the time. It does work in some cases, but as I said earlier, I encountered so many edge cases that I eventually stopped using this approach.

(IIRC, one of the main problems was to handle parametric types)

These days, I tend to simply restart my REPL every once in a while when necessary, relying on system images to make this more painless. When I begin working on a large-ish project with large/long-to-load dependencies, I use PackageCompiler to build a system image incorporating all dependencies (but not the project I’m developing). That way, the only pre-compilation Julia needs to do at each restart is the precompilation of my own code, which I find to be very fast in most instances.

4 Likes

I used to use PackageCompiler for precompilation speed.
However, it sometimes yields errors and I had to restore images to the default system image.

Haven’t you experienced such errors? I just wonder if it happened only in my case.

No I haven’t encountered any error lately. Or at least no “structural” error: it does happen sometimes that I run out of memory when I try to build a sysimage while running a memory-intensive computation (which is the politically correct expression for “having too many web browser tabs open”).

Your comment about “restoring the default system image” make me think that we might not have done the exact same thing. To be more explicit: I generate images with a piece of code along the lines of

using PackageCompiler
create_sysimage(
    [:Plots],
    sysimage_path = joinpath("my_image.so"),
    precompile_execution_file = "precomp.jl",
    # cpu_target = "generic;sandybridge,-xsaveopt,clone_all;haswell,-rdrnd,base(1)"
)

that is to say that I never use the replace_default keyword argument and therefore never have to use restore_default_sysimage if things go wrong.

I then invoke Julia with the -J my_sysimage.so command-line switch (either from the command-line or from VScode; this can be configured in the Julia extension settings)

4 Likes

Your idea does not work for parametric types because:

struct Foo1{T}
    x::T
end
Foo{T} = Foo1{T}
bar = Foo(1.0)
baz(xInt::Foo) = Foo(x.x -1.0)

struct Foo2{T}
    y::T
end
Foo{T} = Foo2{T}
baz(x::Foo) = Foo(x.x -1.0)

gives:

julia> Foo{T} = Foo2{T}
ERROR: invalid redefinition of constant Foo
Stacktrace:
 [1] top-level scope at REPL[6]:1