[ANN] StrucRev - revise type definitions as you go

Package Announcements
, ,
1

The package is intended to enable redefining structs within Revise.jl based workflow. It exports the macro @strev which (temporarily) wraps your struct like this

@strev struct S0
    x
end

becomes

module StrucRev_S0
  struct S0
      x
  end
end
S0 = StrucRev_S0.S0

If you edit S0, you will get a warning like
WARNING: replacing module StrucRev_S0
the S0 variable in your module will be re-bound to this new StrucRev_S0, and you can continue to code.

In case you want to define outer constructor methods, the usage is like following

@strev begin
    struct S0
        x::Int
    end
    S0(x::Real) = S0(Int(round(x))
end

If your struct definition contains references to some external variables, they will be imported into the struct-wrapping module.

Another usage is

@strev const c = 1

which will rewrite it to

c::typeof(1) = 1

thus enabling you to experiment with the value of c (but not with its type).

After you are finalized your types & constants, you simply remove all occurences of @strev (and, well, re-start Julia).

For more usage examples see runtests.jl

The package is not registered yet, and can be installed from GitHub. I haven’t yer done any really thorough test with it.

I still have some “imposter syndrome doubts” :worried: - first, if it is that simple, why nobody implemented it yet (or maybe it is already done), and, second, I am definitely not good at macroprogramming and maybe very naive and made some terrible mistakes. If that is not the case and the package is of some value, I plan to add some documentation, more testing, CI, and then register it.

Thanks are due to @disberd for advice and a piece of code.

P.S. Sure there exist packages for the same purpose, which somehow escaped my attention till now - see discussion here

23 Likes
2

I’m probably getting something wrong but the module wrapping seems like how Pluto notebooks are implemented, where you can redefine structs. Of course this more directly wraps the struct instead of a cell, and can be used in non-Pluto notebooks.

2 Likes
3

@Benny is exactly right. This is how Pluto does it! It moves the entire notebook into a new module each time a struct changes and ports forward everything it can from the old module.

5 Likes
4

Very cool package!

I just want to suggest a name change if possible: StructRev instead of StrucRev

I’m also thinking about @rev instead of @strev but maybe this is more controversial :smiley: at the same time it plays better with the revision of constant variables, I’m not sure if there is some other macro out there already called @rev

4 Likes
5

Or yet still: StructRevise may be more idiomatic for Julia packages.

13 Likes
6

I may have missed something, but if I understand correctly what StructRev does, I would anticipate the following issue:

# This is the equivalent for @strev Foo IIUC
julia> module Module_Foo
       struct Foo
           x::Int
       end
       end
Main.Module_Foo

julia> Foo = Module_Foo.Foo
Main.Module_Foo.Foo

# Now we define a method working on `Foo`
julia> foo(a::Foo) = a.x
foo (generic function with 1 method)

# And we use it on a newly created instance
julia> a = Foo(42)
Main.Module_Foo.Foo(42)

# So far so good
julia> foo(a)
42

Now what happens when we try to change the definition of Foo:

# Again, this is the expansion of @strev Foo IIUC
julia> module Module_Foo
       struct Foo
           x::Int
           y::Int
       end
       end
WARNING: replacing module Module_Foo.
Main.Module_Foo

julia> Foo = Module_Foo.Foo
Main.Module_Foo.Foo

# We have to create a new instance (the old one is outdated, fair enough)
julia> b = Foo(42, 43)
Main.Module_Foo.Foo(42, 43)

# But the problem is, methods that have been defined to accept the old
# struct definition don't know about the new one.
#
# (And arguably the error message is confusing!)
julia> foo(b)
ERROR: MethodError: no method matching foo(::Main.Module_Foo.Foo)

Closest candidates are:
  foo(::Main.Module_Foo.Foo)
   @ Main REPL[3]:1

Does StrucRev have a way to deal with this?

As far as I understand, tools like Pluto avoid this by putting everything into a new module: the new struct definition, all method definitions and all uses of the structure.

4 Likes
7

Yes, you are right, that appears to be a problem.

It is currently possible to put all constructors into the new module, and it will be possible to implement putting everything else therein, but then it would be definitely much less elegant.

8

I once experimented with an idea along the lines of:

# file `mysrc.jl`

# This macro does nothing
# But when anything passed to the macro changes, `Revise` will have to
# revise everything. This is a (hacky?) way of forcing a bunch of
# definitions to be revised together.
macro revise_together(expr)
    esc(expr)
end

# We use it to group both the struct definition and the related methods
@revise_together begin
    struct Foo_v1
        x :: Int
    end
    Foo = Foo_v1

    foo(a::Foo) = a.x
end

Now, in a REPL:

julia> includet("mysrc.jl")

julia> a = Foo(42)
Foo_v1(42)

julia> foo(a)
42

If you now change the struct definition in the file to (in the real thing, the versioning would be done by an other macro):

    struct Foo_v2
        x :: Int
        y :: Int
    end
    Foo = Foo_v2

you can go on in the same REPL:

julia> b = Foo(42, 43)
Foo_v2(42, 43)

julia> foo(b)
42

julia> methods(foo)
# 1 method for generic function "foo" from Main:
 [1] foo(a::Foo_v2)
     @ /tmp/mysrc.jl:12

I actually can’t remember why I didn’t explore this further. Of course there is the issue that it’s not always practical to group all methods pertaining to a struct near the struct definition. But I’m kind of confusely remembering that there was a more fundamental reason why the effectivity of this technique was limited, but I just can’t remember exactly why.

2 Likes
9

Now compared my approach with the “competition” - the packages RedefStructs.jl and ProtoStructs.jl .

RedefStructs.jl appears to be broken under Julia 1.10 , at least it errored on a minimal struct definition.

ProtoStructs.jl worked, however it suffers from the same problem as described above by @ffevotte *.

My package appears on the first glance to have all the abilities of ProtoStructs.jl (it supports @kwdef BTW), and provides the possibility to (re-)define inner and outer constructors, which appear to lack in the above package.

Thus it is indeed worth to register my package in its current form (which I was not sure). Will be done after adding some documentation.

The future plan is to enable co-wrapping of related methods. In the limiting case wrapping of “everything”.

@Tortar, @Alec_Loudenback - bikeshedding is welcome, and naming is difficult. For the macro name, I’d stay with the current one. For the package name, I’m not sure yet, but StructRevise could indeed be a better choice.

*EDIT: It is a different problem, see below.

10

While I think your approach is worthwhile it is not true that ProtoStructs.jl suffers from the same problems:

julia> using ProtoStructs

julia> @proto struct Foo
                  x::Int
              end

julia> foo(a::Foo) = a.x
foo (generic function with 1 method)

julia> a = Foo(42)
Foo(42)

julia> foo(a)
42

julia> @proto struct Foo
                  x::Int
                  y::Int
              end

julia> b = Foo(42, 43)
Foo(42, 43)

julia> foo(b)
42
11

It could be a different problem, but with an “includet” file I got an error

Error: Failed to revise ...
│   exception =
│    AssertionError: ld[idx] < typemax(eltype(ld))

Right now it’s midnight here, tomorrow I’d provide more details.

12

yes workflows using Revise don’t play well at the moment with ProtoStructs, but this is not a fundamental limitation as far as I can tell, more than nobody has made a PR fixing it yet

13

Reported the issue on the GitHub

14

made a PR trying to fix the compatibilty problem: Create revisableproto macro for compatibility with Revise by Tortar · Pull Request #35 · BeastyBlacksmith/ProtoStructs.jl · GitHub if someone is interested/would like to propose a better solution

1 Like
15

Is it somehow possible do detect whether the code runs under Revise or outside of it? There should be a way, I think.

BTW I’ve done a short test of your PR in a “Package under Revise” - both re-definition of structs and the of dependent methods appear to function OK.

1 Like
16

yes, it could be maybe possible, I asked a question about it Is there a way to know if `Revise` is enabled?

17

Now that this thread got attention from “This month in Julia world” - the current state:

  • ProtoStructs.jl works now under Revise, too. The only substantial limitation appears to be lack of support for inner constructors.
  • No further progress for StrucRev yet due to lack of time, with some thought about the path to solve the problems to be discussed later.
4 Likes
18

In the meanwhile there may be a chance for a general solution in sight

and most probably the problem is above my competence anyway. Still, the following idea, rather vague one:

module Module_Foo
    struct Foo
        x::Int
    end
end
Foo = Module_Foo.Foo


# now we want to re-define Foo
fm = methodswith(Foo)

sources = [m.source_ref for m in fm] # pseudocode

# delete "old" methods
for m in fm
    Base.delete_method(m)
end

# re-define Foo
module Module_Foo
    struct Foo
        x::Int
        y::Int
    end
end
Foo = Module_Foo.Foo 

for s in sources
    recompile(s) # pseudocode
end

in separate place:

foo(a::Foo) = a.x

Any idea what to put on the place of the pseudocode?