[ANN] MutableArgumentContracts.jl (experimental, not yet registered)

Package Announcements
,
1 
foo!(x, y, z)

By convention, the ! suffix indicates the function is mutating one or more arguments. But which argument is being mutated? Just the first, as is convention? What if it’s more?

For a while I had a thought about using the dispatch system to provide a way to enforce self-documentation of which arguments of a mutating function are mutated, and here it is:

@! function foo!(x::!, y::!{T}, z) where {T}
    x .+= y .+ z
end

a = [1, 2, 3];
b = [3, 2, 1];
foo!(@!(a), @!(b), 3)

In an imagined future world, this is something Julia could possibly do natively and less clumsily.

I just finally got around to trying it out. I’m sure it’s deeply imperfect, but I wanted to get the concept out of my head, and happy to take any feedback! Especially the name, for if/when I add it to the registry.

9 Likes
2

Nice it is great to see people experiment with this! I was surprised more than once by calling a function and an unexpected argument is mutated. Often this results in subtle bugs. I thought about implementing a package like yours myself, but your syntax is much nicer, than what I had in mind.
I also have the somewhat complementary MutationChecks.jl that can help debugging surprising mutations. But it definitely has many rough edges.

It might be nice to add an (optional because slow) checking mode to your package that enforces only mutating the declared args.

2 Likes
3

Nice idea.

I have a question about the particular example: It marks y as mutable, even though the function doesn’t mutate y. So, I assume it’s just to show the syntax?

4

Yes :slight_smile: Just an illustration of syntax.

2 Likes
5

Thanks! I’ll look into that.

6

My immediate reactions:

  1. I probably would prefer a short and pronouncable name Mut{} over macros so things print more straightforwardly.
  2. This doesn’t play entirely well with dispatching to methods with abstract type annotations because type parameters are invariant. For example, f(x::DenseVector) is more specific than f(x::AbstractVector), but f(x::MutableArgument{DenseVector}) is not more specific than f(x::MutableArgument{AbstractVector}), and in fact f(MutableArgument([1,2,3])) would lack a method where f([1,2,3]) would have one. This wouldn’t be a problem for transforming ::Any to ::MutableArgument instead of ::MutableArgument{Any}, or for methods with concrete type annotations.

My untested idea to fix 2 is ugly without a macro so 1 has to be forgotten, but you can use the covariance of Tuple’s type parameters. Tuple{AbstractVector} alone could easily be mix up with existing code taking ([1,2],), so you could include a new singleton type for dispatch Tuple{Mut, AbstractVector}, and you pass (Mut(), x) into functions.

In practice I’m not sure how well I could use this. I wouldn’t use this on mutable arguments that aren’t intended to be mutated (like the y in your example), so the method can also accept an immutable argument at that position. I would definitely use this on arguments that are attempted to be mutated, even if something attempts to pass in an immutable argument; seeing Mut() show up next to an immutable value would be a wakeup call. What I’m not sure about is arguments that only might be mutated; I’d err toward Mut but I would find a 3rd option more informative, though putting that information in the type annotation system will interfere with dispatch in a way I don’t want (Mut can’t subtype MayMut). If this ends up as documentation with no functionality, I’d rather indicate intention of mutation in the docstring if it exists or write comments in the source, like foo!(x #=mutated=#,..., or even weirdly name variables x!!, so I don’t need to manually wrap inputs, like in MutableArgument. An example of functionality would be to make MutableArgument error when given a semantically immutable type, which is distinct from the composite type’s direct mutability (ismutable) e.g. BigInt, String, array wrappers like views.

3 Likes
7

I figured I was probably missing something there, thanks! I’ll mull over that.

8

I at least I think it’s always the first. That should be a convention, it doesn’t seem helpful to not standardize that, if not already, though of course people can not match such a style-guide.

If your package is not registered, I suggest changes.

Am I reading this wrong, y (and z) are not mutated?! So is this only a bad example?

I would suggest you only allow a prefix, forst 1, 2, 3 etc. parameters mutated (is more flexibility really needed?), so could this be done:

@! 2 function foo!(x, y::{T}, z) where {T}

simply have count of how many, 1 could be used, but seems could be implied, and then your package not used.

In your case parameters are not typed, except for this “magic” to enable, e.g. x::! but it seems it can’t be extended to e.g. x::String!.

I like that you’ve thought about this “problem”, but I would want to go in the opposite direction, no mutable functions…

I realize this is a thing in Julia, a core part, and Julia isn’t Clojure, but I think in most cases you still shouldn’t use mutability, any if maybe only for first parameter. For the exceptions, when you want mutability, how commonly is it not just the first parameter? Is the more general form really needed or a code-smell?

1 Like
9

This is something I have been wanting for a long time now, and I hope you will get somewhere with the idea further. Perhaps I can give you an API suggestion which may be more pleasant to the eyes. Instead of using ! with arguments before the types, I would much prefer the use of the special keyword mut, something like:

@strict function foo!(mut x, mut y::T, z) where {T}
    x .+= y .+ z
end

At the call site, this could look as follows:

foo!(@mut a, @mut b, 3)
10

This would fail at parsing because mut x is not valid syntax; macros can only transform valid expressions. @mut would work there. I’m not comfortable with this because it reminds me of mut in Rust, and mutability there is very different because instances are tied more directly to variables, ie immutable variable cannot be reassigned.

2 Likes
11

This would fail at parsing because mut x is not valid syntax;

This is sad. Anyways good luck :+1:

12

I do concur with you that a keyword of some sort would be simpler and cleaner; but as they said, just not possible with only macros to work with.

Also, unfortunately, macros take precedence over a comma, so the parentheses are needed to isolate the macro to operate on just the one argument.

13

As described in an earlier post, the example was just to illustrate syntax.

Times when the mutated argument might not be the first? If the first is a function instead, or you need to provide scratch space. Personally, I don’t like putting scratch space arguments in the first positions.

While specifying a number in front might be simpler and would enforce that design, I think it gets away from the self-documenting aspect. The intention is as much to communicate to the human reader as to the computer. And you’d also need to deal with function-first-argument mutating functions.

The intent of ::!{String} is to be the equivalent of ::String, though as was suggested earlier I need to look at that more closely.

As long as we have to worry about slow heap allocations, we will need to worry about mutating functions, I think. And even if it’s not ideal design, I’d rather people have the tools they need to communicate what they’re doing.

1 Like
14

It’s good to experiment with this sort of thing. I think the sort of solution you’re trying for here needs to be a core language feature, and short of that it doesn’t solve the actual problem it’s addressing.

Here’s what I mean: a function that uses this macro forces the calling code to acknowledge that it knows that parameters will be (might be) mutated. To call the function one has to import the macro and use a special syntax. And ok, in a way that’s stronger than clear documentation, you essentially make the programmer sign a consent form for mutation of the collections.

But this isn’t actually solving the problem, because if I expect and want those parameters mutated, it’s just extra steps. Furthermore, since it isn’t a language feature, we can’t force everyone to use it, so it’s still the status quo out there, which is that when you pass a collection to a method, it might mutate it before handing it back.

What would go a lot further is a pair of functions, freeze and thaw, with methods specialized for all the mutable collection types in Base. If you call freeze(d::Dict), you get back a FrozenDict <: AbstractDict, which simply doesn’t respond to the methods for mutating the dictionary. Julia handles this kind of wrapping very efficiently, it’s zero extra allocation and the methods on the base object are almost always inlined into the underlying methods. thaw, of course, just gives me my Dict back.

That lets me insist that a collection is not to be mutated, if something tries, I get an error right away. If I’m chasing down a mutation bug, I can just freeze the variable and see what’s doing it by the stacktrace.

This comes with some minor inconveniences, admittedly, like methods which only take a Dict (though this is nearly always lazy/wrong and can be corrected with the correct abstract signature), you can’t freeze a Dict that’s an immutable struct member (because that is itself mutation), but it would go much further to solving the unwanted-mutation problem than an opt-in method signature macro imho.

As a language feature, I do like the idea you’re exploring here, although it would take some very far-reaching changes to Julia to allow it to reject code which mutates a reference without a signature allowing it to do so. But in terms of what can be done to prevent unwanted mutation, frozen collections gets closer to solving the problem.

2 Likes
15

I appreciate your thoughts, and I concur that it’s unlikely most anyone would want to depend on this package for a public API.

Your proposal seems interesting. How would your proposal work for nested types? and can it be generalized to any type or would it have to be only specified containers?

It might be nice if it were possible to make all types recursively read only by default, and only the site of construction is capable of passing along the write-access view. Then you’d be able to trace the whole chain of write-access to the origin.

(of course this “access” should be overridable if truly desired)

16

There isn’t a general-purpose mechanism to wrap a collection in a struct, unfortunately. I actually started working on a package to make that process easier, which is on the back burner for now.

The good news is that in Base there just aren’t that many of them: Array/Matrix/Vector, Dict, Set, and that’s it unless I’m forgetting something.

For packages which provide collections one would need to extend the freeze and unfreeze method, make a wrapper struct for each new collection type, and provide the allowed methods on that struct (which is the only time consuming part but it’s mostly boilerplate), hasmethod(Type) can reveal the functions which need methods. I’ve done this for a specialized Dict class, and the one wrinkle is that some methods like merge! return the Dict, so they need a tiny bit of extra logic to return the wrapper, not the wrapped.

In this case, just don’t provide the methods with a ! in them and the job is largely done :slight_smile:

If the idea caught on I’m sure maintainers would be open to a PR adding that, and might even do it on request (best not to assume! but it rarely hurts to ask).

I don’t think that one would want recursive freezing to be the default for nested collections, since that entails a whole traversal of the collection (recursively) but a freezeall method which did the job would be possible, although one would have to make sure to freeze from leaf to root, since you’d be reassigning e.g. a nested Dict such that the value is FrozenDict, the exact thing you can’t do on an already frozen Dict.

Making the outer layer frozen would be a very cheap operation by contrast. Better to freeze the inner collections ad hoc and then add them to the outer one, most of the time anyway.

1 Like
17

I slightly prefer marking the function argument type as inout, like in Verilog or Fortran

18
  1. This doesn’t play entirely well with dispatching to methods with abstract type annotations because type parameters are invariant. For example, f(x::DenseVector) is more specific than f(x::AbstractVector), but f(x::MutableArgument{DenseVector}) is not more specific than f(x::MutableArgument{AbstractVector}), and in fact f(MutableArgument([1,2,3])) would lack a method where f([1,2,3]) would have one. This wouldn’t be a problem for transforming ::Any to ::MutableArgument instead of ::MutableArgument{Any}, or for methods with concrete type annotations.

I’m probably missing something, but would changing the conversion from !{T} => MutableArgument{T} to instead MutableArgument{<:T} be sufficient?

At a minimum it looks like I’m getting correct dispatching with that adjustment:

@! function foo!(x::!, y::!{DenseVector}, z)
           x .+= y .+= z
       end

@! function foo!(x::!, y::!{AbstractVector}, z)
           x .+= y .+= z
       end

julia> @which foo!(@!(a), @!([;]), 3)
foo!(x::MutableArgument, y::MutableArgument{<:DenseVector}, z) where T
     @ Main none:0

julia> using SparseArrays;
julia> @which foo!(@!(a), @!(sparse([;])), 3)
foo!(x::MutableArgument, y::MutableArgument{<:AbstractVector}, z) where T
     @ Main none:0
1 Like
19

Sorry if I missed this somewhere in the thread, but is there a reason why this isn’t just written as

@! function foo(!a, !y::DenseVector, z)
    ...
end

?

20

definitely not always, e.g. rand!(rng::AbstractRNG, ...)

well, I guess technically the rng is mutated, but that’s not really what the ! is getting at I think

1 Like