Cannot use size(::Array{Int, 2}): ERROR: LoadError: UndefRefError: access to undefined reference

[Stephen_Vavasis]

The proposed patch to the error messages will definitely improve things, but the problem still remains in the case that the operator is shadowed before it is used. For example, the code below when invoked as test_plus.f(2,3) prints 0 rather than 11.

I agree that an additional warning will annoy the experts, but there needs to be some consideration for the new guys (and even for the experts who sometimes make typos).

module test_plus

function f(a,b)
    tests = [a - b == 0, a * b == 0, a + b = 0]
    if in(true, tests)
        println("either a=b, a=-b, or a or b is zero")
    end
    println(2+9)
    nothing
end

end

[yuyichao]

@Stephen_Vavasis

I agree that an additional warning will annoy the experts, but there needs to be some consideration for the new guys (and even for the experts who sometimes make typos).

And that is exactly what linting tools for.

[Stephen_Vavasis]

I couldn’t disagree with you more on this!

If you guys are touting Julia as “A Fresh Approach to C”, then fine, feel free to tell users who get caught in semantic traps that they ought to use Lint. But it seems that you actually want to tout Julia as a fresh approach to Matlab, R and Numpy, You should consider how Matlab, R and Numpy might deal with this.

Indeed, there is a historical comparison. In the early 1990s, Matlab suffered from the following shoot-yourself-in-the-foot scenario. In the expression a(v), where a was a vector length n and v was a vector of all 1’s of length n, the meaning was ambiguous. This could either mean boolean addressing, in which case the answer should be a(1:n), or it could mean an integer vector address, in which case the answer should be a(1)*ones(1,n). The Matlab interpreter chose one of the two ways to resolve the ambiguity. Programmers who overlooked the ambiguity (even experienced ones like me) and expected the other resolution incurred subtle program bugs that might not occur in the first 9999 loop iterations but would strike in the 10000th.

Mathworks fixed this problem by introducing a new type, logical, and forbidding the use of ordinary integers for boolean addressing. This change broke almost everyone’s code, but at the time I recall most serious Matlab users approved of the change.

Here is my proposal: for the code in my previous post, the compiler should say:

WARNING: Shadowing, +(::Any,::Any) hides Base.+. Use @OK_to_shadow macro to suppress this message. In f(::Int64, ::Int64), test_plus.jl:4
Then I could either fix the bug, or if I had really intended to redefine +, then I would change my code to

tests = [a - b == 0, a * b == 0, (@OK_to_shadow a + b = 0)]

Is that really so annoying to experts? Compare it to the hours of annoyance by the OP and myself in trying to figure out this semantic trap.

[yuyichao]

@Stephen_Vavasis

Is that really so annoying to experts? Compare it to the hours of annoyance by the OP and myself in trying to figure out this semantic trap.

Yes because as I mentioned before, this makes adding functions to base breaking change.

[Stephen_Vavasis]

There is a straightforward workaround to this objection. Make a list of the 100 or so most common functions in Base such as the operators (+,-,:,etc), common functions like size, length, collect, in, etc., and require some special syntax to redefine this fixed set of functions if the redefinition appears inside another function definition. The list can stay fixed even as Base grows.

Old-timers like me can recall that early versions of Fortran were widely mocked because programmer error could lead to the possibility of redefining constants (like 2.1). Julia should not fall into the same trap!

[Tamas_Papp]

@Stephen_Vavasis

Fortran were widely mocked because programmer error could lead to the possibility of redefining constants (like 2.1)

Although these things are subjective, I see the examples you bring up from other languages (including the Matlab one above) as much more insidious than the one that started this topic, which can be handled by better error messages and/or a linter.

Also, your proposed fix would lead to two kinds of functions in Base, effectively a huge special case.

[Stephen_Vavasis]

I agree that the traps I mentioned in Matlab and Fortran were more insidious than the redefinition trap in Julia. The reason that they were more insidious is that they could spring up in a program that was seemingly tested and working well, and they could not be caught at compile time. In contrast, it is unlikely that a program with the Julia trap could ever work properly, and furthermore, as you and @yuyichao observe, the trap can be caught at compile time.

But the flipside of this argument is that, since it can be caught at compile-time, then it should be! The Matlab trap required a significant language change and the Fortran trap required the invention of write-protected RAM to fix. The Julia trap is easy to fix.

As for adding special cases, yes, I am calling for more special cases. To put it bluntly, I am pitting “language purity” against “protecting programmers against typos”. Do you think “language purity” should win?

[yuyichao]

@Stephen_Vavasis

the trap can be caught at compile time.

No it can’t.

@Stephen_Vavasis

Do you think “language purity” should win?

Absolutely. When the two conflict, the check belongs to the linting tool.

[Stephen_Vavasis]

@yuyichao, can you explain why the trap can’t be caught at compile time? In other words, if I can see that my function assigns a definition to +(::Any,::Any), why can’t the compiler see it?

Meanwhile, with respect to Lint, which I have never used prior to today, I installed it and tried it on the two code examples that I posted earlier here. And indeed, on both examples, it gave a warning in the erroneous line that an argument was declared but not used. It did not give a warning about shadowing Base.

Seems like good news? Not really, because on my actually application code, Lint crashed. (I’m going to try to make a minimal example to file an issue.) I segmented my code to get Lint to run, and it gave me several incorrect warnings. Further, on many segments, Lint gave dozens of warnings about arguments declared but not used! So if the only sign of the redefinition trap is an argument declared but not used, I would not have spotted the trap with Lint in a big code.

[Aside: Why so many unused function arguments in my code? Because one reasonable way to use Julia dispatch is:

   function take_time_step(tsm::ImplicitMidpoint, t0, t1, weight)
   function take_time_step(tsm::BackwardEuler, t0, t1, weight)

Not all of those versions of take_time_step use all four arguments.]

So, at least in my case, Lint would not have helped me spot the trap.

[yuyichao]

@Stephen_Vavasis

why can’t the compiler see it?

Because it can’t tell if you want to do that.

@Stephen_Vavasis

It did not give a warning about shadowing Base.

I didn’t say it’s in there. I said it should be added there if it is not already.

@Stephen_Vavasis

So, at least in my case, Lint would not have helped me spot the trap.

And that’s exactly the other issue if you are adding more of these smart warnings to Base. They’ll get mis-triggered all the time and people will just always surpress them. A linting system with different warning levels will help with this and so yes it’ll be able to help you spot the trap if implemented.

[Stephen_Vavasis]

@yuyichao, can you give an example of an existing code (e.g. from a package) in which a commonly used Base function like size or + is intentionally shadowed by a definition inside a function body?

[yuyichao]

Commonly used functions shouldn’t, but how do you define commonly used?

[Stephen_Vavasis]

There are now at least three documented examples (my original github issue from a year ago, this posting, and the previous discourse posting to which I responded a few months ago) of real Julia users getting caught by this semantic trap. This argues in favor of changes. Your arguments against changing the status quo, if you don’t mind my summarizing your points, are:

(1) The resulting warning will annoy programmers who actually want to shadow a commonly used Base function inside a function body. They will still be annoyed even if they have a macro to suppress the warning. We don’t have any examples of these programmers.

(2) Producing this warning requires a special casing, and special-casing in a programming language is inherently undesirable.

(3) If this warning is instituted, then floodgates will be open for every other programmer in the Julia community to demand additional warnings to prevent their pet programming error. The warnings will eventually become so numerous that they will be useless.

(4) Warning about this programming error is not the domain of the compiler; rather it is the domain of Lint. Not the Lint that exists now, but a more full-featured Lint that someone will create some time in the future.

You have not convinced me!

I’m sure I must sound like a big complainer, but I’d like to also be part of the solution. I am not sufficiently knowledgeable to write the compiler code that will detect that a Base function is being shadowed, but if you or someone other knowledgeable person wants to make a PR, I could contribute to the effort by proposing a list of Base functions that deserve the warning.

[yuyichao]

@Stephen_Vavasis

This argues in favor of changes.

To the linter yes.

@Stephen_Vavasis

(1) The resulting warning will annoy programmers who actually want to shadow a commonly used Base function inside a function body. They will still be annoyed even if they have a macro to suppress the warning. We don’t have any examples of these programmers.

No that’s not my point. My point is that this is intended behavior that happens accidentally all the time.

@Stephen_Vavasis

You have not convinced me!

I don’t have to.

@Stephen_Vavasis

I could contribute to the effort by proposing a list of Base functions that deserve the warning.

We will not have such a list of functions in base.