How to upgrade code for a symmetric distance matrix from julia-1.0 to julia-1.6

As part of my code for julia-1.6 I try to calculate an NxN symmetric ‘distance’ matrix, where ‘distance’ is a function of 4 vectors (extracted as columns from a DataFrame),and two parameters (‘R2’ and ‘range’).
While the code below worked with julia-1.0, it now gets stuck without an error message.
( I checked the ‘distance’ function separately; nothing wrong with it.)

for i = 1:(N-1)
  for j = (i+1):N
    d2[i,j] = ((z_pred[i] - z_pred[j])^2)/R2 + (s2[i] + s2[j]).*(1 - exp.(-3*(sqrt.((x[i] - x[j])^2 + (y[i] - y[j])^2)/range)))
  end
end
d2 = d2 + d2'

What is N? What is d2? What is z_pred? I’ll stop there, my point is basically make an MWE so that people can reproduce your issues, and ideally read:

2 Likes

Sorry for not putting my question in the right way. I’ll try to do it better now by providing a simplified version of the problematic part of my code. See below.

# A simplified version of my calculation of a distance matrix 'd2':
N = 5
d2 = zeros(N,N)
R2 = 0.36
range = 582
z_pred = (1, 8, 3, 2, 4)
s2 = (0.4, 0.2, 0.3, 0.9, 0.1)
x = (-50, 30, 12, -3, 19)
y = (7, 3.5, 4.2, 10, -25)
for i = 1:(N-1)
  for j = (i+1):N
    d2[i,j] = ((z_pred[i] - z_pred[j])^2)/R2 + (s2[i] + s2[j]).*(1 - exp.(-3*(sqrt.((x[i] - x[j])^2 + (y[i] - y[j])^2)/range)))
  end
end
d2 = d2 + d2'
println(d2)

To my surprise this small example worked and produced a good 5x5 matrix.
So I’ve still no idea why the real code doesn’t work.

This is the reason MWE’s are so important, because they teach the person afflicted by the problem that their problem is not where they think it is, before they rope other people into finding it.

4 Likes

I wouldn’t be that harsh. But MWEs are great! They make your questions better, and often solve the problem outright.

I’ll add one piece of advice: wrap your code in a function. It makes the code more self-contained and performant, and removes problems caused by various hidden state you may have hanging around your workspace.

A bare loop hanging out there exposed in the global scope, looks totally unnatural and ugly to me now.

Do you mean all the code in one function? What I have now is 4 files, say A, B, C and D. The codes in B, C and D are each wrapped in a function.
Any run follows A > B > C or A > B > D, depending on a parameter setting in A.

This is the reason why I did not see how to make a MWE

I meant put the main part of your MWE in a function.

You could still show us the error message at least?

1 Like

As others have told, without further information is difficult to know with certainty, but let’s try a guess:

If the error really happens in the code you have posted, it may be either that:

  • The vectors used inside the loop cannot be indexed with the i, j values that are defined (e.g. they have less than N values).
  • Same with the matrix d2 (it is smaller than N×N).
  • Some variable used in the loop does not exist.

Or maybe the error does not happen there?

So you’re at least using the DataFrames package. Maybe it’s not the older version of Julia what gives you problems, but the version of that package. How do you extract the columns? Long ago you could do dataframe[columnname], in current versions that’s not allowed, and you have to write dataframe[!, columnname] or dataframe[:, columname] (copy).

Part of the problem is that there isn’t any error message. Julia simply halts

The way to debug something like this can be:

  1. Be sure that you don’t have any global variable anywhere in your functions, particularly that one that halts. This is a good advice not only because of the peculiarity of the Julia performance that depends on this, but as a general advice in any programming language - it allows the localization of the problem much more easily.

  2. Make a copy of the code and and try o remove as much as code as you can while keeping the error. That will produce a MWE, which may be minimal enough to be shared, or at least will help you to localize the problem.

  3. Try to split the code into smaller functions, being sure that none of these functions depend on global variables. This is a very good programming practice as well, because then, if your program halts in one of this specific functions, and it is dependent only the exact input arguments, you can localize the problem in the operations of that function.

1 Like
  • The vectors used inside the loop cannot be indexed with the i, j values that are defined (e.g. they have less than N values).
  • Same with the matrix d2 (it is smaller than N×N).

I have checked this before, by printing i and j within the loop. At the end of the (very long) list came i = 4381 and j = 4382, as it should.

  • Some variable used in the loop does not exist.

I have checked this also before, by doing the calculation of one generalised distance. That went well, so all variables do exist.

Or maybe the error does not happen there?

I tried to locate the error in the way I’m used to: by printing a short message (“untill here”) at crucial places.

In this case there was a message just before the loop, but not after it.

This is still ambigous as in

  • returns to the REPL prompt (then you could investigate global variable state)
  • aborts to the OS shell (then I’d expect to see some kind of crash dump)
  • returns to the OS shell (then I’d expect to see some kind of call to exit in your code)
  • diverges, endless loop (then I’d expect to see at least one Julia thread active in your system monitor)
  • diverges due to dead lock

I think your possibillity nr 4 is the case, as I seen in CALL Stack “running"

In this case I’d instrument the program with println or show to monitor progress or would use a debugger to step through it.

Edit: I’m not sure if a screenshot would help us to better understand the problem?

I have to apologize, because some minutes ago it appeared that the problem wasn’t in the distance matrix calculation, but in a vector declaration close after the matrix piece of code.
I did have println(“till here”) directly after matrix code, but that didn’t show up. The reason must have been that I have forgotten to save a change before running.
Now that I ran the code it did show up.
Please accept my appology.

4 Likes

I have to apologize, because some minutes ago it appeared that the problem wasn’t in the distance matrix calculation, but in a vector declaration close after the matrix piece of code.
I did have println(“till here”) directly after matrix code, but that didn’t show up. The reason must have been that I have forgotten to save a change before running.
Now that I ran the code it did show up.
Please accept my appology.

Glad to know that you eventually found the error. As an extra piece of advice, I recommend you to use the nice logging tools that Julia has to manage those “debugging” messages: instead of

println("until here")

you can write this in your code

@debug "until here"

By default those messages won’t show up, but you can “turn them on” with the following instruction:

ENV["JULIA_DEBUG"] = Main

And when you feel like disabling them again, just do:

ENV["JULIA_DEBUG"] = ""

A bonus of doing it like this is that you don’t have to redefine the functions to clear the debugging messages, and that’s good for performance.

More detailed information here:
https://docs.julialang.org/en/v1/stdlib/Logging/

1 Like