Problem with Interpolations. Did it change?

Hi everyone, I am updating a code of mine and on it I need to be able to interpolate a table. Mind that there a steps on the table (where the same value of X, has 2 values of Y1) and I used to have this lines:

    ivp      = LinearInterpolation(model[:,1],  model[:,2]);
    iivp = zeros(Float64, kmax); # inverse of ivp
    for k=1:kmax
        iivp[k] = 1/ivp(vz[k]);
    end

model is a table (see bellow) where the firsts columns had repeated values. It never gave me a problem before, but now I get:

ERROR: knot-vectors must be unique and sorted in increasing order

Any idea?

model=[ -8.0 5.7 3.46 2.72
0.0 5.8 3.46 2.72
20.0 6.0 3.46 2.72
20.0 6.5 3.85 2.92
35.0 6.7 3.85 2.92
35.0 8.04 4.48 3.3198
77.5 8.045 4.49 3.3455
120.0 8.05 4.5 3.3713
165.0 8.175 4.509 3.3985
210.0 8.3 4.518 3.4258
210.0 8.3 4.523 3.4258
260.0 8.4825 4.609 3.4561
310.0 8.665 4.696 3.4864
360.0 8.8475 4.783 3.5167
410.0 9.03 4.87 3.547
410.0 9.36 5.08 3.7557
460.0 9.528 5.186 3.8175
510.0 9.696 5.292 3.8793
560.0 9.864 5.398 3.941
610.0 10.032 5.504 4.0028
660.0 10.2 5.61 4.0646
660.0 10.79 5.96 4.3714
710.0 10.9229 6.0897 4.401
760.0 11.0558 6.2095 4.4305
809.5 11.1353 6.2426 4.4596
859.0 11.2221 6.2798 4.4885
908.5 11.3068 6.316 4.5173
958.0 11.3896 6.3512 4.5459
1007.5 11.4705 6.3854 4.5744
1057.0 11.5495 6.4187 4.6028
1106.5 11.6269 6.451 4.631
1156.0 11.7026 6.4828 4.6591
1205.5 11.7766 6.5138 4.687
1255.0 11.8491 6.5439 4.7148
1304.5 11.92 6.5727 4.7424
1354.0 11.9895 6.6008 4.7699
1403.5 12.0577 6.6285 4.7973
1453.0 12.1245 6.6555 4.8245
1502.5 12.1912 6.6815 4.8515
1552.0 12.255 6.7073 4.8785
1601.5 12.3185 6.7326 4.9052
1651.0 12.3819 6.7573 4.9319
1700.5 12.4426 6.7815 4.9584
1750.0 12.5031 6.8052 4.9847
1799.5 12.5631 6.8286 5.0109
1849.0 12.6221 6.8515 5.037
1898.5 12.6804 6.8742 5.0629
1948.0 12.7382 6.8972 5.0887
1997.5 12.7956 6.9194 5.1143
2047.0 12.8526 6.9418 5.1398
2096.5 12.9096 6.9627 5.1652
2146.0 12.9668 6.9855 5.1904
2195.5 13.0222 7.0063 5.2154
2245.0 13.0783 7.0281 5.2403
2294.5 13.1336 7.05 5.2651
2344.0 13.1894 7.072 5.2898
2393.5 13.2465 7.0931 5.3142
2443.0 13.3018 7.1144 5.3386
2492.5 13.3585 7.1369 5.3628
2542.0 13.4156 7.1586 5.3869
2591.5 13.4741 7.1807 5.4108
2640.0 13.5312 7.2031 5.4345
2690.0 13.59 7.2258 5.4582
2740.0 13.6494 7.249 5.4817
2740.0 13.6494 7.249 5.4817
2789.67 13.653 7.2597 5.5051
2839.33 13.6566 7.2704 5.5284
2891.5 13.6602 7.2811 5.5515
2891.5 8.0 0.0 9.9145
2939.33 8.0382 0.0 9.9942
2989.66 8.1283 0.0 10.0722
3039.99 8.2213 0.0 10.1485
3090.32 8.3122 0.0 10.2233
3140.66 8.4001 0.0 10.2964
3190.99 8.4861 0.0 10.3679
3241.32 8.5692 0.0 10.4378
3291.65 8.6496 0.0 10.5062
3341.98 8.7283 0.0 10.5731
3392.31 8.8036 0.0 10.6385
3442.64 8.8761 0.0 10.7023
3492.97 8.9461 0.0 10.7647
3543.3 9.0138 0.0 10.8257
3593.64 9.0792 0.0 10.8852
3643.97 9.1426 0.0 10.9434
3694.3 9.2042 0.0 11.0001
3744.63 9.2634 0.0 11.0555
3794.96 9.3205 0.0 11.1095
3845.29 9.376 0.0 11.1623
3895.62 9.4297 0.0 11.2137
3945.95 9.4814 0.0 11.2639
3996.28 9.5306 0.0 11.3127
4046.62 9.5777 0.0 11.3604
4096.95 9.6232 0.0 11.4069
4147.28 9.6673 0.0 11.4521
4197.61 9.71 0.0 11.4962
4247.94 9.7513 0.0 11.5391
4298.27 9.7914 0.0 11.5809
4348.6 9.8304 0.0 11.6216
4398.93 9.8682 0.0 11.6612
4449.26 9.9051 0.0 11.6998
4499.6 9.941 0.0 11.7373
4549.93 9.9761 0.0 11.7737
4600.26 10.0103 0.0 11.8092
4650.59 10.0439 0.0 11.8437
4700.92 10.0768 0.0 11.8772
4801.58 10.1415 0.0 11.9414
4851.91 10.1739 0.0 11.9722
4902.24 10.2049 0.0 12.0001
4952.58 10.2329 0.0 12.0311
5002.91 10.2565 0.0 12.0593
5053.24 10.2745 0.0 12.0867
5103.57 10.2854 0.0 12.1133
5153.5 10.289 0.0 12.1391
5153.5 11.0427 3.5043 12.7037
5204.61 11.0585 3.5187 12.7289
5255.32 11.0718 3.5314 12.753
5306.04 11.085 3.5435 12.776
5356.75 11.0983 3.5551 12.798
5407.46 11.1166 3.5661 12.8188
5458.17 11.1316 3.5765 12.8387
5508.89 11.1457 3.5864 12.8574
5559.6 11.159 3.5957 12.8751
5610.31 11.1715 3.6044 12.8917
5661.02 11.1832 3.6126 12.9072
5711.74 11.1941 3.6202 12.9217
5813.16 11.2134 3.6337 12.9474
5863.87 11.2219 3.6396 12.9586
5914.59 11.2295 3.645 12.9688
5965.3 11.2364 3.6498 12.9779
6016.01 11.2424 3.654 12.9859
6066.72 11.2477 3.6577 12.9929
6117.44 11.2521 3.6608 12.9988
6168.15 11.2557 3.6633 13.0036
6218.86 11.2586 3.6653 13.0074
6269.57 11.2606 3.6667 13.01
6320.29 11.2618 3.6675 13.0117
6371.0 11.2622 3.6678 13.0122]

As the error says, your knot vector can’t have repeated values. It’s ill-determined: do you pick one of the values at random? Take the mean? You’ll need to decide how you want the interpolation to behave, and clean your data appropriately to remove repeated values.

This change was introduced in this commit, so in the short term, you can just pin the version to the prior release (v0.13.1).

1 Like

This is very inconvinient … most of the data I work with looks like this.
Is there a way for me to β€œgrab” a copy of the old interpolation so I can keep it and not be bound to the Interpolations package?

Or maybe it could become an option in the package.

The package is MIT expat licensed, so you have

[…] the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. […]

So yeah, if you feel the need to copy the state of Interpolations pre v0.13.1 you can do so. You can even fork it I believe and install from your own repository. Just don’t try to register it in General :slight_smile:

3 Likes

I’d very much recommend properly preprocessing your data though.

6 Likes

Given that it is a conceptual model of Earth’s structure, I rather not change it. But I welcome any suggestions

Thanks. Maybe I will find how to fork it

If you want to use version 0.13.1 you can just pin it at that version, no need to fork?

In terms of changing the model, I guess the first step would be to see what Interpolations 0.13 does when there are duplicates in the knot vector, and whether this is actually what you expect (the question above is quite relevant I think, it’s not clearly to me what should be happening when you present Interpolations with such data).

2 Likes

But Interpolations already made a choice for you in past versions (afaict it just takes the first value). So now you’ll just need to do the same manually, with the bonus that you actually know what’s going on.

3 Likes

Here is a quick example @nilshg

It does choose the first value for the knot as @pfitzseb says, but understand that right after is another:


julia> ivp(19.999)
5.9999899999999995

julia> ivp(20.001)
6.5000133333333325

julia> ivp(20)
6.0

This is exactly what I need, but the option of choosing the value from the second line or the average (as a value of the knot) would be cool to have.

That does sounds like something that should be happening in pre-processing: basically you stick your data in a DataFrame and then do something like

combine(groupby(data, :knots, sort = true), observations => first)

Then you pass the result of this to itp. Of course you can then replace first with mean, last, or whatever other function you might need.

2 Likes

In fairness, the underlying PR is (arguably) a breaking change, and should have coincided with a version bump, but it wasn’t even mentioned in any of the release notes (afaict).The old behavior was ambiguous, but it wasn’t incorrect, per se, and this change should have been more clearly publicized.

4 Likes

I will have to give this a try. Is data here a vector?

Definitely.
Also, in the spirit of making the package as flexible as possible I think keeping it as a option is a good idea. Maybe a little function can be added to preprocess the data for this kind of case.

In my example data is a DataFrame - I’m not sure what your actual data structure looks like, but I was thinking of something like this:

julia> using DataFrames, Statistics

julia> data = DataFrame(knots = [1, 2, 2, 3], obs1 = rand(4), obs2 = rand(4))
4Γ—3 DataFrame
 Row β”‚ knots  obs1       obs2     
     β”‚ Int64  Float64    Float64  
─────┼────────────────────────────
   1 β”‚     1  0.817886   0.915149
   2 β”‚     2  0.604035   0.323521
   3 β”‚     2  0.242594   0.243895
   4 β”‚     3  0.0296911  0.494106

julia> observations = [:obs1, :obs2];

julia> combine(groupby(data, :knots), observations .=> first .=> observations)
3Γ—3 DataFrame
 Row β”‚ knots  obs1       obs2     
     β”‚ Int64  Float64    Float64  
─────┼────────────────────────────
   1 β”‚     1  0.817886   0.915149
   2 β”‚     2  0.604035   0.323521
   3 β”‚     3  0.0296911  0.494106

julia> combine(groupby(data, :knots), observations .=> mean .=> observations)
3Γ—3 DataFrame
 Row β”‚ knots  obs1       obs2     
     β”‚ Int64  Float64    Float64  
─────┼────────────────────────────
   1 β”‚     1  0.817886   0.915149
   2 β”‚     2  0.423314   0.283708
   3 β”‚     3  0.0296911  0.494106

(in your actual use case knots might be more than one column if you’re sampling on a higher dimensional grid, but you hopefully get the gist)

1 Like