Polyfit (in Polynomials.jl ?) not working in new Julia version?

General Usage
1

Hi, has the Polynomials.polyfit command been discontinued or changed?

I recently updated from Julia (JuliaPro) 1.4.2-1 to Julia 1.10.0, and ran a large simulation program in the new version, which crashed with an error on polyfit:
ERROR: UndefVarError: `polyfit` not defined

This DOES work (still) on my old computer running Julia 1.4.2, but crashes on my new computer running Julia 1.10.0.

I believe that the package which this polyfit command is used from was indeed Polynomials.jl, though I was running a number of other packages that it MIGHT have come from instead (e.g., Interpolations, Optim, RollingFunctions, BSplineKit, etc.). Also, there seems to be a separate “PolyFit” package as well (which I wasn’t using), which is confusing.

For definiteness, here is a code sample that DOES work in my old installation of Julia 1.4.2, but FAILS in my new installation of Julia 1.10.0:

TryXfit = Array{Float64}(undef, 15)
TryYfit = Array{Float64}(undef, 15)

for NP in 1:15
    TryXfit[NP] = NP
    TryYfit[NP] = NP+((NP+27)/(NP+3))
end

polyfit(TryXfit, TryYfit, 3)

This ALSO still fails in the same way if I change the last line to:
Polynomials.polyfit(TryXfit, TryYfit, 3)

2

It looks like polyfit has been replaced by fit:

julia> fit(TryXfit, TryYfit, 3)
Polynomial(7.988124979954325 - 0.2357892152912869*x + 0.1026726309821674*x^2 - 0.0030526831531622577*x^3)

Otherwise, you should use

using Polynomials.PolyCompat
julia> polyfit(TryXfit, TryYfit, 3)

but it calls the same fit routine: Polynomials.jl/src/legacy/Poly.jl at master · JuliaMath/Polynomials.jl · GitHub, as you can see using @which polyfit(TryXfit, TryYfit, 3).

3 Likes
3

FYI: I believe the API intentionally changed with v4 of the package (and the older would still work in 1.10, as is, without changes; though you likely want to adapt to the newer breaking changes).

Many people upgrade packages also when upgrading Julia, since at times the packages are NOT (or some dependency) compatible. To fix those problems, so it’s sort of bad to also have the opposite problem.

4

Thanks mzaffalon, those solutions seem to work, giving similar polynomials as on the old computer & old Julia version (identical to the first 12 digits or so, then some small differences due to limited precision I guess).

My only concern is that the data types of the output seem to be slightly different:

Old Computer & Old Julia:

PolyOld = polyfit(xArray, yArray, 3)
=> Poly(-0.1510753117152796 + 1.1626777881704835*x - 0.006217857121466347*x^2 + 0.00015409159837458824*x^3)
typeof(PolyOld)
=> Poly{Float64}

New computer & New Julia:

PolyNew1 = Polynomials.fit(xArray, yArray, 3)
=> Polynomial(-0.15107531171528035 + 1.1626777881704835*x - 0.006217857121466373*x^2 + 0.00015409159837459016*x^3)
typeof(PolyNew1)
=> Polynomial{Float64, :x} (alias for Polynomials.MutableDensePolynomial{Polynomials.StandardBasis, Float64, :x})

Alternatively for New Computer & New Julia:

using Polynomials.PolyCompat
PolyNew2 = polyfit(xArray, yArray, 3)
=> Poly(-0.15107531171528035 + 1.1626777881704835*x - 0.006217857121466373*x^2 + 0.00015409159837459016*x^3)
typeof(PolyNew2)
=> Poly{Float64, :x}

I realize that all three outputs are basically the same thing, but I’d like to be sure that there isn’t something weird happening down the road in my (long & complicated) program due to some kind of data type mismatch.
(Also, I’m guessing that “best practices” would be to avoid your PolyCompat strategy, and just go with Polynomials.fit ?)

Thanks again!

5

Yes, just go with Polynomials.fit. This was changed quite some time ago and every time there is a breaking change, PolyCompat just begs to go since it was deprecated before “v1.0.” It remains to support older code bases with minimal adjustment. The difference you see in printout is due to changes in the underlying type that shouldn’t cause any difference when using the polynomials, but proved useful for generalizing the applicability.

1 Like
6

Actually, there WAS another error (in practically the very next line) due, I believe, to the new variable type (or some subtlety of how it’s handled differently):

FitPolynomial = Polynomials.fit(xArray, yArray, 3)

ArrayResiduals = yArray .- FitPolynomial(xArray)
# ERROR: MethodError: no method matching one(::Vector{Float64})

This DOES work with the old PolyFit. To be specific:

OLD Computer & Julia:

typeof(xArray)
=> Array{Float64,1}
typeof(FitPolynomial)
=> Poly{Float64}
typeof(FitPolynomial(xArray))
=> Array{Float64,1}

NEW Julia & Computer:

typeof(xArray)
=> Vector{Float64} (alias for Array{Float64, 1})
typeof(FitPolynomial)
=> Polynomial{Float64, :x} (alias for Polynomials.MutableDensePolynomial{Polynomials.StandardBasis, Float64, :x})
typeof(FitPolynomial(xArray)) 
#ERROR: MethodError: no method matching one(::Vector{Float64})
#
#Closest candidates are:
#  one(::Type{Union{}}, Any...)
#   @ Base number.jl:349
#  one(::Type{Gray24})
#   @ ColorVectorSpace C:\Users\phybdb\.julia\packages\ColorVectorSpace\tLy1N\src\ColorVectorSpace.jl:46
#  one(::Type{SparsityDetection.TermCombination})
#   @ SparsityDetection C:\Users\phybdb\.julia\packages\SparsityDetection\E7o7R\src\hessian.jl:8

Any idea what the correct new syntax for this would be? Thanks.

1 Like
7

should be replaced by

ArrayResiduals = yArray - FitPolynomial.(xArray)

note the dot in FitPolynomial.(xArray): here is the documentation Functions · The Julia Language.
On the other hand you don’t need the other dot on the subtraction because the two vectors have the same dimensions.

8

Aaargh! Both of these syntax structures used to work the SAME WAY in Old Julia for this calculation.

Has the usage of the “dot” terminology changed generally in the new version, or is this just a weird outcome of having a slightly different data type coming out of this specific change from old polyfit to new Polynomials.fit?

(I have LOTS of “dot” terminology commands in my programs, I hope this won’t lead to various crashes, or (worse) spuriously different results…!)

9

Not that I know. I found Polynomials.jl new way more consistent and I prefer it.

The old code does not work under Polynomials.jl v4.0.6 either: what Polynomials.jl version were you using with Julia v1.4?

1 Like
10

The Old Computer & Julia version was Polynomials v0.6.1, the new version was (as you say) Polynomials v4.0.6.

Oddly, using your other suggested strategy:
using Polynomials.PolyCompat
julia> polyfit(TryXfit, TryYfit, 3)

did make polyfit work again, but did NOT enable FitPolynomial(xArray) to work again. It was STILL necessary to do FitPolynomial.(xArray), so I suspect there is something generally different about how current Julia employs the “dot” broadcasting option. (Which seems to go against Julia’s promise of backwards compatibility!)

11

It looks like this method Polynomials.jl/src/Polynomials.jl at v0.6.1 · JuliaMath/Polynomials.jl · GitHub is missing in the new Polynomial version.

I found it with @which p(TryXfit) after downgrading Polynomials.jl to your version (with using Pkg; Pkg.add(name="Polynomials", version="0.6.1"))

Given that those methods are marked as legacy, I would not argue too strongly on the backwards compatibility.

12

Julia has mechanisms to create reproducible software environments. I suggest that we use them.

It seems to me that you upgraded both Julia and the package dependencies that you were using. In principle, you should be able to load the old package versions you used with Julia 1.4.2 in Julia 1.10.0. If that fails there are relatively easy mechanisms to run both versions side by side on the same computer.

I recommend locating the Project.toml and Manifest.toml from the old computer, making a backup copy, and then moving that over to the new computer. It should be in a folder such as ~/.julia/environments/v1.4 unless you created an environment specific for this project (recommended). Those files should allow you to reproduce the same environment in the new computer.

While it would be tempting to copy the TOML files into ``~/.julia/environments/v1.10on the new computer, I suggest putting them into~/.julia/environments/v1.4. You can then activate thsi environment by typing ] activate @v1.4` in the REPL. Then running the “instantiate” command. Alternatively, you can copy and paste these exact lines.

using Pkg
pkg"activate @v1.4"
pkg"instantiate"

You may also need to run pkg"resolve".

Did you use juliaup to install Julia on the new computer? Or perhaps the Microsoft Store? If so you can use juliaup to add a Julia 1.4.2 channel although this may different slightly due to using JuliaPro.

3 Likes
13

Hi all, my program found another reason to freeze indefinitely in the New Julia, this time I traced it to the following command, which used to work:

ImportantPlot = scatter(vec(xArray), vec(tArray),
        marker_z = vec(PlottingDataMatrix),
        markershape=:square, markersize=1.25, markerstrokewidth = 0.0,
        color=cgrad([:darkred,:red,:pink,:white,:lightblue,:blue,:darkblue]),
        clims=(maximum(abs.(PlottingDataMatrix)).*(-1.01, 1.01)), legend=false,
        title = "TitleStuff", xlabel = string(
        "x (Dist: Sqrt[A x B] (W/Osc!), Rel. to x at t = ",
        tLow,")"), ylabel = "ystuff");
        display(ImportantPlot)

Please let me know if you can see the error(s) I’m making here in Julia 1.10.

Also, mkitti: Yes, I did install with Juliaup. And, originally I tried going that route: taking my old Manifest.toml & Project.toml files and copying them to various directories; I tried the “activate” and “instantiate” and “resolve” commands; tried somehow to do a “per-project” environment; all of that, and NOTHING HAPPENED. (Do I type things in a command window? Do I type things in Visual Studio Code? What do I put where? What do I type where? I couldn’t figure it out.) So I said “the heck with it” and just upgraded to the latest Julia version (and still dealing with those errors).

If I can’t get Julia 1.10 working with my code, I might be forced to go back to Julia 1.4. Do you think you have the stamina to list what I have to do step-by-step, the way you would explain it to a 2nd-grader?

Thanks again to all for any help…

1 Like
14

It is essential to learn who to use local environments (per project) as explained here: Working with Julia projects | Julia programming notes

You could also try the freeze() command of PkgHelpers.jl on the old computer after creating a local environment on the old Julia computer.

15

Thanks ufechner7, I will try again. (I am going to start or continue a separate discussion thread on that.)

Adding in some preliminary stuff I left out from the above crashing command:

using OrdinaryDiffEq
using DiffEqBase
using DifferentialEquations
using DiffEqOperators
using BandedMatrices
using Sundials # for CVODE_BDF as a solver
using ModelingToolkit
using SparseArrays
using SparsityDetection

@time SolnOfDiffEqn = DifferentialEquations.solve(probForDiffEqn,
        CVODE_BDF(linear_solver=:GMRES), maxiters = Int(1e7),
        dense=true, reltol=1e-12, abstol=1e-12, progress=true)

PlottingDataMatrix = [SolnOfDiffEqn.u[i][j,2] for
                        i=1:4:length(SolnOfDiffEqn.u),
                        j=1:1:size(SolnOfDiffEqn.u[1])[1]]

Then:

ImportantPlot = scatter(vec(xArray), vec(tArray),
        marker_z = vec(PlottingDataMatrix),
        markershape=:square, markersize=1.25, markerstrokewidth = 0.0,
        color=cgrad([:darkred,:red,:pink,:white,:lightblue,:blue,:darkblue]),
        clims=(maximum(abs.(PlottingDataMatrix)).*(-1.01, 1.01)), legend=false,
        title = "TitleStuff", xlabel = string(
        "x (Dist: Sqrt[A x B] (W/Osc!), Rel. to x at t = ",
        tLow,")"), ylabel = "ystuff");
        display(ImportantPlot)
16
  1. Create a new folder called “oldenv142”
  2. Copy Project.toml and Manifest.toml into the folder “oldenv142”
  3. In the Julia REPL,
julia> cd "/path/to/oldenv142"

julia> using Pkg

julia> pkg"activate ."

julia> pkg"instantiate"

If that does not work, create a new folder “oldenv142_resolved”

julia> cd "/path/to/oldenv142"

julia> using Pkg

julia> pkg"activate ."

julia> pkg"resolve"

julia> pkg"instantiate"
17

Ok thanks mkitti.
For the 2 approaches:
(1) Update the code to work in the new environment,
(2) Re-install the old environment that was previously working,
I have tried to keep the two discussion threads separate, but they keep mixing.

First, for approach (1) – which seems the better long-term plan – is there any obvious mistake in my code above, which might cause it to fail in Julia 1.10.0, even though it worked in Julia 1.4.2?

In parallel, for approach (2), I will try what you said above.
Does it only use the old Julia & packages for programs in the “oldenv142” directory, and nowhere else?
(Also, if I ever add a new package, how do I stop the other packages from updating whenever they feel like it? Does compat have something to do with it?)

Thanks…

1 Like
18

Hi all, thanks everyone for all of your advice, my code is now working, with mostly no errors.

My problem with the program freezing during the plot command was an unexpected hardware issue, I have figured out how to work around it.

I have used your advice on activate, instantiate, etc., to run different environments in different directories, it seems to be working ok.

One last question related to the directories where I do update to Julia 1.10: I am getting a warning error on packages ModelingToolkit, Symbolics, and DomainSets, where the Warning messages are all of the form:

Precompiling ModelingToolkit
        Info Given ModelingToolkit was explicitly requested, output will be shown live
WARNING: Method definition isapprox(IntervalSets.AbstractInterval{T} where T, IntervalSets.AbstractInterval{T} where T) in module IntervalSets at C:\Users\Myself\.julia\packages\IntervalSets\viB6k\src\IntervalSets.jl:144 overwritten in module DomainSets at C:\Users\Myself\.julia\packages\DomainSets\aafhp\src\domains\interval.jl:52.it
ERROR: Method overwriting is not permitted during Module precompilation. Use `__precompile__(false)` to opt-out of precompilation.
  ? DomainSets
  ? Symbolics
  ? ModelingToolkit
[ Info: Skipping precompilation since __precompile__(false). Importing ModelingToolkit [961ee093-0014-501f-94e3-6117800e7a78].

These Warning/Error messages do not crash the code or stop it from running, but I would like to know what’s going on, and how (& if I should) silence the warning messages.

Thanks again!