MLJ.save() and restoring with machine() don't work

Paulo_Refosco · February 21, 2024, 6:13pm

Hello!

I made a simple XGBoost model with the pipeline below, fitted it with data and tried to save it and then restore it, but when restoring it (even if in the same notebook) I get error on trying to apply predict. Any idea on how to solve it?

XGBC = @load XGBoostClassifier
xgb = XGBC()
ohe = OneHotEncoder()

# Pipeline OneHotEncoder > XGBoost
xgb_pipe = ohe |> xgb

# Setting Target and Features tables:
y, X = unpack(df, ==(:y_label), col->true)

train, test = partition(1:length(y), 0.1, shuffle=true)

xgbm = machine(xgb_pipe, X, y, cache=false)
fit!(xgbm, rows=train, verbosity=0)

MLJ.save("mach_xgb_pipe.jls", xgbm)

# Restoring the model and using for predictions:
mach_restored = machine("mach_xgb_pipe.jls")

yhat = predict_mode(mach_restored , X[test,:])

Error message:

Error: Failed to apply the operation `predict` to the machine machine(:xg_boost_classifier, …), which receives it's data arguments from one or more nodes in a learning network. Possibly, one of these nodes is delivering data that is incompatible with the machine's model.
│ Model (xg_boost_classifier):
│ input_scitype = Unknown
│ target_scitype =Unknown
│ output_scitype =Unknown
│ 
│ Incoming data:
│ arg of predict	scitype
│ -------------------------------------------
│ Node @818 → :one_hot_encoder	Table{AbstractVector{Continuous}}
│ 
│ Learning network sources:
│ source	scitype
│ -------------------------------------------
│ Source @791	Table{Union{AbstractVector{Continuous}, AbstractVector{Multiclass{10}}, AbstractVector{Multiclass{2}}, AbstractVector{Multiclass{89}}, AbstractVector{Multiclass{6}}}}
│ Source @496	AbstractVector{OrderedFactor{2}}
└ @ MLJBase C:\Users\User\.julia\packages\MLJBase\mIaqI\src\composition\learning_networks\nodes.jl:153
XGBoostError: (caller: XGBoosterPredictFromDMatrix)
[14:39:55] /workspace/srcdir/xgboost/src/c_api/c_api.cc:1059: Booster has not been initialized or has already been disposed.

Stacktrace:
  [1] _apply(y_plus::Tuple{Node{Machine{Symbol, true}}, Machine{Symbol, true}}, input::DataFrame; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ MLJBase C:\Users\User\.julia\packages\MLJBase\mIaqI\src\composition\learning_networks\nodes.jl:159
  [2] _apply
    @ C:\Users\User\.julia\packages\MLJBase\mIaqI\src\composition\learning_networks\nodes.jl:144 [inlined]
  [3] (::Node{Machine{Symbol, true}})(Xnew::DataFrame)
    @ MLJBase C:\Users\User\.julia\packages\MLJBase\mIaqI\src\composition\learning_networks\nodes.jl:140
  [4] output_and_report(signature::MLJBase.Signature{NamedTuple{(:predict, :transform), Tuple{Node{Machine{Symbol, true}}, Node{Machine{Symbol, true}}}}}, operation::Symbol, Xnew::DataFrame)
    @ MLJBase C:\Users\User\.julia\packages\MLJBase\mIaqI\src\composition\learning_networks\signatures.jl:374
  [5] predict(model::MLJBase.ProbabilisticPipeline{NamedTuple{(:one_hot_encoder, :xg_boost_classifier), Tuple{Unsupervised, Probabilistic}}, MLJModelInterface.predict}, fitresult::MLJBase.Signature{NamedTuple{(:predict, :transform), Tuple{Node{Machine{Symbol, true}}, Node{Machine{Symbol, true}}}}}, Xnew::DataFrame)
    @ MLJBase C:\Users\User\.julia\packages\MLJBase\mIaqI\src\operations.jl:191
  [6] predict(mach::Machine{MLJBase.ProbabilisticPipeline{NamedTuple{(:one_hot_encoder, :xg_boost_classifier), Tuple{Unsupervised, Probabilistic}}, MLJModelInterface.predict}, false}, Xraw::DataFrame)
    @ MLJBase C:\Users\User\.julia\packages\MLJBase\mIaqI\src\operations.jl:133
  [7] predict
    @ C:\Users\User\.julia\packages\MLJTuning\drqMP\src\tuned_models.jl:795 [inlined]
  [8] predict_mode(m::MLJTuning.ProbabilisticTunedModel{Grid, MLJBase.ProbabilisticPipeline{NamedTuple{(:one_hot_encoder, :xg_boost_classifier), Tuple{Unsupervised, Probabilistic}}, MLJModelInterface.predict}}, fitresult::Machine{MLJBase.ProbabilisticPipeline{NamedTuple{(:one_hot_encoder, :xg_boost_classifier), Tuple{Unsupervised, Probabilistic}}, MLJModelInterface.predict}, false}, Xnew::DataFrame)
    @ MLJBase C:\Users\User\.julia\packages\MLJBase\mIaqI\src\interface\model_api.jl:11
  [9] predict_mode(mach::Machine{MLJTuning.ProbabilisticTunedModel{Grid, MLJBase.ProbabilisticPipeline{NamedTuple{(:one_hot_encoder, :xg_boost_classifier), Tuple{Unsupervised, Probabilistic}}, MLJModelInterface.predict}}, false}, Xraw::DataFrame)
    @ MLJBase C:\Users\User\.julia\packages\MLJBase\mIaqI\src\operations.jl:133
 [10] top-level scope
    @ In[113]:5

Side note: the model does work when applied directly to “predict”/“predict_mode”.
I also tried, without success:

using JLSO

smach = serializable(mach_tuned_xgb_pipe)
JLSO.save("machine_serialized.jlso", :machine => smach)

loaded_mach = JLSO.load("machine_serialized.jlso")[:machine]
restore!(loaded_mach)

yhat = predict_mode(loaded_mach, X[test,:])

Thanks for helping!

ablaom · February 21, 2024, 10:58pm

Thanks for reporting. This is issue has been reported elsewhere, and its good to have your example because I was previously unable to reproduce it. Here is the issue link: Serialized Composite Model Fails with XGBoost · Issue #927 · JuliaAI/MLJBase.jl · GitHub

Paulo_Refosco · February 22, 2024, 10:21am

Thanks a lot for the quick answer @ablaom !

I just provided a not-working code with an example csv file attached in the git issue (Serialized Composite Model Fails with XGBoost · Issue #927 · JuliaAI/MLJBase.jl · GitHub). Let me know if there is anything else that I can do (and also when/if it gets solved!)

Thanks!

ablaom · February 23, 2024, 3:04am

Note a possible workaround is to use the gradient boosted trees provided by the pure Julia implementation EvoTrees.jl . These should serialise fine in pipelines.

Paulo_Refosco · February 23, 2024, 2:23pm

@ablaom , do you think it is something you will be able to solve in the near future (both because of direction of efforts and/or priorities)? I’m pursuing an organizational shift towards Julia, and since xgboost is widely adopted here, I’ll need either a workaround or maybe even switching to using directly XGBoost.jl (even though I find MLJ usage more user-friendly, although I also can’t get the xgb model to show feature importances when using MLJ (but this probably to my lack of coding knowledge)).

Anyway, it was great to have such prompt answers from you! (thanks a lot for that!).

Paulo_Refosco · February 23, 2024, 3:19pm

By the way, the easiest workaround I figured is to keep using MLJ, but break the model away from the pipeline (so not using xgb_pipe = ohe |> xgb , but instead treating the raw data with “ohe” in a machine, then another machine to input this treated data and running xgb. Then it’s a matter of saving both machines (ohe and xgb) and restoring them etc). This way is working. The problem seems to be on pipelines…

ablaom · February 26, 2024, 2:01am

do you think it is something you will be able to solve in the near future

I’ll try to look at it this week.

although I also can’t get the xgb model to show feature importances when using MLJ

Definitely raise an issue if you are having trouble. It’s likely that feature importances of a supervised model within a pipeline are not directly accessible, and so adding an issue to request this makes sense.

ablaom · February 29, 2024, 10:37pm

Fix coming soon: Fix problem with serialization of nested models when component model overload `save`/`restore` by ablaom · Pull Request #960 · JuliaAI/MLJBase.jl · GitHub

ablaom · March 1, 2024, 2:11am

This should be resolved by updating to latest versions (you need MLJBase 1.1.2). Feel free to re-open Serialized Composite Model Fails with XGBoost · Issue #927 · JuliaAI/MLJBase.jl · GitHub if some issue persists.

@Paulo_Refosco Would you mind editing the post title to make it more specific? The issue was limited to XGBoost.jl models.

Paulo_Refosco · March 1, 2024, 2:48pm

Thanks @ablaom !

As I mentioned in Serialized Composite Model Fails with XGBoost · Issue #927 · JuliaAI/MLJBase.jl · GitHub, saving and restoring a Pipeline including XGBoost using MLJ.save() now is working for me!

For the title, I definitely don’t mind changing it, but, maybe due to my profile here (or maybe because I don’t know how to use it), it seems I am unable to change the titles of my topics… I’ll look further into it though.

I will consider raising the issue on features importances and also let me point out that I tried saving the tuned xgb pipe model and restoring it ended up in the same error as we were getting for the “regular” xgb pipe. But I don’t know if it would really be expected to work on the tuned machine, so just pointing out.

And again, thanks very much for all the efforts. It helped me a lot!

ablaom · March 3, 2024, 8:55pm

github.com/alan-turing-institute/MLJ.jl

Deserialisation fails for wrappers like `TunedModel` when atomic model overloads `save/restore`

opened 08:55PM - 03 Mar 24 UTC

ablaom

XGBoost.jl models have non-persistent `fitresult`s which means that they cannot …be directly serialised. That's not a problem, because such models can overload MLJModelInterface's `save` and `restore`. However, it [has been reported](https://discourse.julialang.org/t/mlj-save-and-restoring-with-machine-dont-work/110535/10?u=ablaom) that XGBoost models wrapped in `TunedModel` don't deserialise properly. Here is a MWE: First, we defined a supervised model, `EphemeralRegressor` with an ephemeral fitresult. For this model we overload `save/restore` to ensure deserialization works, provided you use the [correct API](https://alan-turing-institute.github.io/MLJ.jl/dev/machines/#Saving-machines). <details> ```julia using Statistics, MLJBase, Test, MLJTuning, Serialization, StatisticalMeasures import MLJModelInterface # define a model with non-persistent fitresult: thing = [] struct EphemeralRegressor <: Deterministic end function MLJModelInterface.fit(::EphemeralRegressor, verbosity, X, y) # if I serialize/deserialized `thing` then `view` below changes: view = objectid(thing) fitresult = (thing, view, mean(y)) return fitresult, nothing, NamedTuple() end function MLJModelInterface.predict(::EphemeralRegressor, fitresult, X) thing, view, μ = fitresult return view == objectid(thing) ? fill(μ, nrows(X)) : throw(ErrorException("dead fitresult")) end function MLJModelInterface.save(::EphemeralRegressor, fitresult) thing, _, μ = fitresult return (thing, μ) end function MLJModelInterface.restore(::EphemeralRegressor, serialized_fitresult) thing, μ = serialized_fitresult view = objectid(thing) return (thing, view, μ) end # EphemeralRegressor cannot be directly serialized: X, y = (; x = rand(3)), fill(42.0, 3) model = EphemeralRegressor() mach = machine(model, X, y) |> fit! io = IOBuffer() serialize(io, mach) seekstart(io) mach2 = deserialize(io) @test_throws ErrorException("dead fitresult") predict(mach2, 42) # But it can be serialized/deserialized using correct API: io = IOBuffer() serialize(io, serializable(mach)) seekstart(io) mach2 = restore!(deserialize(io)) @test MLJBase.predict(mach2, (; x = rand(2))) ≈ fill(42.0, 2) ``` </details> But wrapping this model using `TunedModel` leads to deserialization failure: ```julia tmodel = TunedModel( models=fill(EphemeralRegressor(), 2), measure = l2, ) mach = machine(tmodel, X, y) |> fit! io = IOBuffer() serialize(io, serializable(mach)) seekstart(io) mach2 = restore!(deserialize(io)) MLJBase.predict(mach2, (; x = rand(2))) # ERROR: dead fitresult # Stacktrace: # [1] predict(::EphemeralRegressor, fitresult::Tuple{Vector{Any}, UInt64, Float64}, X::@NamedTuple{x::Vector{Float64}}) # @ Main ./REPL[7]:3 # # < truncated trace > ``` The remedy is to properly "forward" the `save`/`restore` methods in all model wrappers. We can exclude any wrapper model implemented as `NetworkComposite` (ie, using learning networks) as they already overload `save` and `restore` [properly](https://github.com/JuliaAI/MLJBase.jl/pull/960). To do: - [ ] `TunedModel` - [ ] `IteratedModel` - [ ] `EnsembleModel` - [ ] `BinaryThresholdPredictor` ?

ablaom · March 7, 2024, 8:06pm

Above issue resolved by MLJTuning 0.8.2