# How to train MLJ model on DataFrame, but apply on Vector of Arrays?

**URL:** <https://discourse.julialang.org/t/how-to-train-mlj-model-on-dataframe-but-apply-on-vector-of-arrays/114669>\
**Category:** Machine Learning\
**Tags:** question, mlj\
**Created:** [May 24, 2024, 9:40am UTC](https://discourse.julialang.org/t/how-to-train-mlj-model-on-dataframe-but-apply-on-vector-of-arrays/114669 "2024-05-24T09:40:05Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![niltsz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/niltsz/32/30060_2.png) [@niltsz](https://discourse.julialang.org/u/niltsz)\
**Post date:** [May 24, 2024, 9:40am UTC](https://discourse.julialang.org/t/how-to-train-mlj-model-on-dataframe-but-apply-on-vector-of-arrays/114669/1 "2024-05-24T09:40:05Z")

</div>

I train my MLJ Model (EvoTrees.jl) on a Julia DataFrame with the scalars x1, x2, and x3 as features and y as label. However, when doing the inference, I have higher dimensional arrays instead of tabular data. Any suggestions how to code this in a nice way?

```julia
df = DataFrame()
#input is nearest neighbor from a grid
df.x1 = rand(100)
df.x2 = rand(100)
df.x3 = rand(100)
#target is measured at a point within grid
df.y = rand(100)

ETR = MLJ.@load EvoTreeRegressor pkg=EvoTrees
evotree = ETR()
mach = machine(evotree, df[:, Not(:y)], df.y)
fit!(mach)

######## inference ########
X1_inf = rand(100, 100, 10)
X2_inf = rand(100, 100, 10)
X3_inf = rand(100, 100, 10)

#this is what I want to do, 
#I can come up with ways that work but feel wrong, 
#like three for loops and casting, but how to do it in a smart way?
Y_inf = MLJ.predict(mach, [X1_inf, X2_inf, X3_inf])

```

The most simple way would be to train on arrays instead of a DataFrame and then broadcast, or, during inference, to make a for loop over all indices and then cast it to a DataFrame. But is there a smarter way?

For context if this helps, I have a climate model that is gridded (latitude, longitude, height), and a plane trajectory going through the grid. For training, I take the weather (e.g. temperature, air pressure and wind direction) at the nearest neighbor from the grid as features and the measurements (e.g. relative humidity) from the plane as target, so tabular data. For inference, I want to feed in all grid points of the weather model and get out a grid as well.

---

<div class="post-metadata">

**Author:** ![bertschi](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bertschi/32/33462_2.png) [@bertschi](https://discourse.julialang.org/u/bertschi)\
**Post date:** [May 24, 2024, 3:49pm UTC](https://discourse.julialang.org/t/how-to-train-mlj-model-on-dataframe-but-apply-on-vector-of-arrays/114669/2 "2024-05-24T15:49:50Z")

</div>

Seems that `predict` already works on arrays, i.e.,

```julia-repl
julia> MLJ.predict(mach, rand(4, 3)) # Just pass 3 columns instead of a data frame
4-element Vector{Float32}:
 0.354614
 0.889316
 0.60572267
 0.23449774

```

Given that you have several options to run over the grid:

1. Apply model elementwise and collect results:

```julia-repl
julia> @time stack((x1,x2,x3) -> only(MLJ.predict(mach, [x1 x2 x3])), X1_inf, X2_inf, X3_inf) |> size
18.615153 seconds (65.36 M allocations: 6.271 GiB, 4.73% gc time, 0.48% compilation time)
(100, 100, 10)

```

This is rather slow though
2. Create a suitable array, e.g., via reshaping

```julia
function predict_grid(mach, x1, x2, x3)
    s = size(x1)
    @assert s == size(x2) == size(x3)
    x = reshape(stack([x1, x2, x3]), prod(s), 3)
    reshape(MLJ.predict(mach, x), s...)
end

```

which runs much faster

```julia-repl
julia> @time predict_grid(mach, X1_inf, X2_inf, X3_inf) |> size
0.235947 seconds (664 allocations: 3.402 MiB)
(100, 100, 10)

```

---

<div class="post-metadata">

**Author:** ![niltsz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/niltsz/32/30060_2.png) [@niltsz](https://discourse.julialang.org/u/niltsz)\
**Post date:** [May 25, 2024, 5:44pm UTC](https://discourse.julialang.org/t/how-to-train-mlj-model-on-dataframe-but-apply-on-vector-of-arrays/114669/4 "2024-05-25T17:44:36Z")

</div>

@bertschi @brandon698sherrick  
Thanks a lot for the help. I will reshape my data in a separate funciton.